≡ Menu

Lighttpd block wget useragent for specific urls

One of regular reader asks a question:

My website powered by Lighttpd web server. I’d like to block Wget useragent for entire my domain.com site except for /downloads/ url section. How do I configure lighttpd?

You need to use $HTTP filed useragent and url combination. Just open your lighttpd.conf file and append code as follows.

Lighttpd block useragent wget configuration

# vi /etc/lighttpd/lighttpd.conf
Append config directive as follows:

$HTTP["useragent"] =~ "Wget" { 
        $HTTP["url"] !~ "^/download($|/)" { 
                url.access-deny = ( "" )  
        }
  }

Where,

  • $HTTP[“useragent”] : Match on useragent i.e. Wget
  • $HTTP[“url”] : Match on url section such as /download/*. If there are nested blocks, this must be the most inner block.
  • =~ : Perl style regular expression match
  • !~ : Perl style regular expression not match

Just restart the webserver, enter:
# /etc/init.d/lighttpd restart

Now user can run wget on http://domain.com/download/* urls but not on http://domain.com/file.html or http://domain.com/dir/file

Share this on:

Your support makes a big difference:
I have a small favor to ask. More people are reading the nixCraft. Many of you block advertising which is your right, and advertising revenues are not sufficient to cover my operating costs. So you can see why I need to ask for your help. The nixCraft, takes a lot of my time and hard work to produce. If you use nixCraft, who likes it, helps me with donations:
Become a Supporter →    Make a contribution via Paypal/Bitcoin →   

Don't Miss Any Linux and Unix Tips

Get nixCraft in your inbox. It's free:



{ 5 comments… add one }
  • ear August 14, 2007, 10:40 pm

    This is pointless imo.
    Users will still be able to use wget on your whole domain if they use wget –user-agent ….

  • 悉尼 July 8, 2009, 7:41 am

    Is it possible to match all the blank agent?

  • pat October 1, 2010, 10:05 am

    # user-agent empty or made of any number of spaces rejected
    $HTTP[“useragent”] =~ “^ *$” {
    url.access-deny = ( “” )
    }

    there’s a single space between ^ and *

  • Nananaan February 15, 2012, 2:11 pm

    Just use –header=”User-agent: blalbalb” to circumvent this from wget.

  • Munwar Hussain December 30, 2014, 5:50 pm

    dear

    i am not understand you think

    so kindly teach us detal line by line and step by step

Security: Are you a robot or human?

Leave a Comment


   Tagged with: , , ,