Lighttpd block wget useragent for specific urls

by on August 12, 2007 · 4 comments· LAST UPDATED August 12, 2007

in , ,

One of regular reader asks a question:

My website powered by Lighttpd web server. I’d like to block Wget useragent for entire my domain.com site except for /downloads/ url section. How do I configure lighttpd?

You need to use $HTTP filed useragent and url combination. Just open your lighttpd.conf file and append code as follows.

Lighttpd block useragent wget configuration

# vi /etc/lighttpd/lighttpd.conf
Append config directive as follows:

$HTTP["useragent"] =~ "Wget" {
        $HTTP["url"] !~ "^/download($|/)" {
                url.access-deny = ( "" )
        }
  }

Where,

  • $HTTP["useragent"] : Match on useragent i.e. Wget
  • $HTTP["url"] : Match on url section such as /download/*. If there are nested blocks, this must be the most inner block.
  • =~ : Perl style regular expression match
  • !~ : Perl style regular expression not match

Just restart the webserver, enter:
# /etc/init.d/lighttpd restart

Now user can run wget on http://domain.com/download/* urls but not on http://domain.com/file.html or http://domain.com/dir/file

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 4 comments… read them below or add one }

1 ear August 14, 2007 at 10:40 pm

This is pointless imo.
Users will still be able to use wget on your whole domain if they use wget –user-agent ….

Reply

2 悉尼 July 8, 2009 at 7:41 am

Is it possible to match all the blank agent?

Reply

3 pat October 1, 2010 at 10:05 am

# user-agent empty or made of any number of spaces rejected
$HTTP["useragent"] =~ “^ *$” {
url.access-deny = ( “” )
}

there’s a single space between ^ and *

Reply

4 Nananaan February 15, 2012 at 2:11 pm

Just use –header=”User-agent: blalbalb” to circumvent this from wget.

Reply

Leave a Comment

Tagged as: , , ,

Previous post:

Next post: