Lighttpd block wget useragent for specific urls

One of regular reader asks a question:

ADVERTISEMENTS

My website powered by Lighttpd web server. I’d like to block Wget useragent for entire my domain.com site except for /downloads/ url section. How do I configure lighttpd?

You need to use $HTTP filed useragent and url combination. Just open your lighttpd.conf file and append code as follows.

Lighttpd block useragent wget configuration

# vi /etc/lighttpd/lighttpd.conf
Append config directive as follows:

$HTTP["useragent"] =~ "Wget" {
        $HTTP["url"] !~ "^/download($|/)" {
                url.access-deny = ( "" )
        }
  }

Where,

  • $HTTP[“useragent”] : Match on useragent i.e. Wget
  • $HTTP[“url”] : Match on url section such as /download/*. If there are nested blocks, this must be the most inner block.
  • =~ : Perl style regular expression match
  • !~ : Perl style regular expression not match

Just restart the webserver, enter:
# /etc/init.d/lighttpd restart

Now user can run wget on http://domain.com/download/* urls but not on http://domain.com/file.html or http://domain.com/dir/file

🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source & DevOps topics via:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04

ADVERTISEMENTS
5 comments… add one
  • ear Aug 14, 2007 @ 22:40

    This is pointless imo.
    Users will still be able to use wget on your whole domain if they use wget –user-agent ….

  • 悉尼 Jul 8, 2009 @ 7:41

    Is it possible to match all the blank agent?

  • pat Oct 1, 2010 @ 10:05

    # user-agent empty or made of any number of spaces rejected
    $HTTP[“useragent”] =~ “^ *$” {
    url.access-deny = ( “” )
    }

    there’s a single space between ^ and *

  • Nananaan Feb 15, 2012 @ 14:11

    Just use –header=”User-agent: blalbalb” to circumvent this from wget.

  • Munwar Hussain Dec 30, 2014 @ 17:50

    dear

    i am not understand you think

    so kindly teach us detal line by line and step by step

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.