Save Disk Space under Apache Web server

Recently I had noticed some of our clients web sites generating lot of errors (on shared hosting account). Therefore, I decided to see what is going on. To my surprise most of the errors were related to two files only robots.txt and favicon.ico.

Many of the search engine and other robots use robots.txt file. Other browsers try to open special ICON file called favicon.ico. A Favicon is a multi-resolution image included on nearly all professional developed sites. Recent growth of Mozilla Firefox is also one of the causes to fill up error file as they try to open favicon.ico file.
A robot is a program that automatically traverses the Web’s hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. You can control or configure agents via robots.txt file.

As most of the end user do not creates robots.txt and favicon.ico files. A error log entry generated in users error log file (for e.g. /chroot/www/a/ file), every time someone visit site. If site receives decent traffic, error log file becomes larger in quick time, wasting in disk space and not to mention complaints from hosting customer about disk space problem 😉

Solution was simple we sent an email to all of our customer describing the problem and suggested them to put both files in root directory of web hosting account. If you maintain shared hosting Apache web server, then you can put robots.txt and favicon.ico files by default to avoid errors.

For more info please see:
* Robots.txt file and FAQ.

🐧 Get the latest tutorials on Linux, Open Source & DevOps via RSS feed or Weekly email newsletter.

🐧 0 comments... add one
CategoryList of Unix and Linux commands
Disk space analyzersncdu pydf
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network UtilitiesNetHogs dig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
0 comments… add one

Leave a Reply

Your email address will not be published.

Use HTML <pre>...</pre> for code samples. Still have questions? Post it on our forum