Save Disk Space under Apache Web server

Posted by Vivek Gite [Last updated: October 11, 2005]

Recently I had noticed some of our clients web sites generating lot of errors (on shared hosting account). Therefore, I decided to see what is going on. To my surprise most of the errors were related to two files only robots.txt and favicon.ico.

Many of the search engine and other robots use robots.txt file. Other browsers try to open special ICON file called favicon.ico. A Favicon is a multi-resolution image included on nearly all professional developed sites. Recent growth of Mozilla Firefox is also one of the causes to fill up error file as they try to open favicon.ico file.
A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. You can control or configure agents via robots.txt file.

As most of the end user do not creates robots.txt and favicon.ico files. A error log entry generated in users error log file (for e.g. /chroot/www/a/abc.com/logs/error.log file), every time someone visit site. If site receives decent traffic, error log file becomes larger in quick time, wasting in disk space and not to mention complaints from hosting customer about disk space problem
Solution was simple we sent an email to all of our customer describing the problem and suggested them to put both files in root directory of web hosting account. If you maintain shared hosting Apache web server, then you can put robots.txt and favicon.ico files by default to avoid errors.