Recently I had noticed some of our clients web sites generating lot of errors (on shared hosting account). Therefore, I decided to see what is going on. To my surprise most of the errors were related to two files only robots.txt and favicon.ico.
Many of the search engine and other robots use robots.txt file. Other browsers try to open special ICON file called favicon.ico. A Favicon is a multi-resolution image included on nearly all professional developed sites. Recent growth of Mozilla Firefox is also one of the causes to fill up error file as they try to open favicon.ico file.
A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. You can control or configure agents via robots.txt file.
As most of the end user do not creates robots.txt and favicon.ico files. A error log entry generated in users error log file (for e.g. /chroot/www/a/abc.com/logs/error.log file), every time someone visit site. If site receives decent traffic, error log file becomes larger in quick time, wasting in disk space and not to mention complaints from hosting customer about disk space problem ;)
Solution was simple we sent an email to all of our customer describing the problem and suggested them to put both files in root directory of web hosting account. If you maintain shared hosting Apache web server, then you can put robots.txt and favicon.ico files by default to avoid errors.
- 30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X
- Top 30 Nmap Command Examples For Sys/Network Admins
- 25 PHP Security Best Practices For Sys Admins
- 20 Linux System Monitoring Tools Every SysAdmin Should Know
- 20 Linux Server Hardening Security Tips
- Linux: 20 Iptables Examples For New SysAdmins
- Top 20 OpenSSH Server Best Security Practices
- Top 20 Nginx WebServer Best Security Practices
- 20 Examples: Make Sure Unix / Linux Configuration Files Are Free From Syntax Errors
- 15 Greatest Open Source Terminal Applications Of 2012
- My 10 UNIX Command Line Mistakes
- Top 10 Open Source Web-Based Project Management Software
- Top 5 Email Client For Linux, Mac OS X, and Windows Users
- The Novice Guide To Buying A Linux Laptop