Compressing large Apache log files on a shared hosting server

Posted by Vivek Gite [Last updated: December 31, 2005]

If your site is quite popular then your apache log file could quickly becomes large. It could be anywhere between 200MB or 1 GB per month. Downloading such large apache log file not just take lot of time but bandwidth too. We have lots of shared hosting customer on apache web server. If log files are small, they you can download it easily using ftp program.

Nevertheless, when log file becomes larger, downloading them is a big problem. For security reason no hosting provider provides ssh access, further system(), exec() and other php functions are disabled in php.

How the hell customer will get his/her apache raw log file. With the help of php script you can compress file and then download it. Php provides function called gzwrite() which is binary safe gz-file writer.

Syntax:

int gzwrite ( resource zp, string string [, int length] )

Where,

zp : is a file pointer
string : the string to write
length : the number of uncompressed bytes to write

PHP script:

<?
// How to use?
// http://mydomain.com/private/gzip?i=access.log&o=/access.log.gz
function gzip ($in, $out, $param="1")
{
if (!file_exists ($in) || !is_readable ($in))
 return false;
if ((!file_exists ($out) && !is_writable (dirname ($out)) ||
(file_exists($out) &amp;& !is_writable($out)) ))
 return false;

$in_file = fopen ($in, "rb");
if (!$out_file = gzopen ($out, "wb".$param)) {
 return false;
}

while (!feof ($in_file)) {
 $buffer = fgets ($in_file, 4096);
 gzwrite ($out_file, $buffer, 4096);
}

fclose ($in_file);
gzclose ($out_file);

return true;
}

$in=$_GET['i'];
$out=$_GET['o'];
echo "<html><head><title>Php Gzip program</title></head><body>\n";
if (
echo "Status:";
if ( (gzip($in,$out,"9")) == false ){
echo "<h1>Failed</h1>\n";
}
else {
echo "<h1>Done</h1>\n";
}
echo "</body></html>";
?>

Just upload script to your webserver directory in password-protected directory. Then run php script by typing http url:

http://mydomain/private/gzip.php?i=access.log&o=access.log.gz

Where,

i=file.log: Raw Apache log file (as a Input file)
o=file.log.gz: Output file i.e. compressed file

Result

Raw access.log file size 267730KB before running gzip.php
Raw access.log.gz file size 16301KB after compressing using gzip.php

Make sure private directory is apache password protected, otherwise anyone can use your script. Please note that I am not a php expert or regular programmer, if you have a better solution or script please feel free to post it.

Reference:

See php.net/gzwrite and other php.net documents

It is now New Years Eve. Year 2005 was a good year for all of us at nixCraft. We are proud of our small community and shared learning that we do here. Thanks to everyone who has contributed to increase our knowledge of Linux and Open source by leaving a comment or links.

Happy New Year [Naye Varsha Ki Shubhkamanyen (hindi)]
We do appreciate you and hope that you will have a great New Year celebration.

Tell us how we're doing: Please answer a few questions about your experience to help us improve nixCraft.

Discussion on This Article:

Anonymous Says: December 31st, 2005 at 8:09 pm
Happy New Year to everyone at nixCraft. Thanks for writing all wonderful articles. Hope to see more in 2006.
christopher Says: December 31st, 2005 at 8:41 pm
Thank you for all of the wonderful Linux and UNIX tips you guys provided! Best wishes for 2006!!!

Christopher
Anonymous Says: January 2nd, 2006 at 7:10 pm
Best Wishes for 2006 and keep up the excellent work!