Compressing large Apache log files on a shared hosting server

by on December 31, 2005 · 3 comments· LAST UPDATED December 31, 2005

in

If your site is quite popular then your apache log file could quickly becomes large. It could be anywhere between 200MB or 1 GB per month. Downloading such large apache log file not just take lot of time but bandwidth too. We have lots of shared hosting customer on apache web server. If log files are small, they you can download it easily using ftp program.

Nevertheless, when log file becomes larger, downloading them is a big problem. For security reason no hosting provider provides ssh access, further system(), exec() and other php functions are disabled in php.

How the hell customer will get his/her apache raw log file. With the help of php script you can compress file and then download it. Php provides function called gzwrite() which is binary safe gz-file writer.

Syntax:

int gzwrite ( resource zp, string string [, int length] )

Where,

  • zp : is a file pointer
  • string : the string to write
  • length : the number of uncompressed bytes to write

PHP script:

<?
// How to use?
// http://mydomain.com/private/gzip?i=access.log&o=/access.log.gz
function gzip ($in, $out, $param="1")
{
if (!file_exists ($in) || !is_readable ($in))
 return false;
if ((!file_exists ($out) && !is_writable (dirname ($out)) ||
(file_exists($out) &amp;& !is_writable($out)) ))
 return false;
$in_file = fopen ($in, "rb");
if (!$out_file = gzopen ($out, "wb".$param)) {
 return false;
}
while (!feof ($in_file)) {
 $buffer = fgets ($in_file, 4096);
 gzwrite ($out_file, $buffer, 4096);
}
fclose ($in_file);
gzclose ($out_file);
return true;
}
$in=$_GET['i'];
$out=$_GET['o'];
echo "<html><head><title>Php Gzip program</title></head><body>\n";
if (
echo "Status:";
if ( (gzip($in,$out,"9")) == false ){
echo "<h1>Failed</h1>\n";
}
else {
echo "<h1>Done</h1>\n";
}
echo "</body></html>";
?>

Just upload script to your webserver directory in password-protected directory. Then run php script by typing http url:

http://mydomain/private/gzip.php?i=access.log&o=access.log.gz

Where,

  • i=file.log: Raw Apache log file (as a Input file)
  • o=file.log.gz: Output file i.e. compressed file

Result

  • Raw access.log file size 267730KB before running gzip.php
  • Raw access.log.gz file size 16301KB after compressing using gzip.php

Make sure private directory is apache password protected, otherwise anyone can use your script. Please note that I am not a php expert or regular programmer, if you have a better solution or script please feel free to post it.

Reference:

It is now New Years Eve. Year 2005 was a good year for all of us at nixCraft. We are proud of our small community and shared learning that we do here. Thanks to everyone who has contributed to increase our knowledge of Linux and Open source by leaving a comment or links.

Happy New Year [Naye Varsha Ki Shubhkamanyen (hindi)]
We do appreciate you and hope that you will have a great New Year celebration.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 3 comments… read them below or add one }

1 Anonymous December 31, 2005 at 8:09 pm

Happy New Year to everyone at nixCraft. Thanks for writing all wonderful articles. Hope to see more in 2006.

Reply

2 christopher December 31, 2005 at 8:41 pm

Thank you for all of the wonderful Linux and UNIX tips you guys provided! Best wishes for 2006!!!

Christopher

Reply

3 Anonymous January 2, 2006 at 7:10 pm

Best Wishes for 2006 and keep up the excellent work!

Reply

Leave a Comment

Previous post:

Next post: