Apache piped logs

What I set out to do for this task was to have Apache create a log file for each site on a mass virtual hosting setup, using mod_vhost_alias and store them in a structured directory. Such as /var/log/apache/www.domain.com/2010/10/30/access_log. Creating the directories as it goes, the end result was a perl script used with Acache’s piped logs feature.

Apache does a pretty good job of letting you do fun custom things with you http logs, by if you happen to be using mod_vhost_alias you will find that your options become limited, one big log or nothing. Mass virtual hosting with Apache is basically a good thing, if you have a highly subscribed server you can run it with a single virtual host. One size fits all, though there are some limitations on how you can use it with other configuration options.

These lack of additional configuration options, for example using the same syntax in CustomLog as you do in VirtualDocumentRoot, or securing PHP/CGI and chroot the sites in their home directories (this one we will come to in a later post).

First, we create our perl script.
For example, /usr/local/bin/vhost_access.pl


use strict;

my $logEntry;
my $accessLogName = "access_log";
my $logDir = "/var/log/apache";
my $vhostLogDir;

sub writeLog;

while( <> )

    $logEntry = $_;

    # get the vhost from this log entry:
    $logEntry=~/(.*?) /;
    my $vhost = $1;
    my ($year, $month, $day) = ((localtime)[5]+1900,
                               sprintf("%02d", (localtime)[4]+1),
                               sprintf("%02d", (localtime)[3])

    # remove 'www'. from domain path
    $vhost =~ s!^www.?!!i;

    # Name of access logfiles:
    $vhostLogDir = "$logDir/$vhost/$year/$month/$day";

# write a log entry to a file
sub writeLog()
    my $logDir = shift @_;

    if( ! -d $logDir )
        `/bin/mkdir -p $logDir`;

    open(FD, ">>$logDir/$accessLogName");

    # strip off the vhost data:
    $logEntry =~s/.*? //;

    print FD $logEntry;
    close FD;

Next add the following line into your httpd.conf (It does *not* need to be inside the <VirtualHost tag>).

LogFormat "%V %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" vcombined
CustomLog "|/usr/local/bin/vhost_access.pl" vcombined

Now restart Apache. By doing `ps ax` or equivalent should now show the additional process for vhost_access.pl.

That’s it! Generate some traffic (If not being access already by the world) and enjoy tailing your new log files.

You may also like...

  • bob

    This is a weak solution because it causes an additional open() and close() syscall for every single log line. Does anyone have some code that keeps the files open for writing?

  • I agree it would be better to cut down on syscalls, the issue really is that we don’t know what the log file path is going to be until we have the log line. Only when we have it can we analyse it and work out which vhost it’s for.