Introduction
In the beginning, there was the site; and it was good. Then Doug looked down and said "who visits this?" A quick Google search later brought me to a program called Webalizer.
Webalizer processes log files, and computes all sorts of information and generates lovely graphs. All of this data is then outputted as HTML and PNG files. Perfect for viewing on the web!
Alas, the original version does not seem to properly support IIS 5.0. Fortunately, Webalizer was released under the GNU GPL. Allowing other companies/individuals the ability to modify the source. Which led me a Canadian company called Stone Steps. They have a version that processes IIS log files.
Victory is mine! Well, Almost.
The version of Webalizer from Stone Steps seems to process IIS logs very well. The documentation says that Webalizer reconfigures its parser to handle whatever fields are in the log files. It's been my experience (YMMV) that Webalizer works best when the log files contain every single field made available by IIS in the extended log format.
IIS, by default, logs requests by date. Meaning that any request for November, 14, 2005 will be stored in a file called ex051114.log. Unfortunately, Webalizer cannot just be given a folder full of logs to process. You have to incrementally process each file in the folder. Fortunately, this is available via the -p argument. The incremental data is stored, by default, in a file called webalizer.hist. This has the side benefit that future runs against this file (unless the file has changed) are ignored, since it has already been processed.
The use of the incremental argument is very easy. Stone Steps even has an answer in their FAQ for this very purpose.
Customization
What I like most about Webalizer is the ability to customize the output. Options can be fed via the commandline, or via a configuration file. Perfect for scheduling. The documentation that comes with Webalizer, plus the Stone Steps website is invaluable for coming up with exactly what you want to see.
Set Up
My directory stucture is as follows (this is important for the automation I use later):
-
sites\
webalizer\ (Webalizer's Install Location)
Process Logs.bat (Automation File, See Below)
Process Logs.wsf (Automation File, See Below)
Automation
First we'll start off with a simple BAT file that processes the log files.
set site="dougjenkinson.net"
for /F %%f in ('dir /O:N /B sites\%site%\logs\W3SVC1\*.log') do (
webalizer -p -F iis -c sites\%site%\%site%.conf -o sites\%site%\report sites\%site%\logs\W3SVC1\%%f
)
unset site
Command Line Arguments:
- -p: Incremental processing.
- -F iis: Log Format iis.
- -c sites\%site%\%site%.conf: Use a specific configuration file.
- -o sites\%site%\report: Output director.
- sites\%site%\logs\%%f: The log file to process.
It's pretty easy to set this up with a batch file. But, when you manage multiple sites, you need a more elegant solution that scales to additional sites.
Windows Scripting Host Jobs
<package>
<job id="WickedStrategery">
<script language="VBScript">
Option Explicit
Dim strSiteName: strSiteName = "dougjenkinson.net"
Dim strLogName: strLogName = "W3SVC1"
Dim strSiteFolder: strSiteFolder = "sites\" & strSiteName
Dim strConfigFile: strConfigFile = strSiteFolder & "\" & strSiteName & ".conf"
Dim strOutputFolder: strOutputFolder = strSiteFolder & "\report\"
Dim strDNSFile: strDNSFile = "../../dbcache.db"
Dim fso, objLogFolder, objLogFile, objShell
Set fso = WScript.CreateObject("Scripting.FileSystemObject")
Set objShell = WScript.CreateObject("WScript.Shell")
Set objLogFolder = fso.GetFolder("sites\" & strSiteName & "\logs\" & strLogName & "\")
For Each objLogFile In objLogFolder.Files
'Call WScript.Echo(strSiteName & " - " & objLogFile.Name)
Call objShell.Run("webalizer\webalizer -p -Q -F iis -c " & strConfigFile & " -D " & strDNSFile & " -N 10 -o " & strOutputFolder & " """ & objLogFile.Path & """", 0, True)
Next: Set objLogFile = Nothing
Set objLogFolder = Nothing
Set objShell = Nothing
Set fso = Nothing
Call WScript.Echo("Processing Completed.")
</script>
</job>
</package>
Command Line Arguments Used:
- -p: Incremental processing.
- -Q: Super-quiet, no output to console.
- -F iis: Log Format iis.
- -c strConfigFile: Use a specific configuration file.
- -D strDNSFile: Use a specific DNS resolution database.
- -N 10: Use 10 child processes for DNS resolution.
- -o strOutputFolder: Output director.
- objLogFile.Path: The log file to process.
Conclusion
Webalizer is a great tool. Hopefully, either of the above scripts will prove useful for processing IIS log files.