Wednesday, March 26, 2014

Log Parsing

I love log files, they have so much value.
I work with IIS sites a lot and IIS log files are wonderful things. I also enjoy looking at Squid, Apache, and ELB logs. Really, I enjoy it.

I have two tools I especially love for IIS log file parsing.
This Log Parser is amazing; it gives you SQL syntax for quering text files including IIS log files or more generally CSV files. I think it does more but those two are the ones I'm most often dealing with. It's really fast at processing too.
The other tool I love for taking a peek at IIS log files is Microsoft Excel. Excel is a wonderful bit of software and doesn't mind chewing on large log files. I import them as data from a text file, space delimited, and then format the whole thing as a table. This gives me filtering and sorting on the columns which allows me to answer questions like 'which request with a response code in the 200s takes the longest', 'which requests are 404-ing most often'.

Alongside graphing solutions (I'm a fan of Graphite) parsing the log files is powerful; the graph will show some weirdness such as an increase in average response time, or a large number of errors and the log files will give you the details. Hopefully you'll find an answer to the deviant behaviour there.
I've found this tutorial to be an excellent introduction to Graphite and it has some tutorials on setting it up and configuring it.

In general I find that graphing solutions will give you a question, the logs will give you the answer.

No comments: