Elastic Stack: Process IIS Logs

Overview

In this tutorial, I will show you how we can read IIS Logs, process, and send them to Elasticsearch for further analysis. There are many graphs from IIS Logs that give us useful information about our site traffic and performance

• IIS Average time-taken: shows overall site performance/response time
• IIS Requests over Time: shows site load
• IIS Average time-taken per site: shows site performance/response time per cs-host
• IIS Average time-taken per server: shows site performance/response time per s-computer
• IIS Response Codes: 200, 301, 403, etc.

More details about IIS Log fields: https://technet.microsoft.com/en-us/library/cc754702(v=ws.10).aspx

We can also parse GeoIP info from client IP and users' devices, OS, and browsers from cs(UserAgent) field.

Some abbreviations:

• Logstash: LS
• Elasticsearch: ES
• Kibana: KB

If you are new to Elastic Stack, you should start with this.

Diagram

Let's start by looking the following diagram:

IIS Log Processing Diagram

There are many tools to read and forward logs in real time, but I prefer nxlog  for its rich features, lightweight, fast, and simplicity. We can use Filebeat to read and ship logs to LS and let LS handle the processing; however, when we are looking at tens of thousands of web requests, or log lines, per second, I think that shifting the processing part to the source of the logs allows us to process faster at a lower resource cost. Typically, I would let LS do as less processing as possible.

Configurations

nxlog

nxlog can be downloaded for free from its website. The latest version is

https://nxlog.co/system/files/products/files/1/nxlog-ce-2.9.1716.msi

Just install nxlog and overwrite c:\Program Files (x86)\nxlog\conf\nxlog.conf with the following configuration
https://gist.github.com/anhlqn/60bb3dc3134e141eafa718add195336b#file-iislog-nxlog-conf
There are 3 main parts in an nxlog config:

• input: where to read logs (file, tcp, udp, command output, etc.)
• output: where to send logs (file, tcp, udp, etc.)
• route: route any input to any output (we can route multiple inputs to multiple inputs at the same time)

Let me explain key parts of this config file.

This block rotates nxlog internal logs as specified in the schedule

    Module      xm_fileop

    # Check the size of our log file every x time and rotate if it is larger than x MB
   
    Every   1 min
    Exec    if (file_size('%NXLOGFILE%') >= 20M) file_cycle('%NXLOGFILE%', 5);
   
# Rotate our log file every week on sunday at midnight
   
    When @weekly
    Exec    file_cycle('%NXLOGFILE%', 5);

This contains the field names that we will send to LS. You will need to check all fields in IIS Logs in order for nxlog CSV module to parse correctly. Feel free to use the field names you want, but do not user the dash "-" because nxlog does not allow "-" in field names (LS and ES do not care).

    Module xm_csv
    Fields $date, $time, $s_sitename, $s_computername, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs_version, $cs_UserAgent, $cs_Cookie, $cs_Referer, $cs_host, $sc_status, $sc_substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
    Delimiter ' '
    QuoteChar   '"'
    EscapeControl FALSE
    UndefValue -
date and time fields are concatenated into GMTTime before sending to LS. We also want to lowercase $s_computername and $cs_uri_stem and delete some unnecessary fields. Finally, CSV data are converted to JSON with to_json().
    Module    im_file
    File    'c:\inetpub\logs\LogFiles\\*ex*.log'
    ReadFromLast False
    SavePos True
Recursive True
   
if $raw_event =~ /^#/ drop();
else {
w3c->parse_csv();
$GMTTime = $date + " " + $time;
$type = "iislog";
}
$s_computername = lc($s_computername);
$cs_uri_stem = lc($cs_uri_stem);
delete($cs_Cookie);
delete($sc_substatus);
delete($sc_win32_status);
delete($EventReceivedTime);
delete($SourceModuleName);
delete($SourceModuleType);
delete($date);
delete($time);
to_json();

Note: There must be two \\ in
File    'c:\inetpub\logs\LogFiles\\*ex*.log'
Change Host and Port to match your LS server config

 

    Module  om_tcp
    Host    192.168.1.10
    Port    5544
    OutputType  LineBased
This tells nxlog to route all logs it gets from in_iislog to out_logstash

    Path in_iislog => out_logstash
After we overwrite the configuration file, just start nxlog service with
sc start nxlog

Logstash

Since we have nxlog handle most of the processing, LS configuration is much simpler. You can get the config file here. I assume that you are familiar with LS configuration, but feel free to leave comments if you need any clarification.

Elasticsearch

You should have known a bit about ES mapping templates before trying to index data, so create a new template for our new IIS Log

Summary

We've had all necessary configurations ready, it's now time to throw some IIS Log files into the configured folder or start browsing our websites to generate some logs. Next step is to create visualizations in Kibana to analyze our logs. If you are not used to Kibana, this series is a good place to start.