Elastic Stack: Process IIS Logs

Overview

In this tutorial, I will show you how we can read IIS Logs, process, and send them to Elasticsearch for further analysis. There are many graphs from IIS Logs that give us useful information about our site traffic and performance

• IIS Average time-taken: shows overall site performance/response time
• IIS Requests over Time: shows site load
• IIS Average time-taken per site: shows site performance/response time per cs-host
• IIS Average time-taken per server: shows site performance/response time per s-computer
• IIS Response Codes: 200, 301, 403, etc.

More details about IIS Log fields: https://technet.microsoft.com/en-us/library/cc754702(v=ws.10).aspx

We can also parse GeoIP info from client IP and users' devices, OS, and browsers from cs(UserAgent) field.

Some abbreviations:

• Logstash: LS
• Elasticsearch: ES
• Kibana: KB

If you are new to Elastic Stack, you should start with this.

Diagram

Let's start by looking the following diagram:

IIS Log Processing Diagram

There are many tools to read and forward logs in real time, but I prefer nxlog  for its rich features, lightweight, fast, and simplicity. We can use Filebeat to read and ship logs to LS and let LS handle the processing; however, when we are looking at tens of thousands of web requests, or log lines, per second, I think that shifting the processing part to the source of the logs allows us to process faster at a lower resource cost. Typically, I would let LS do as less processing as possible.

Enable HAProxy logging on CentOS

By default, HAProxy will not log to files unless we make some modifications
1. Create rsyslog configuration file
nano /etc/rsyslog/haproxy.conf
Add these lines to the file
# Enable UDP port 514 to listen to incoming log messages from haproxy
$ModLoad imudp
$UDPServerRun 514
$template Haproxy,"%msg%\n"
local0.=info -/var/log/haproxy/haproxy.log;Haproxy
local0.notice -/var/log/haproxy/admin.log;Haproxy
# don't log anywhere else
local0.* ~
Restart rsyslog service
/etc/init.d/rsyslog restart
Ref: http://blog.hintcafe.com/post/33689067443/haproxy-logging-with-rsyslog-on-linux
2. Modify the log rotate config to match the new folder:
nano /etc/logrotate.d/haproxy
Change
/var/log/haproxy.log {
    daily
    rotate 10
    missingok
[...]
to
/var/log/haproxy/*.log {
    daily
    rotate 10
    missingok
[...]
Now we can check if HAProxy logging is working.
tail -f /var/log/haproxy/haproxy.log

Log client's IP address in apache log when using HAProxy and ISPConfig

If we use HAProxy and ISPConfig to publish websites, by default, Apache log will log only the IP of the HAproxy server. To log client's IP in Apache log, we have to:

1. Config HAProxy
Add
option forwardfor to backend option in HAProxy config file, then reload haproxy
service haproxy reload
2. Change the LogFormat for ISPConfig site
Edit ispconfig config file nano /etc/httpd/conf/sites-available/ispconfig.conf Replace LogFormat "%v %h %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\"" combined_ispconfig with LogFormat "%v %{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\"" combined_ispconfig
3. Change the LogFormat for httpd
Edit httpd.conf file nano /etc/httpd/conf/httpd.conf Replace LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined with LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined Save the file and restart apache server service httpd restart
Check the log again. Not as other guides, I still keep the %h because we sometimes need to test the webserver directly. Also, we want to log if there is any other IP accessing our webserver besides the HAProxy.