Overview
In this tutorial, I will show you how we can read IIS Logs, process, and send them to Elasticsearch for further analysis. There are many graphs from IIS Logs that give us useful information about our site traffic and performance- IIS Average time-taken: shows overall site performance/response time
- IIS Requests over Time: shows site load
- IIS Average time-taken per site: shows site performance/response time per cs-host
- IIS Average time-taken per server: shows site performance/response time per s-computer
- IIS Response Codes: 200, 301, 403, etc.
More details about IIS Log fields: https://technet.microsoft.com/en-us/library/cc754702(v=ws.10).aspx
We can also parse GeoIP info from client IP and users' devices, OS, and browsers from cs(UserAgent) field.
Some abbreviations:
- Logstash: LS
- Elasticsearch: ES
- Kibana: KB
If you are new to Elastic Stack, you should start with this.
Diagram
Let's start by looking the following diagram:
 |
| IIS Log Processing Diagram |
There are many tools to read and forward logs in real time, but I prefer nxlog for its rich features, lightweight, fast, and simplicity. We can use Filebeat to read and ship logs to LS and let LS handle the processing; however, when we are looking at tens of thousands of web requests, or log lines, per second, I think that shifting the processing part to the source of the logs allows us to process faster at a lower resource cost. Typically, I would let LS do as less processing as possible.
