Jul 23, 2016

Elastic Stack: Process IIS Logs

Overview

In this tutorial, I will show you how we can read IIS Logs, process, and send them to Elasticsearch for further analysis. There are many graphs from IIS Logs that give us useful information about our site traffic and performance
  • IIS Average time-taken: shows overall site performance/response time
  • IIS Requests over Time: shows site load
  • IIS Average time-taken per site: shows site performance/response time per cs-host
  • IIS Average time-taken per server: shows site performance/response time per s-computer
  • IIS Response Codes: 200, 301, 403, etc.
More details about IIS Log fields: https://technet.microsoft.com/en-us/library/cc754702(v=ws.10).aspx

We can also parse GeoIP info from client IP and users' devices, OS, and browsers from cs(UserAgent) field.

Some abbreviations:
  • Logstash: LS
  • Elasticsearch: ES
  • Kibana: KB
If you are new to Elastic Stack, you should start with this.

Diagram

Let's start by looking the following diagram:
IIS Log Processing Diagram
There are many tools to read and forward logs in real time, but I prefer nxlog  for its rich features, lightweight, fast, and simplicity. We can use Filebeat to read and ship logs to LS and let LS handle the processing; however, when we are looking at tens of thousands of web requests, or log lines, per second, I think that shifting the processing part to the source of the logs allows us to process faster at a lower resource cost. Typically, I would let LS do as less processing as possible.

Jul 17, 2016

Logstash Config: Check if a field exists or not

There are times when we want to check if a field exists or not before performing an action.

To check if a field named field_1 exists

if [field_1] {
    mutate {}
    do something else
}


To check if a field named field_1 does not exist

if ![field_1] {
    mutate {}
    do something else
}