Elastic Stack: Process IIS Logs

Overview

In this tutorial, I will show you how we can read IIS Logs, process, and send them to Elasticsearch for further analysis. There are many graphs from IIS Logs that give us useful information about our site traffic and performance

• IIS Average time-taken: shows overall site performance/response time
• IIS Requests over Time: shows site load
• IIS Average time-taken per site: shows site performance/response time per cs-host
• IIS Average time-taken per server: shows site performance/response time per s-computer
• IIS Response Codes: 200, 301, 403, etc.

More details about IIS Log fields: https://technet.microsoft.com/en-us/library/cc754702(v=ws.10).aspx

We can also parse GeoIP info from client IP and users' devices, OS, and browsers from cs(UserAgent) field.

Some abbreviations:

• Logstash: LS
• Elasticsearch: ES
• Kibana: KB

If you are new to Elastic Stack, you should start with this.

Diagram

Let's start by looking the following diagram:

IIS Log Processing Diagram

There are many tools to read and forward logs in real time, but I prefer nxlog  for its rich features, lightweight, fast, and simplicity. We can use Filebeat to read and ship logs to LS and let LS handle the processing; however, when we are looking at tens of thousands of web requests, or log lines, per second, I think that shifting the processing part to the source of the logs allows us to process faster at a lower resource cost. Typically, I would let LS do as less processing as possible.

Logstash Config: Check if a field exists or not

There are times when we want to check if a field exists or not before performing an action.

To check if a field named field_1 exists

if [field_1] {
    mutate {}
    do something else
}


To check if a field named field_1 does not exist

if ![field_1] {
    mutate {}
    do something else
}

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 4

Part 1: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and.html
Part 2: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and_13.html
Part 3: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and_91.html

Map User Location within ELK stack

Install Sense on Kibana

Before we create GeoIP fields into Elasticsearch (ES), let's install Sense on Kibana so that we have a great UI to interact with Elasticsearch instead of using curl.

Open a Command Prompt and go to

C:\ELK\kibana\bin

Run

kibana plugin --install elastic/sense

Restart Kibana service and open Kibana.

Sense UI

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 3

Part 1: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and.html
Part 2: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and_13.html

Customize nProbe and Logstash configuration

Overview

In the previous part, we have created a basic visualization and a dashboard in Kibana for NetFlow data; however, do we really need all NetFlow fields? NetFlow v9 has more than 50 field types, so it is better if we export only meaningful fields.

As Logstash received NetFlow fields from nProbe, we can configure it to process those data and add more fields or tags to analyze our network traffic better

Configure nProbe to export only significant NetFlow fields

We can run nProbe with option -T followed by a template to export only the fields we are interested in. The following template is a good start
-T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %SRC_MASK %DST_MASK %IN_SRC_MAC %OUT_DST_MAC %L7_PROTO_NAME %PROTOCOL_MAP %PROTOCOL"
Note: %IN_SRC_MAC %OUT_DST_MAC only shows data when we run nProbe with a mirrored port.
We can start nProbe with a template by running
nprobe.exe /c -b 1 -V 9 --collector-port 2055 -i none -n none --json-label --tcp 127.0.0.1:5544 -T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %SRC_MASK %DST_MASK %IN_SRC_MAC %OUT_DST_MAC %L7_PROTO_NAME %PROTOCOL_MAP %PROTOCOL"
We should now see those fields in Kibana Discover

New NetFlow fields

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 2

Send NetFlow data to ELK, create searches, visualizations, and dashboards in Kibana
If you haven't read part 1, please visit http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and.html

In this part, we will try to create a nice dashboard in Kibana as below

A view of traffic

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 1

Install Elasticsearch, Logstash, and Kibana on Windows Server 2012 R2

Overview

Source: https://en.wikipedia.org/wiki/NetFlow

By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:

• Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors.

• Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter.

• Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example.

In this tutorial, we will use:

• NetFlow generator (https://www.paessler.com/tools/netflowgenerator) as flow exporter
• nProbe (http://www.ntop.org/products/netflow/nprobe/) as flow collector
• Elasticsearch + Logstash + Kibana (ELK https://www.elastic.co) to receive, store, analyze, and display Netflow data

System Diagram

The diagram above shows how Netflow data are processed.

A simple network diagram is created for this tutorial

Network Diagram

ELK and nProbe will be installed on 192.168.1.50, and sample NetFlow data will be generated from 192.168.1.60.

Let's start by setting up an ELK stack on Windows Server 2012 R2

Mobile devices - Connected or Disconnected?

http://www.removed.social/