Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 4

Part 1: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and.html
Part 2: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and_13.html
Part 3: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and_91.html

Map User Location within ELK stack

Install Sense on Kibana

Before we create GeoIP fields into Elasticsearch (ES), let's install Sense on Kibana so that we have a great UI to interact with Elasticsearch instead of using curl.

Open a Command Prompt and go to

C:\ELK\kibana\bin

Run

kibana plugin --install elastic/sense

Restart Kibana service and open Kibana.

Sense UI

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 3

Part 1: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and.html
Part 2: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and_13.html

Customize nProbe and Logstash configuration

Overview

In the previous part, we have created a basic visualization and a dashboard in Kibana for NetFlow data; however, do we really need all NetFlow fields? NetFlow v9 has more than 50 field types, so it is better if we export only meaningful fields.

As Logstash received NetFlow fields from nProbe, we can configure it to process those data and add more fields or tags to analyze our network traffic better

Configure nProbe to export only significant NetFlow fields

We can run nProbe with option -T followed by a template to export only the fields we are interested in. The following template is a good start
-T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %SRC_MASK %DST_MASK %IN_SRC_MAC %OUT_DST_MAC %L7_PROTO_NAME %PROTOCOL_MAP %PROTOCOL"
Note: %IN_SRC_MAC %OUT_DST_MAC only shows data when we run nProbe with a mirrored port.
We can start nProbe with a template by running
nprobe.exe /c -b 1 -V 9 --collector-port 2055 -i none -n none --json-label --tcp 127.0.0.1:5544 -T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %SRC_MASK %DST_MASK %IN_SRC_MAC %OUT_DST_MAC %L7_PROTO_NAME %PROTOCOL_MAP %PROTOCOL"
We should now see those fields in Kibana Discover

New NetFlow fields

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 2

Send NetFlow data to ELK, create searches, visualizations, and dashboards in Kibana
If you haven't read part 1, please visit http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and.html

In this part, we will try to create a nice dashboard in Kibana as below

A view of traffic

Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 1

Install Elasticsearch, Logstash, and Kibana on Windows Server 2012 R2

Overview

Source: https://en.wikipedia.org/wiki/NetFlow

By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:

• Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors.

• Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter.

• Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example.

In this tutorial, we will use:

• NetFlow generator (https://www.paessler.com/tools/netflowgenerator) as flow exporter
• nProbe (http://www.ntop.org/products/netflow/nprobe/) as flow collector
• Elasticsearch + Logstash + Kibana (ELK https://www.elastic.co) to receive, store, analyze, and display Netflow data

System Diagram

The diagram above shows how Netflow data are processed.

A simple network diagram is created for this tutorial

Network Diagram

ELK and nProbe will be installed on 192.168.1.50, and sample NetFlow data will be generated from 192.168.1.60.

Let's start by setting up an ELK stack on Windows Server 2012 R2