Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 1

Install Elasticsearch, Logstash, and Kibana on Windows Server 2012 R2

Overview

Source: https://en.wikipedia.org/wiki/NetFlow

By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:

• Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors.

• Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter.

• Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example.

In this tutorial, we will use:

• NetFlow generator (https://www.paessler.com/tools/netflowgenerator) as flow exporter
• nProbe (http://www.ntop.org/products/netflow/nprobe/) as flow collector
• Elasticsearch + Logstash + Kibana (ELK https://www.elastic.co) to receive, store, analyze, and display Netflow data

System Diagram

The diagram above shows how Netflow data are processed.

A simple network diagram is created for this tutorial

Network Diagram

ELK and nProbe will be installed on 192.168.1.50, and sample NetFlow data will be generated from 192.168.1.60.

Let's start by setting up an ELK stack on Windows Server 2012 R2

Ref: https://www.ulyaoth.net/resources/tutorial-install-logstash-and-kibana-on-a-windows-server.34/. I eliminate some extra steps to simplify the tutorial.

Install Prerequisites

1. Download and install latest JRE from Oracle (currently jre-8u66-windows-x64.exe). You may install either JDK or JRE.
2. Add JAVA_HOME environment variable. Open System Properties

Create a new system variable

Set JRE folder path, change this path if you update JRE later

Install ELK

1. Download ELK for Windows from https://www.elastic.co
2. Create a folder for ELK at C:\ELK, extract all ELK packages

3. Open Command Prompt and change to elasticsearch\bin folder (Tip: You can use Shift + Right Click on a folder to open Command Prompt at that folder)

To install Elasticsearch, run
service install

If you plan to install more than one Elasticsearch instance on the same server, edit the file elasticsearch\bin\service.bat and change
set SERVICE_ID=elasticsearch-service-x64
to
set SERVICE_ID=elasticsearch-service-x64-<custom-name>
Open Elasticsearch service manager, start Elasticsearch service
service manager

Update path to new JRE if necessary

Start service

Open a web browser, go to http://localhost:9200. We should see as the screenshot below

4. NSSM (http://nssm.cc) is used to install and run both Logstash and Kibana as a Windows service

Download

http://nssm.cc/release/nssm-2.24.zip

Extract and copy file nssm.exe from nssm-2.24\win64 to

C:\ELK\logstash\bin
C:\ELK\kibana\bin

5. To install Kibana, open Command Prompt at C:\ELK\kibana\bin and run

nssm install kibana

Locate kibana.bat

Install service

Go to http://localhost:5601 to see if Kibana is working

6. To install Logstash, 

Go to C:\ELK\Logstash\bin, create a Logstash config file named logstash.conf and paste the following lines

input {

tcp {

port => 5544

  type => "netflow"

}

}

filter {

json {

source => "message"

}

}

output {

stdout { 

codec => rubydebug 

}

elasticsearch {

hosts => ["localhost:9200"]

index => "netflow-%{+YYYY.MM.dd}"

}

}

Create a run.bat file in the same folder and paste:

logstash.bat agent -f logstash.conf

Open Command Prompt at C:\ELK\logstash\bin and run

nssm install logstash

Locate run.bat

Start service

Check if Logstash is listening on port 5544

Summary

Our ELK stack is ready to receive NetFlow and other data. In the next part, we will install nProbe, generate NetFlow data, and play with Kibana.
Part 2: http://blog.sysadmin.live/2015/11/process-netflow-with-nprobe-and_13.html