Process NetFlow with nProbe and Elasticsearch, Logstash, and Kibana - Part 1

Install Elasticsearch, Logstash, and Kibana on Windows Server 2012 R2

Overview

Source: https://en.wikipedia.org/wiki/NetFlow

By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:

• Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors.

• Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter.

• Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example.

In this tutorial, we will use:

• NetFlow generator (https://www.paessler.com/tools/netflowgenerator) as flow exporter
• nProbe (http://www.ntop.org/products/netflow/nprobe/) as flow collector
• Elasticsearch + Logstash + Kibana (ELK https://www.elastic.co) to receive, store, analyze, and display Netflow data

System Diagram

The diagram above shows how Netflow data are processed.

A simple network diagram is created for this tutorial

Network Diagram

ELK and nProbe will be installed on 192.168.1.50, and sample NetFlow data will be generated from 192.168.1.60.

Let's start by setting up an ELK stack on Windows Server 2012 R2

Mobile devices - Connected or Disconnected?

http://www.removed.social/

Disable multi location access log in ISPConfig setup

This applies to ISPConfig 3.0.5.3 on Debian Wheezy 7.3.

By default, ISPConfig will save access log for each site at /var/log/ispconfig/httpd/<sitename>/access.log, and Apache also saves log for each vhost at /var/log/apache2/other_vhosts_access.log. If we have many sites hosted on the same server (of course that’s why we use ISPConfig), we do not want the same access log in 2 different locations which consumes server resources.

We can disable vhost access log of Apache by editing the config file
nano /etc/apache2/conf.d/other-vhosts-access-logthen comment out the line
#CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined Restart apache service apache2 restart

Change Network location from Public to Private in Windows 8.1

If we are using Windows 8 or 8.1 at home or at work, and others cannot access shared folders on our PCs, then maybe our network profile is set to Public as default. If we are lucky, follow this post can help us change the profile to Private to enable file sharing http://community.spiceworks.com/how_to/show/18934-change-network-location-from-public-to-private-in-windows-8.

However, on my Windows 8 or 8.1 PCs, I cannot right click on the Network icon to change the profile as instructed, so there is another way.

mpt-statusd: detected non-optimal RAID status

If we install Debian Wheezy 7.3 on VMware, we may get a lot of errors in /var/log/messages like:
mpt-statusd: detected non-optimal RAID status Because there is no RAID device within a VM, unless you setup RAID, we can disable the mpt-statusd
service mpt-statusd stop
update-rc.d mpt-statusd remove

How to install GlusterFS 3.4.x server and client on Debian Wheezy 7.3

1. Install glusterFS 3.4.x

Add the GPG key to apt
wget -O - http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.2/Debian/pubkey.gpg | apt-key add - Add the source and update package list
echo deb http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.2/Debian/apt wheezy main > /etc/apt/sources.list.d/gluster.list

apt-get update Install gluster server and client apt-get install glusterfs-server glusterfs-client Ref: http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.2/Debian/README

2. Dealing with mounting issue at boot time

With GlusterFS 3.4.2, we may get in trouble trying to mount the gluster volume via /etc/fstab
<server-ip>:/gluster_volume /var/mount glusterfs defaults,_netdev 0 0 This may not mount the volume as we expect in previous version of glusterFS. Therefore, we have to use a script to mount our glusterFS volume at boot time.

How to apply for an Individual Taxpayer ID Number (ITIN) when filing tax

If you do not yet have an ITIN, you can include the application in your tax filing. Follow these steps to apply for an ITIN and file your tax return:

1. Complete IRS Form W-7.
2. Complete your federal and state tax returns using the number 999-88-9999 in place of a Social Security Number.
3. Print the tax returns and cross off the 999-88-9999 from the SSN field. Do not attempt to e-file your return.
4. Write "Applying for ITIN; Application attached" next to the SSN field.
5. Mail the signed federal return, the completed Form W-7, and all documentation required by the Form W-7 Instructions to the address specified in the Form W-7 Instructions. (Alternatively, you may choose to take all this information in-person to a local IRS office for processing.)
6. If you need to file a state tax return: Mail the signed state return and a copy of Form W-7 to your state's regular filing address.

Source: Turbotax
The free e-file from TurboTax is easy to use with a nice design.