Log Insight

Log Insight + Netflow = Awesome

This is just another awesome use case for Log Insight. I have setup the vSphere Distributed switch to send netflow to a netflow proxy which then sends them as syslog messages to Log Insight. The reason for this is that Log Insight can’t ingest netflow messages natively.

What you then end up with is a lot of logged netflow messages like the one below. Ignore the extracted field names, these can be designed as you wish. I went with speed 🙂


You might think what can this be used for and why should I care ?

Log Insight + Netflow = Awesome

I have created a dashboard and some widgets to give you an idea of what can be done with this kind of information.


First of is traffic flows, which VMs talk a lot together and how much data does they transfer. Depend on how your network environment is setup this might call for vMotioning some VMs together to limit north/south network traffic flows and keep it all local in the host. We also get to see which protocols are being used and how much traffic each protocol is responsible for.


What about VMs which communicates over insecure protocols ?


What about data transferred ?


The possibilities are almost limitless.

Another use case could be to see what VMs a VM talks to. Maybe create an application group in vRops based on the actual network traffic flow instead of guessing.

What about adding netflow from other network devices and firewall traffic ? Now you start to be able to map what is going on in your entire network. So you can start acting on it!

This is where the power of Log Insight comes in.

Continue reading part 2 where configuration files, scripts and content pack is available.


By the way I’m not the only one who sees this. See this Youtube video, which recently appeared from VMware on Microsegmentation with NSX and Log Insight. https://www.youtube.com/watch?v=9aRMEx4cB34

Log Insight is used to show the traffic flow between virtual machines in order to create the microsegmentation with NSX, pretty cool.

10 thoughts on “Log Insight + Netflow = Awesome

  1. Great idea, Michael. You should consider sharing your dashboards as a content pack. But for the use case of finding what VM another VM talks to, that’s exactly what Infrastructure Navigator does, and it does it extremely well. That is another tool, in addition to Log Insight, that enough people aren’t using.

    1. Hi Chip

      Thanks for your comments. Haven’t shared the dashboard as this is still very much working progress… But in time I will share it all.

      I somewhat disagree with you on Infrastructure Navigator, I love the idea of Infrastructure Navigator, but the product has a few things against it.
      1.) It’s agent based
      2.) Netstat is only run hourly
      3.) Adding to an Application created by Infrastructure Navigator is not easy

      A real world problem I had, was a customer having a VM as a PDF render at will. This never got discovered because it ran for a very short time each time I got asked to render a PDF.
      This problem could be solve by using Log Insight. A problem with Log Insight would then be I doesn’t know which application holds the connection, making it more guess work to which app the connection was open.

      In the best of worlds VMware Tools(or Infrastructure Navigator) could be used to provide netstat infomation and Log Insight could deliver netflow infomation and mapping these two together, should make up a strong way of mapping applications and there real world behavior.

      Once again thanks for read and commenting.

    1. Hi Yves

      Yes, no problem. Only reason that I haven’t disclosed this, is that I still somewhat a working progress the way I have implemented it. But the tools used are open source NFDUMP. nfcapd to capture the netflow and nfdump which reads nfcapd files and saves them as text files, that the Log Insight agent ingests. Not the most pretty solution and still have some thing to work out like timestamps, which currently is based on the ingestion time, which isn’t good enough.

      This should be the installation procedure I used(if I documented my work good enough)
      apt-get install gcc flex librrd-dev make

      cd /usr/local/src/

      wget https://sourceforge.net/projects/nfdump/files/stable/nfdump-1.6.13/nfdump-1.6.13.tar.gz
      gzip -dc nfdump-1.6.13.tar.gz | tar -xf -
      cd nfdump-1.6.13

      ./configure --enable-nsel --enable-sflow
      make install

      This is the daemon I created
      # Provides: nfcapd
      # Required-Start: $local_fs $network
      # Required-Stop: $local_fs
      # Default-Start: '2 '3 '4 '5
      # Default-Stop: '0 '1 '6
      # Short-Description: nfcapd daemon
      # Description: nfcapd daemon
      ### END INIT INFO

      start() {
      echo -n "Starting : NetFlow Capture Daemon"
      if [ -f /var/run/nfcapd.pid ]; then
      PID=$(cat /var/run/nfcapd.pid)
      echo NetFlow Capture Daemon already running: $PID
      exit 2;

      /usr/local/bin/nfcapd -z -w -D -T all -l /tmp/ -I any -t '60 -P /var/run/nfcapd.pid
      sleep '1
      PID=$(cat /var/run/nfcapd.pid)
      echo NetFlow Capture Daemon is started: $PID

      stop() {
      echo -n "Shutting down : NetFlow Capture Daemon"

      kill $(cat /var/run/nfcapd.pid)

      if [ -f /var/run/nfcapd.pid ]; then
      PID=$(cat /var/run/nfcapd.pid)
      echo NetFlow Capture Daemon is running: $PID
      exit 2;

      echo NetFlow Capture Daemon is stopped

      case "$1" in
      sleep '1
      echo "Usage: {start|stop|status|restart]"
      exit '1
      exit $?

      This is a crontab clean up script that i run every minute – DNS is there to IP to DNS resolution.


      for OUTPUT in $(ls /tmp/nfcapd.2*)
      DATE=$(date +"%Y%m%d%H%M%S%N")
      nfdump -r $OUTPUT -D $DNS -o csv | grep '20 > $INPUT.$DATE
      rm $OUTPUT

      find /var/log/netflow* -mmin +15 -exec rm {} \;

      For some unknown reason at least to me, vimeo videos gets embedded every time a number is standing alone in my comment. So please remove the “‘” before the numbers in order to make the scripts work.

      Like I said work in progress… Thanks for reading and commenting.

  2. Thanks Michael ,it work’s perfectly in my environment .Can we measure traffic in Kbs instead of bytes.

    1. Sorry no… LI doesn’t support mathatical operation. Go to loginsight.vmware.com and make a surgestion to at the feature, this is my best advise 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *