10 thoughts on “Log Insight + Netflow = Awesome”

1. Great idea, Michael. You should consider sharing your dashboards as a content pack. But for the use case of finding what VM another VM talks to, that’s exactly what Infrastructure Navigator does, and it does it extremely well. That is another tool, in addition to Log Insight, that enough people aren’t using.

   Reply

   1. Hi Chip

      Thanks for your comments. Haven’t shared the dashboard as this is still very much working progress… But in time I will share it all.

      I somewhat disagree with you on Infrastructure Navigator, I love the idea of Infrastructure Navigator, but the product has a few things against it.
      1.) It’s agent based
      2.) Netstat is only run hourly
      3.) Adding to an Application created by Infrastructure Navigator is not easy

      A real world problem I had, was a customer having a VM as a PDF render at will. This never got discovered because it ran for a very short time each time I got asked to render a PDF.
      This problem could be solve by using Log Insight. A problem with Log Insight would then be I doesn’t know which application holds the connection, making it more guess work to which app the connection was open.

      In the best of worlds VMware Tools(or Infrastructure Navigator) could be used to provide netstat infomation and Log Insight could deliver netflow infomation and mapping these two together, should make up a strong way of mapping applications and there real world behavior.

      Once again thanks for read and commenting.

      Reply

2. Hi Michael,

   would you mind sharing which proxy you use? Would like to play around with this a bit…

   Thanks

   Yves

   Reply

   1. Hi Yves

      Yes, no problem. Only reason that I haven’t disclosed this, is that I still somewhat a working progress the way I have implemented it. But the tools used are open source NFDUMP. nfcapd to capture the netflow and nfdump which reads nfcapd files and saves them as text files, that the Log Insight agent ingests. Not the most pretty solution and still have some thing to work out like timestamps, which currently is based on the ingestion time, which isn’t good enough.

      This should be the installation procedure I used(if I documented my work good enough)
      #Install
apt-get install gcc flex librrd-dev make

      cd /usr/local/src/

      wget https://sourceforge.net/projects/nfdump/files/stable/nfdump-1.6.13/nfdump-1.6.13.tar.gz
      gzip -dc nfdump-1.6.13.tar.gz | tar -xf -
      cd nfdump-1.6.13

      ./configure --enable-nsel --enable-sflow
make
make install

      This is the daemon I created
      #!/bin/bash
#
#/etc/rc.d/init.d/nfcapd
#
### BEGIN INIT INFO
# Provides: nfcapd
# Required-Start: $local_fs $network
# Required-Stop: $local_fs
# Default-Start: '2 '3 '4 '5
# Default-Stop: '0 '1 '6
# Short-Description: nfcapd daemon
# Description: nfcapd daemon
### END INIT INFO
#
#
#
#

      start() {
      echo -n "Starting : NetFlow Capture Daemon"
      echo
      if [ -f /var/run/nfcapd.pid ]; then
      PID=$(cat /var/run/nfcapd.pid)
      echo NetFlow Capture Daemon already running: $PID
      exit 2;

      else
      /usr/local/bin/nfcapd -z -w -D -T all -l /tmp/ -I any -t '60 -P /var/run/nfcapd.pid
      sleep '1
      PID=$(cat /var/run/nfcapd.pid)
      echo NetFlow Capture Daemon is started: $PID
      return
      fi
      }

      stop() {
      echo -n "Shutting down : NetFlow Capture Daemon"
      echo

      kill $(cat /var/run/nfcapd.pid)
      return
      }

      status(){
      if [ -f /var/run/nfcapd.pid ]; then
      PID=$(cat /var/run/nfcapd.pid)
      echo NetFlow Capture Daemon is running: $PID
      exit 2;

      else
      echo NetFlow Capture Daemon is stopped
      return
      fi
      }

      case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
restart)
stop
sleep '1
start
;;
*)
echo "Usage: {start|stop|status|restart]"
exit '1
;;
esac
exit $?

      This is a crontab clean up script that i run every minute – DNS is there to IP to DNS resolution.
      #
#/etc/cron.d/netflow-clean.sh
#

      DNS="10.14.160.9"
      INPUT="/var/log/netflow"

      for OUTPUT in $(ls /tmp/nfcapd.2*)
      do
      DATE=$(date +"%Y%m%d%H%M%S%N")
      nfdump -r $OUTPUT -D $DNS -o csv | grep '20 > $INPUT.$DATE
      rm $OUTPUT
      done

      find /var/log/netflow* -mmin +15 -exec rm {} \;


      For some unknown reason at least to me, vimeo videos gets embedded every time a number is standing alone in my comment. So please remove the “‘” before the numbers in order to make the scripts work.

      Like I said work in progress… Thanks for reading and commenting.

      Reply

3. Any Update

   Reply

   1. What do you need an update on ?

      Reply

4. Customize dashboard or content pak for Loginsight which will show in Loginsight.

   Reply

   1. Hi

      I have put everything on github – https://github.com/MichaelRyom/LogInsight-Netflow

      Another blog post will come when in done writing but the above link should get you started.

      Reply

5. Thanks Michael ,it work’s perfectly in my environment .Can we measure traffic in Kbs instead of bytes.

   Reply

   1. Sorry no… LI doesn’t support mathatical operation. Go to loginsight.vmware.com and make a surgestion to at the feature, this is my best advise 🙂

      Reply