During a POC of Log Insight which has been long running, features of Log Insight have evolved. From a single node, to clustered nodes and now to have an internal loadbalancer. As a lot of devices are logging to the ip address of the first node, which has evoled into the master of a cluster. It seemed easiere to just change the ip address of the master node and set the master node’s old IP address to that of the ILB, than to change the syslog setting on all devices which are logging to Log Insight (Note if you only have esxi hosts, it would be simplere to just change the ip to which the ESXi hosts send logs).

 

!!!THIS IS NOT SUPPORTED BY VMWARE TO MY KNOWLEDGE. PROCEED AT YOUR OWN RISK!!!

 

Solution Overview

The steps are as following:

1. Changing IP address of Log Insight Master Node
2. Correct the config setting of the master
3. Correct the config settings of any workers

 

Changing IP address of Log Insight Master Node

The steps are pretty simple, as Steven Flanders (Twitter/Web) points out in his answer on VMTN

This is a copy/paste of his anwser, which is also in the release notes – See source for full details:

• You cannot change the network properties of the Log Insight virtual appliance at run time
  Log Insight does not support changing the IP address, DNS, Gateway, or Hostname of the virtual appliance at run time.
  Workaround: You must change them in vCenter vApp properties and perform a full VM shut down and then a power on.

1. Open a vSphere Client and locate the Log Insight virtual appliance.
2. Shut down the guest operating system of Log Insight and power off the virtual appliance.
3. Right-click the virtual appliance, select Edit Settings, and apply the changes to network properties.
4. Power on the virtual appliance.
   Note: You must re-enable launch in context each time you change the network properties of the Log Insight virtual appliance.

Source

 

 

Changing Master nodes settings

 

I assume you have done step one and that you know both the old and the new ip of the Master node.

 

1. You need to ssh to the Master node; Login as root with the password you original created when Log Insight Master Node were deployed.
2. Verify the ip address has change of the node by running the command: ifconfig

If it’s still the same reboot and verify again

3. Next we need to verify that Log Insight is NOT running by running this command:  /etc/init.d/loginsight status

You should see the status “daemonNotRunning”

4. Now change dir to the location of the Log Insight config files: cd /storage/core/loginsight/config/
5. Do in “ls” and you should see files with the same nameing standard – Note the number behind the hashtag (#) could be different.

 

 

File explanation

•  krb5.conf – Contains details about domain realm
• loginsight-config.xml# – This is the config files containing the config of the loginsight node and relationships. The highest number is the newest and the most recent config.
• node-token – Contains the nodes token string. This is also visable in the loginsight-config.xml

The loginsight-config.xml file will only contain the parts you have configured via the GUI, so most configurations will be different

 

6. Use “vi” to open the lastest version of the loginsight-config.xml in this example: vi loginsight-config.xml#46

Press the “i” or “Insert” button on the keyboard and change all the places where you see the old IP address, to the new ip address. Lastly quit and save by hitting “ESC” and then “:” afterwards type “wq” and hit “Enter”

7. So the config of the Master should be correct now. Try rebooting the node, and then verify if log insight is running, use the command: /etc/init.d/loginsight status.

If the status show “Log Insight is running” – Go to “Changing worker nodes settings” chapter, else continue with next object in the list

 

8. First we need to verify why the log insight note is not running, use the command:  tail -n 50 /storage/core/loginsight/var/runtime.log

To get the last 50 lines from the log file

9. If you see this message “StartupFailedException: Daemon startup failed: Failed to start Cassandra Server” the problem is the Cassandra db and you should look the this KB

 

Command to help troubleshoot:

tail -n 50 /storage/core/loginsight/var/runtime.log – This shows the last 50 lines in the runtime.log file; If you want a live tail of the log use the “-f”

/etc/init.d/loginsight status – Shows the status of the log insight daemon

/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool status – This shows the nodes and status, found by the Master as configured in the loginsight-config.xml file. Be award that you wont see the other nodes before you have changed there config settings aswell.

/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool stopdaemon -This stops the Cassandra Daemon

 

10. Reboot and verify that the log insight daemon is running/starting by it self.

 

Changing worker nodes settings

 

This simpler version of what we just did above – Remember you need to do these steps on all workers in your cluster.

 

1. SSH in to a worker.
2. cd /storage/core/loginsight/config/
3. vi loginsight-config.xml# (remember to change the newest version of the file.) – Change the ip address of the Master node and save the file again,
4. Reboot and verify that the loginsight daemon is running.
5. Run the command: /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool status

Verify that both Master and worker is in the list and that the lines start with “UN”.

The first letter is the “Status” it can be “UP” or “Down” (U or D) and the next letter is the “State” which can be “Normal”,”Leaving”,”Joining” or “Moving” (N,L,J or M)

 

 If it didn’t go as planned for some reason try the following:

Nodes not in the list wait 10 min and run the command again.

Nodes still not in the list try rebooting – Verify the loginsight daemon is running.

Loginsight daemon not running – Check the runtime.log file

Loginsight daemon not running – Check Cassandra is running – nodetool status command will not output if Cassandra is not running.

Cassandra not running – Look at the KB article

 

Last option, revert the config back and reboot the VMs.

 

If everything worked as planned you are now ready to setup the Internal LoadBalancer.