Script

My failures into patch management

This is not my usual problem solving blog post, this is about my failures into patch management – Meaning that I failed! This is my thoughts and some of the work I have done in order to get a better understanding of the hardware’s firmware and driver versions – Which should have ended up in being a complete way of gaining control over firmware and drivers versions, in order to insure that firmware and driver are aligned across the datacenter.

Going back around four years I was working with around ten ESXi hosts and 300 VMs or so. Going back six month I was working with 100+ hosts and 3000+ VMs and today with my current employer this number is significant higher. Looking at these three examples there is an order for a magnitude in between each example, this is pure coincidental, but it does help to better understand the issue. Looking at patch management for ten hosts, I can honestly say, there were no documented procedure for doing patch management(I’m only talking firmware/driver not ESXi version) of the hosts. The simple reason for this is that managing ten hosts isn’t that big of a problem, it’s quite easy to look up the individual hosts firmware and driver versions. Looking at the next example 100+ hosts, now it isn’t as trivial, you just don’t look up 100+ hosts manually to see if the environment is aligned and in compliance, not to talk about 1000+ or 10.000+ hosts. Now you need a solution to handle this for you or you end up using all your time on reporting and not adding value to your company.

I started looking at way to get the data I needed out of the ESXi hosts – I wanted it to be as simple and generic as possible. Meaning I didn’t want a one off system, it should work on the current hardware and on coming version of the hardware platform. That meant that a lot of command line tools like vsish and esxcli were off limits, as they would only report on what I had chosen to get out off ESXi.

First attempt

The below code doesn’t adhere to the above defined standard as it uses vsish – But for a static environment this could be useful or where only a few firmware/driver stats are needed.

$Details = @()
$VMhosts = Get-vmHost
foreach($vmhost in $vmhosts){
$Report = "" | Select-Object -Property Name,Firmware,Cluster
$login = "root@" + "$vmhost"
$Report.Firmware = plink.exe -auto_store_key_in_cache -ssh $login -pw Password "vsish -e cat /net/pNics/vmnic0/properties | grep 'Driver Firmware'"
$Report.Name = $vmhost.name
$Report.Cluster = get-vmhost $vmhost.name | Get-Cluster
$Details += $Report
}

$Details | Export-Csv -Delimiter ";" c:\temp\Host_firmware.csv -NoTypeInformation

 

Second failure

I skipped the plink attempt to get useful data out and instead tried to go to a pure powercli solution. The pros would be a better way of handling username and password security and also this would be structured data, in that sense that it had been structured for me, as this is how powershell is – lots of arrays of data 🙂 But as hardware platforms differ a lot from vendor to vendor or even hardware versions, this was also abandoned. On a side note not all data I needed was available, in this manner. I could get some of the way, but not all the way and as this was the goal, I moved on.

$myhosts = get-vmhost | get-view
$Details = @()
foreach($myhost in $myhosts){
$Report = "" | Select Name,Vendor,Model,UUID,BootTime,HWbios,HWbiosrls,Hba0,hba0_model,hba0_status,Hba1,hba1_model,hba1_status,VAAI_Status

$Report.vendor = $myhost.Hardware.SystemInfo.Vendor
$Report.model = $myhost.Hardware.SystemInfo.Model
$Report.uuid = $myhost.Hardware.SystemInfo.uuid
$Report.BootTime = $myhost.Runtime.BootTime
$Report.name = ($myhost.name).ToUpper()

$HBA = Get-VMHost $myhost.name | Get-VMHostHba | where {$_.type -eq "FibreChannel"}


$i = 0
do {
$hbaid = "hba$i"
$hbamodel = "hba$i"+"_model"
$hbastatus = "hba$i"+"_status"
$Report.$hbaid = $hba[$i].device
$Report.$hbamodel = $hba[$i].Model
$Report.$hbastatus = $hba[$i].status
$i++
}
until ($i -eq 2)

$Report.HWbios = $myhost.Hardware.BiosInfo.BiosVersion
$Report.HWbiosrls = $myhost.Hardware.BiosInfo.ReleaseDate

if(Get-VMHostModule -vmhost $myhost.name | where {$_.name -eq "ibm_vaaip_module"})
{
$Report.VAAI_Status = "Installed"
}elseif(!(Get-VMHostModule -vmhost $myhost.name | where {$_.name -eq "ibm_vaaip_module"})){
$Report.VAAI_Status = "Not found"
}

$Details += $Report
}

$Details | Export-Csv -Delimiter ";" c:\temp\Host_Hardware.csv -NoTypeInformation

Third one is the charm or not

These different attempt have been done now and then, when ever I had some time for it. At some point I was going through and ESXi support bundle and noticed a file containing a dump of firmware and driver versions. I dug into it a bit more and found that running this tool on the ESXi host will output a complete list.

/usr/lib/vmware/vm-support/bin/swfw.sh

I tried to see if I could get it to work in the same way as in the first example – It work but was unstructured. So I look at the swfw.sh script to see if I could identify where it got its information from, in the hopes that I could get some structured data out of it. This is when I saw that it was getting its data from CIM Providers as it states. That helped me a lot as I knew that PowerShell had Cim Cmdlets. After learning how to use Cim Cmdlets I came up with the below script. The first line actually to me some time get working – and some talk with GSS without luck – before I found the issue, if you want to know more see my previous blog post CIM error.

Then comes the part where the Cim Cmdlets does it magic. This part is pretty straightforward, but look at the lines $HWid, where I try to do a profile for the given vendor and the hardware. Next is a function, which is clearly not done. The last part is some of the other ways I had tried to structure the data. This is where I again got stuck – The reason being that I again want it to be as generic, so instead of me trying to tame the data, I would have liked to be able to do a query against VMware hardware compatibility list, that way I would only have to run through the list of firmware/driver and match it against the HCL, so I wouldn’t be forced to do everything by hand.

Set-Item WSMan:\localhost\MaxEnvelopeSizekb 10240
import-module CimCmdlets
$ipaddress = "Host1","Host2"
$HostUsername = "root"
$CIOpt = New-CimSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck -Encoding Utf8 -UseSsl
$Sessions = New-CimSession -Authentication Basic -Credential $HostUsername -ComputerName $Ipaddress -port 443 -SessionOption $CIOpt
foreach($session in $sessions){
$VMhost = Get-CimInstance -CimSession $Session -ClassName CIM_chassis
$software = Get-CimInstance -CimSession $Session -ClassName CIM_SoftwareIdentity | select ElementName,InstanceID,VersionString,Caption,Manufacturer
$Logical = Get-CimInstance -CimSession $Session -ClassName CIM_EnabledLogicalElement | where {$_.creationclassname -ne $null}
$HWid = @{}
$HWid["1"]=@("HP","ProLiant BL465c G7")
$HWid["2"]=@("HP","ProLiant BL460c Gen8")

Parse ($hwid.GetEnumerator() | where {$_.value[0] -match $vmhost.Manufacturer -and $_.value[1] -match $vmhost.Model}).name

}


Function Parse($Config){
$HWconfig = @()
if($config -eq "1"){
$Report = "" | Select UUID,Vendor,Model,Driver,Firmware,DID,VID,SVID,SSID
$Report.UUID = $VMhost.UUID
$Report.Vendor = $VMhost.UUID
$Report.Model = $VMhost.UUID
$Report.Driver = $VMhost.UUID
$Report.Firmware = $VMhost.UUID
$Report.VID = $VMhost.UUID
$Report.DID = $VMhost.UUID
$Report.SVID = $VMhost.UUID
$Report.SSID = $VMhost.UUID
}elseif($config -eq "2"){

}

}




$software | select Description,InstanceID,Manufacturer,VersionString
$Logical | where {$_.SystemCreationClassName -eq "OMC_UnitaryComputerSystem"}
#DEC to Hex
[String]::Format("{0:x}", )
#VID
[String]::Format("{0:x}", $Logical.VendorID)
#DID
[String]::Format("{0:x}", $Logical.PCIDeviceID)
#SVID
[String]::Format("{0:x}", $Logical.SubsystemVendorID)
#SSID
[String]::Format("{0:x}", $Logical.SubsystemID)


#ILO
#$Logical | where {$_.CreationClassName -eq "OMC_IPMISubSystem"}
#$_.CreationClassName = "OMC_IPMIIPProtocolEndpoint"
#$_.ElementName = "Hewlett-Packard Company iLO3 Slave instrumentation"
#$_.ElementName = "Hewlett-Packard Company iLO3 Management Processor Support and Messaging"
#$_.ElementName = "Hewlett-Packard Company Proliant iLO2/iLO3 virtual USB controller"
#ElementName              : Hewlett-Packard Company iLO3 Slave instrumentation
#ElementName              : Hewlett-Packard Company iLO3 Management Processor Support and Messaging #0
#ElementName              : Hewlett-Packard Company Proliant iLO2/iLO3 virtual USB controller #0
#$software ElementName   : BMC Firmware (node 0) 46:10000

$Report.Driver = ""
$Report.Firmware = ($software | where {$_.ElementName -match "BMC Firmware"}).versionstring
$Report.VID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "ilo3" -and $_.vendorid -ne $null})).vendorid[0])
$Report.DID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "ilo3" -and $_.PCIDeviceID -ne $null})).PCIDeviceID[0])
$Report.SVID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "ilo3" -and $_.SubsystemVendorID -ne $null})).SubsystemVendorID[0])
$Report.SSID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "ilo3" -and $_.SubsystemID -ne $null})).SubsystemID[0])
#P220i
$Logical | where {$_.CreationClassName -eq "HPVC_SAController"}
$_.ElementName = "Hewlett-Packard Company Smart Array P220i"
ElementName              : Hewlett-Packard Company Smart Array P220i #0
$software ElementName   : scsi-hpvsa
ElementName   : scsi-hpsa
ElementName   : Smart Array Controller HPSA1

$Report.Driver = ""
$Report.Firmware = ($software | where {$_.ElementName -match "BMC Firmware"}).versionstring
$Report.VID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "Hewlett-Packard Company Smart Array P220i" -and $_.vendorid -ne $null})).vendorid[0])
$Report.DID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "Hewlett-Packard Company Smart Array P220i" -and $_.PCIDeviceID -ne $null})).PCIDeviceID[0])
$Report.SVID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "Hewlett-Packard Company Smart Array P220i" -and $_.SubsystemVendorID -ne $null})).SubsystemVendorID[0])
$Report.SSID = [String]::Format("{0:x}", [int](($Logical | where {$_.ElementName -match "Hewlett-Packard Company Smart Array P220i" -and $_.SubsystemID -ne $null})).SubsystemID[0])
#?BL460G8?
SystemCreationClassName = "OMC_UnitaryComputerSystem"
ElementName                    : Unknown Unknown #31
ElementName              : Unknown Unknown #0
#Emulex 10Gb 554FLB
ElementName              : Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
ElementName              : Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter #0
$software ElementName   : net-be2net
ElementName   : be2net driver
ElementName   : be2net device firmware
#Emulex 10Gb iSCSI UCNA
ElementName              : Emulex Corporation Emulex OneConnect OCe11100 10GbE, iSCSI UCNA
ElementName              : Emulex Corporation Emulex OneConnect OCe11100 10GbE, iSCSI UCNA #0
$software ElementName   : scsi-be2iscsi
ElementName   : ima-be2iscsi
#Qlogic ISP2532
ElementName              : QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
ElementName              : QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA #0
$software ElementName   : ima-qla4xxx
ElementName   : scsi-qla4xxx
#BIOS
$software ElementName   : System BIOS

 

 

 

Patch management wrap up

Thanks for reading this far. I know their is not a lot of answers in this blog post, but hopefully you have learned a new way (or two) to get additional information about the hardware you have deployed. This might not be a finished product and in all like hood will never be so, but it does show how hard it can be to find good ways to ensure the hardware have the correct drivers and firmware.

That’s all for now, take care.

 

Leave a Reply

Your email address will not be published. Required fields are marked *