How to see if a Device/Datastore supports VAAI on ESXi 5 using ESXTOP or ESXCLI

Previously, using my home lab environment I had wondered how to check to see if my shared storage was using VAAI (VMware Storage APIs for Array Integration) – this is a technology that provides hardware acceleration functionality for your storage in vSphere, and was first introduced for vSphere ESX(i) 4.1. Essentially, this allows your hosts to offload certain virtual machine and storage management functions to hardware that supports VAAI.

 

The very first way I found I was able to see if VAAI was active on a datastore was by using ESXTOP. Silly – as it is quite a long-winded way of determining this, but it gave me an idea of whether it was in use or not. Anyway, to get to this ESXTOP view, go to your devices screen (press u) and remove a few columns that are taking up room (press F to add/remove columns), toggle all columns except the DEVICE name column, then add the counters we are interested in for VAAI. (press O – for VAAISTATS). Like so:

 

 

Now you can see the VAAI statistics for your datastores/devices in ESXTOP – notice how my HP/LHN P4000 VSA lab iSCSI datastores are showing VAAI stats as expected, whereas my shared NFS Datastores from a FreeNAS appliance are blanked out / not supported in the screenshot below:

 

 

 

Here is a useful list of ESXTOP counters that are explained in detail if you are wondering what each of these shows exactly: http://communities.vmware.com/docs/DOC-11812 – Section 4.2.9 shows ZERO statistics specifically, which is one of the features VAAI can help hosts offload to hardware.

 

So now, for a slightly easier / clearer way of viewing whether VAAI is active or not – we can use esxcli!

 

Connect up to an ESXi 5 host using SSH or the DCUI and issue the following command (Where the device identifier is specified after -d) – remember to find the device identifier applicable to the device you want to check for VAAI status on – one way of doing this is to identify the datastore using your vSphere client – Hosts & Clusters view -> Select ESXi host -> Configuration -> Storage -> Devices View tab -> Look for device identifier in the name column. Once you have your device ID, issue the command below:

 

esxcli storage core device list -d naa.6000eb3afd81276200000000000000bd

 

You will get a list out of the device along with various settings and feature statuses. We are specifically looking for “VAAI Status” – if it is enabled, we will see “VAAI Status: enabled” as in the example below.

 

 

Brian Ragazzi has also showed an even easier method (thanks Brian!) to get a list of devices along with their VAAI feature support listed using:

 

esxcli storage core device vaai status get

 

using “esxcli storage core device vaai status get” to get VAAI status for devices

 

Interestingly, whilst researching this, I came across a blog post by Jason Langer who also explains the first method I showed above of checking VAAI status – in his case he was using Netapp storage and found some other interesting caveats along the way in terms of getting VAAI enabled on his hosts with his particular device. Well, I hope this post helps and as always, if you spot anything out of place – feel free to chime in on the comments section!

 

Get vCenter User Sessions and Idle times with PowerCLI

Today I was looking into a small “nice to have” notification system for users that had left their vSphere clients open and logged into vCenter. I found this great bit of script to list currently logged in users over at blog.vmpros.nl and thought I would expand on this in my own way to generate a handy list of logged in users and their current idle time – similar to the way the “Sessions” tab in the vSphere client displays user session information. When I got home this evening I expanded on the original script from vmpros.nl to create the following:

 

$Now = Get-Date
$Report = @()
$svcRef = new-object VMware.Vim.ManagedObjectReference
$svcRef.Type = "ServiceInstance"
$svcRef.Value = "ServiceInstance"
$serviceInstance = get-view $svcRef
$sessMgr = get-view $serviceInstance.Content.sessionManager
foreach ($sess in $sessMgr.SessionList){
   $time = $Now - $sess.LastActiveTime
   # Our time calculation returns a TimeSpan object instead of DateTime, therefore formatting needs to be done as follows:
   $SessionIdleTime = '{0:00}:{1:00}:{2:00}' -f $time.Hours, $time.Minutes, $time.Seconds
   $row = New-Object -Type PSObject -Property @{
   		Name = $sess.UserName
		LoginTime = $sess.LoginTime
		IdleTime = $SessionIdleTime
	} ## end New-Object
	$Report += $row
}
$Report

 

[download id=”6″]

 

Here is an example of the output of the script:

 

Using this bit of PowerCLI script, it should be easy for you to create your own notification system based on user session idle time, or some functionality that would disconnect idle users. Let me know if you do improve on the above, or if you have any other suggestions.

 

PowerCLI Advanced Configuration Settings – Setting Host Syslog options

 

This is a PowerCLI script that will script setting up your ESXi host syslogging in two places: A Shared Datastore (that all ESXi hosts have visibility of) and to a remote Syslog host on the network. Alternatively, you could set it to change the Syslog directory to a different local path on each ESXi host (rather than the default). The two advanced configuration settings we’ll be working with are:

 

  • Syslog.global.logDir
  • Syslog.global.logHost

 

Why did I do this? Well, tonight I got a little sidetracked after looking into a small warning on a new scripted ESXi host I deployed – it had not automatically set up logging during installation, my guess, was because it’s local datastore had failed to initialise during installation – I couldn’t see a local datastore after deploying it via a script and had received a warning during installation. I went ahead just to see what happened. This was a new test host in my home lab. So afterwards, I decided I would set it up a remote Syslog location to clear the warning off. Naturally, I wanted to do this in PowerCLI.

 

The following script will take a few options you specify like Shared Datastore name, Syslogging Folder Name, and a remote Syslog host, and go ahead and configure all the ESXi hosts in your specified DataCenter object to use these options you specify. If you have only one datacenter object the script will see this and use that one by default. Otherwise it will ask you for the name of the specific DataCenter you want to work with. It will then create a “ESXi-Syslogging” folder on the shared datastore you specified (if it doesn’t already exist) and under this, create a sub-folder for each ESXi host. It will then configure each ESXi host to send it’s Syslog info to this datastore. As far as I can tell, certain Sysl0g files are linked from /scratch/log on each ESXi host to the datastore specified. For example:

 

usb.log becomes:

usb.log -> /vmfs/volumes/1bec1487-1189988a/ESXi-Syslogging/esxi-04.noobs.local/usb.log

 

Naturally, this wouldn’t be a good idea if the ESXi host could potentially lose access to this datastore, so the script will also specify a remote Syslog host to log to. If you don’t have one, check out the Free “Kiwi Syslog Server”. Here is the resulting script (download via URL below or just copy the script out the block):

[download id=”4″]

 

# Remote Syslogging setup for ESXi Hosts in a particular datacenter object
# By Sean Duffy (http://www.shogan.co.uk)
# Version 1.0
# Get-Date = 15/12/2011

#region Set variables up
$SyslogDatastore = "SharedDatastore2" # The name of the Shared Datastore to use for logging folder location - must be accessible to all ESXi hosts
$SyslogFolderName = "ESXi-Syslogging" # The name of the folder you want to use for logging - must NOT already exist
$RemoteSyslogHost = "udp://192.168.0.85:514" #specify your own Remote Syslog host here in this format
$DataCenters = Get-Datacenter
$FoundMatch = $false
$SyslogDatastoreLogDir = "[" + $SyslogDatastore + "]"
#endregion

#region Determine DataCenter to work with
if (($DataCenters).Count -gt 1) {
	Write-Host "Found more than one DataCenter in your environment, please specify the exact name of the DC you want to work with..."
	$DataCenter = Read-Host
	foreach ($DC in $DataCenters) {
		if ($DC.Name -eq $DataCenter) {
			Write-Host "Found Match ($DataCenter). Proceeding..."
			$FoundMatch = $true
			break
			}
		else {
			$FoundMatch = $false
			}
	}
	if ($FoundMatch -eq $false) {
		Write-Host "That input did not match any valid DataCenters found in your environment. Please run script again, and try again."
		exit		
		}
	}
else {
	$DataCenter = ($DataCenters).Name
	Write-Host "Only one DC found, so using this one. ($DataCenter)"
	}
#endregion

#Enough of the error checking, lets get to the script!

$VMHosts = Get-Datacenter $DataCenter | Get-VMHost | Where {$_.ConnectionState -eq "Connected"}

cd vmstore:
cd ./$DataCenter
cd ./$SyslogDatastore
if (Test-Path $SyslogFolderName) {
	Write-Host "Folder already exists. Configure script with another folder SyslogFolderName & try again."
	exit
}
else {
	mkdir $SyslogFolderName
	cd ./$SyslogFolderName
}

# Crux of the script in here:
foreach ($VMHost in $VMHosts) {
	if (Test-Path $VMHost.Name) {
		Write-Host "Path for ESXi host log directory already exists. Aborting script."
		exit
	}
	else {
		mkdir $VMHost.Name
		Write-Host "Created folder: $VMHost.Name"
		$FullLogPath = $SyslogDatastoreLogDir + "/" + $SyslogFolderName + "/" + $VMHost.Name
		Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name "Syslog.global.logHost" -Value $RemoteSyslogHost
		Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name "Syslog.global.logDir" -Value $FullLogPath
		Write-Host "Paths set: HostLogs = $RemoteSyslogHost LogDir = $FullLogPath"
	}
}

 

Some results of running the above in my lab:

 

Script output after running against 3 x ESXi hosts

 

Advanced settings are showing as expected
Tasks completed by the PowerCLI script

 

Here is the structure of the Logging in my Datastore after the script had run.

 

 

Lastly, I have found that these values won’t update if they are already populated with anything other than the defaults (especially the LogDir advanced setting). You’ll get an error along the lines of:

 

Set-VMHostAdvancedConfiguration : 2011/12/15 11:37:01 PM    Set-VMHostAdvancedConfigurati
on        A general system error occurred: Internal error

 

If this happens, you could either set the value to an empty string, i.e. “” or you could use Set-VMHostAdvancedConfiguration [[-NameValue] <Hashtable>] (using a hashtable of the advanced config setting as the NameValue) to set things back to the way they were originally. The default value for Syslog.global.logDir is = [] /scratch/log for example. It might be a good idea to grab this value and pipe it out into a file somewhere safe before you run the script in case you need to revert back to the default Syslog local location – you would then know what the value should be. You could use something like this to grab the value and put it into a text file:

 

Get-VMHostAdvancedConfiguration -VMHost $VMHost -Name "Syslog.global.logDir" | Out-File C:\default-sysloglocation.txt

 

Also, remember to think this out carefully before changing in your own environment – it may not suit you at all to set your Syslog location to anywhere else other than the local disk of each host. However, you may find it useful to automatically set the remote syslog host advanced setting for each of your ESXi hosts – in that case, simply modify the script above to only apply the one setting instead of both. I hope this helps, and as always, if you have any suggestions, improvements or notice any bugs, please feel free to add your comments.

 

 

Upgrading to vSphere 5.0 from vSphere 4.x – Part 2 – Manually upgrading ESX(i) hosts

 

Introduction

 

In Part 1 of this series, I covered the steps needed to move your vCenter server over to vCenter 5.0, and outlined some points to consider or be aware of. For this post, I’ll be covering two different methods you can use to migrate your ESX(i) hosts over to ESXi 5.0. Which one you use really depends on how big your deployment is. For someone running a small deployment of just a few hosts, it may make more sense to manually upgrade each using an ISO. For others with larger environments, it would be prudent to use an automated method such as VMware Update Manager (VUM). I’ll be covering both methods over the next two posts (today and tomorrow), and will outline the process involved in each.

 

Manually upgrade ESX(i) 4.x to 5.0

 

This is a fairly straightforward process, and simply involves restarting each of your hosts in the cluster, one at a time and booting them off an ESXi 5.0 installer image. To start, you’ll want to prepare each ESXi host as you get to it, by entering it into Maintenance mode. This will migrate any VMs off and allow you to restart it safely with no VM downtime. Ensure all your VMs have vMotioned off before you begin. In my case, my ESXi hosts are virtual, so I simply mounted the VMware ESXi 5.0 installer ISO on each VM. For physical hosts, you’ll want to burn the installation ISO to CD/DVD and pop it into each host of course.

 

Boot off the ISO and select to start the standard installer

 

Once the Installer has loaded, choose the disk you would like to perform the upgrade on. All disks on the host that are VMFS should be marked with an asterisk for you to easily identify.

 

Choose the disk to upgrade

 

 

Press F1 to get the details of the disk you are going to upgrade during installation. The installer should detect your current version of ESX(i) as well as provide a bit of extra information in case you need to confirm you are upgrading the correct disk etc…

 

Viewing the Disk details

 

On the next step, you should get a few options relating to the type of action you want to take on the existing installation of ESX(i). For this type of upgrade, I’ll be choosing to Force Migrate my ESXi 4.1 host to ESXi 5 and preserve the existing local VMFS datastore.

 

 

After continuing, you may get a warning if there are any custom VIBs (VMware Installation Bundle) in your current ESX(i) installation that do not have substitutes on the install ISO. If there are, you should look into whether or not these will still be needed, or check that you have a contingency plan to re-install or reinstate the particular bit of software should you need to after the upgrade. I got a warning about some OEM VMware drivers, but I knew these were just related to the ESXi installation in a VM I am running, so I was happy to continue the upgrade process anyway.

 

Next up, you’ll just need to confirm you wish to Force Migrate to ESXi 5.0.0 on the specified disk you chose earlier.  The installer will also inform you as to whether you will be able to rollback the install or not. Press F11 to continue at this point if you are happy.

 

 

 

Once the migration / install is complete you’ll get the below message. Press Enter to reboot the host.

 

 

 

At this stage, once the host has booted up again, you’ll want to log into it using the vSphere client, or check on it through your vCenter connection in the vSphere client. Make sure everything is configured as you would expect and that it is running smoothly on ESXi 5.0.0. That is all there is to it really – provided all went well, you should have a newly upgraded host ready to run VMs again. The upgrade process is quite quick and straight forward. Test it out by vMotioning a test VM onto the new host and check that there are no issues. Tomorrow, we’ll look at performing more of an autonomous upgrade for your hosts to ESXi 5.0.0 using VUM (VMware Update Manager).

 

More in this series:

 

Part 1 – Upgrading to vSphere 5.0 from vSphere 4.x – Part 1 – vCenter Server
Part 3 – Upgrading to vSphere 5.0 from vSphere 4.x – Part 3 – Using VUM to upgrade ESX(i) hosts

 

How to move your VMware vCenter Server database (SQL)

 

My vSphere 5 lab at home had been feeling a little sluggish when it came to running the vSphere client and working with vCenter – I could see that the drive where my vCenter DB was running (SATA) was taking a bit of hit when I accessed Performance graphs and the like from vCenter. I had a little bit of free SSD storage on another drive (OCZ Vertex 2 SATA drive) that would be perfect for my vCenter DB size, so in the search of better performance in my lab, and a bit of a practise run at moving the vCenter SQL database, I set about moving just the SQL databases on the VM to a new, dedicated SSD-based VMDK. This also meant that I didn’t take up too much space moving the entire VM across to SSD storage, as this kind of storage comes at a premium!

 

This post will cover the process I followed for SQL Server 2005 Express. Most labs environments are likely to be running this edition, especially if you upgrade the lab from vSphere 4.1. The steps are quite similar for a SQL 2008 database, but there are differences, so just make sure you follow the correct KBs if you are on a different edition of SQL Server. Here is the high level overview of what was involved in the whole process.

 

High level overview: vCenter 5 steps on SQL Server 2005 Express

 

  • Shutdown lab and make a full clone of the vCenter VM. Power back up again afterwards – Always good to have a rollback plan!
  • Added a new disk to the VM, located on my SSD-based storage.
  • Backup all my SQL databases on the vCenter VM along with the System databases.
  • Noted down all credentials that vCenter uses to connect to the SQL database and checked I was familiar with all my ODBC settings just in case any of these needed changing or updating.
  • Stopped vCenter and VUM services
  • Performed database move steps carefully, verifying everything each step of the way.
  • Started vCenter and VUM services back up and check all was working as expected.
  • Note that there are some additional considerations if you are planning on moving a vSphere 4.x database. Refer to the VMware KB linked below for more info if you are on vSphere 4.x

 

VMware have a fairly high-level KB on moving your vCenter Server SQL database. You can take a look at it over here to see if you need anything else.

 

The Process

 

  • After making a clone of my vCenter VM, backing up all my SQL databases on the vCenter server, and stopping all my VMware specific services, I started with the Microsoft specific steps for moving a SQL 2005 database.

 

  • First off, you need to detach the VIM_VCDB database. Execute the following SQL query in SQL Server Management Studio:

 

use master
   go
   sp_detach_db 'VIM_VCDB'
   go

 

  • After this query completes successfully, move your VIM_VCDB.mdf and VIM_VCDB.ldf files to the new location (where you are moving the database to). Once moved, go back to Management Studio and execute the next query which will reattach the database. Of course you will need to specify the path your database is now going to be located in – the example below references the path I used.

 

use master
     go
     sp_attach_db 'VIM_VCDB','D:\VCDB_SQLDATA\VIM_VCDB.mdf','D:\VCDB_SQLDATA\VIM_VCDB.ldf'
  go

 

  • After this query is successful, you can run the next stored procedure, which should return the new location of your database, provided it has been moved and reattached correctly.

 

use VIM_VCDB
go
sp_helpfile
go

 

  • Now that the Database has been moved, you should be good to start your vCenter services back up again and do some testing to ensure everything is working as expected.

 

Extra steps

 

If you would also like to move your System databases, things get a little more complex. First off, you will need to make sure your management studio is set to only open up a SQL query window on startup. (Tools -> Options -> Environment -> General ->At Startup -> Open new query Window) this is so that when we enter single-user mode for the SQL Server service (part of moving the System DBs), we don’t get errors trying to run our scripts – as by default SQL Server Management Studio tries to make a SQL connection for it’s object explorer, as well as for your query when you execute it. This means you’ll get an error message as there would already be an active SQL connection using the Object Explorer before you can execute your queries to move the System DBs. The Microsoft KB does not explain this, and it took me a little while to realise that this was the problem, so don’t forget to change this before starting! The rest of the steps can be followed through in this MSDN Article on moving SQL Server System Databases. Just make sure the correct edition of SQL Server is highlighted at the top of the page before you begin.

 

The System Databases include:

 

  • Model
  • MSDB
  • Master
  • Resource
  • TempDB

 

Another tip when moving the System databases that was not mentioned in the Microsoft KB article, and will cause you to get stuck unless already configured, is to set the correct permissions on the new folder your SQL System databases are going to be sitting in. There is a local security group on your SQL server that needs to be assigned “Full Control” on your new System database location. If it doesn’t have this permission, then you will get errors when you try to start services up again. See screenshot below for the service name (the name of this security group depends on the name of your Server and SQL Service):

 

 

That should cover the whole process, as well as provide a few tips on the areas I initially got stuck on (that were not explained in the official Microsoft KBs). If you have a different edition of SQL Server, just switch the MSDN article to the relevant edition and take it from there. Good luck!