recreate

Can’t find/create new VMFS datastore after removing old one using Add Storage Wizard

by

 

Yesterday I was doing some out of hours work for a client who had a VM on a VMFS 3 datastore that only had a 1MB blocksize, but needed a data disk larger than 256GB. Naturally, the datastore needed to be recreated with a larger blocksize, as it is in a vSphere 4 environment and running VMFS 3. I did a storage vMotion to move the data off the datastore, then removed the datastore using one of the ESX hosts. I checked the other ESX hosts and noticed it was still showing there too. Refreshing the storage on each solved this and the datastore disappeared as expected.

 

I then went to re-create the datastore, happy with the knowledge that our SAN would still be presenting the LUN to our ESX hosts. When I arrived at the Add New Disk/LUN wizard, I saw that the LUN was in fact, not showing. Here is the troubleshooting process I used to sort this issue out. It turned out to be two old ESX hosts that were decomissioned a short while back who were still “hanging on” to this datastore.

 

  • Checked each live ESX host for the Datastore and made sure it was removed
  • Rescanned each HBA on the ESX hosts for new storage etc…
  • Tried adding datastore again (still couldn’t find the storage)
  • Logged onto FC SAN and ensured the LUN was still being presented to all ESX hosts in the cluster
  • Used vCenter “Maps” feature to select “VM to Datastore” relationship to ensure no VMs were mapping to this datastore or trying to refer to it anymore (like CDs mounted etc).
  • Under Maps, I noticed the datastore in question was still visible at the very bottom, but nothing related to it – i.e. there were no branches to anything.
  • Went to Datastores view in vCenter, and saw the datastore still listed there but had “(inactive)” next to it’s name.
  • Highlighted the datastore and looked at it’s summary (see screenshot below for example) and saw 2 x ESX hosts still showing as connected to it. I then realised that these would be the old, decommissioned ESX hosts in the inventory still “holding on” to the datastore.
  • I removed these old hosts from the vCenter inventory, and the datastore was then gone. I was then able to re-create it and the presented LUN appeared in the add storage wizard as you would normally expect.
 
datastore showing as (inactive) even after removing it from all connected ESX hosts

 

2 x ESX/ESXi hosts were still connected to the old, removed storage, preventing it from being re-created (This example shows one with 5 hosts connected for reference)

 

So if you find yourself removing a datastore, and when arriving at the add new storage wizard and this LUN is not being presented back to you as an option to create a new datastore with, try the above troubleshooting steps to ensure nothing is still referencing the old, deleted datastore. Once sorted out, you should get the option back again.