Bit of a strange one this – I have not dug deeper to find the root cause, but here is a quick fix for anyone with the issue.
I found we could not open VM console sessions in a vCenter 5.5 environment today. Usually one’s first thought is that it is a DNS or port issue when you see the classic MKS console error in a VM, but in this case I knew that DNS and ports were not an issue, as RDPing direct to the vCenter Server itself, logging in with the C# client and opening VM consoles from there were giving the exact same message. This was the case for the web client as well as the C# client.
The issue was either with the host that VMs were running on, or with the VMs themselves – the simple fix:
vMotion the VM to another host. As soon as this was done, I could open the console session. The underlying issue is still out there, but I have not had the time to dig any deeper to find out the root cause. More discussion and info available from this VMware communities thread: https://communities.vmware.com/thread/450294