vCenter Server Appliance VM fails to boot with fsck failed message

 

I had a storage outage to deal with recently, and after the datastores on this storage were taken down, a vCenter Server Appliance VM on the storage got some corrupted files and would not boot. Upon start up, I was greeted with this message:

vcenter-server-appliance-fsck-failed

The error message reads:

fsck failed.  Please repair manually and reboot.  The root
file system is currently mounted read-only.  To remount it
read-write do

After trying the mount command using the maintenance mode bash shell, I restarted and found that the appliance still did not boot properly. I found a thread on the VMware community forums where someone had the same issue and was able to run e2fsck to fix the disk issues. I tried this and found it fixed a whole heap of disk errors on the /dev/sda3 mount, but on restart I noticed more issues on the /dev/sdb2 mount, so I ran the e2fsck command again for this path, and was able to finally reboot the appliance successfully. The commands I ran to resolve were essentially:

  • mount -n -o remount,rw /
  • e2fsck -y /dev/sda3
  • e2fsck -y /dev/sdb2
  • CTRL-D to reboot after fixing the errors using e2fsck