Fixing file system corruption in the vCenter Server Appliance

I recently had a storage network outage in my lab environment, and after powering back on my vCenter Server Appliance (VCSA) I was rudely greeted with the following information at the console:

fixing-disk-corruption-in-the-vcenter-server-appliance-img01

Ouch!  I’ve never dealt with file system corruption in a VCSA before, and the internet doesn’t seem to contain much information on what to do next.  This post is my effort towards changing that.

The first thing I did was type “journalctl” to view the system logs, as suggested.  That displayed some additional information on the issue.

fixing-disk-corruption-in-the-vcenter-server-appliance-img02

It suggests running fsck manually, without the -a or -p options.  My background is heavily windows-focused, and I’ve never used fsck before, but I eventually figured out the syntax required to fix the problem, which is to run fsck against the damaged partition listed in the top line of the previous screenshot, i.e. fsck /dev/disk/by-parttitionuuid/79d76ed0-0297-4e33-a6bd-252099f2c613

Thankfully, I was able to use tab completion to save me from typing (and probably mis-typing) the entire partition UUID myself.  Running that command kicked off the fsck process and prompted me to fix probably 10-20 different errors:

fixing-disk-corruption-in-the-vcenter-server-appliance-img03

After completing all that, I got a success message.  It might be a good idea to run the whole thing twice, just to be sure.

fixing-disk-corruption-in-the-vcenter-server-appliance-img04

After that, I rebooted my vCenter appliance, and thankfully it booted normally!  I hope this post helps someone out there who is facing this issue.

 

1 comment

  1. I too faced this same problem in my new lab environment; I found that the steps below fixed the issue.

    It’s most helpful that you saved these steps somewhere.

Leave a Reply

Your email address will not be published. Required fields are marked *