Searching vCenter events & tasks in vRealize Log Insight

The list of events and tasks that vCenter maintains for each object in the inventory are extremely useful for forensics analysis in a vSphere environment.  For identifying who created or deleted a VM, resized a vmdk, or shut down a VM or host, events and tasks are where you look. In its default state, however, the events and tasks views in vCenter have some major issues (from a forensic point of view).

First; events and tasks within vCenter can be rolled over.  vCenter comes with a default retention period of 30 days for events and tasks.  You can change the default retention period, but that comes with its own issues, and to be honest keeping a full list of events and tasks for time immemorial isn’t really vCenter’s job.  Second; the list of events or tasks can’t be searched.  It can be filtered, sure, but the filter terms you type only apply to the current page of entries (which displays a maximum of 100 entries).  You can also export a list of events matching a criteria to a csv file, but this will export a list of all matching events from the vCenter DB (so you’ll need to filter it with Excel), and it obviously won’t contain events which are older than the retention period.

This is where vRealize Log Insight (vRLI) comes in handy.  vRLI is a log collection and analytics tool that VMware provides, and can be configured to collect system logs from your ESXi hosts, vCenter Servers and PSC’s, as well as vCenter events, tasks, and alarms.  Now, even a small vSphere environment could generate millions of logs in 24 hours, so the next challenge becomes: how do we actually isolate the events specific to one object (a specific VM, for example)?  The answer is extremely simple, but not entirely obvious.

Under interactive analytics, add a filter for vc_event_type and an operator of exists, then add a second filter for vc_vm_name with an operator of contains and type the VM’s name.  Set your time range, and click search.  That’s it.

For more information, you could take a look at the dashboards which are provided by the VMware - vSphere integration pack, which is installed in vRLI by default.  For example, here’s a dashboard widget showing the vCenter Server tasks by type, over a specified time period.  Clicking the little arrow in the top right corner will take you to the interactive analytics view, where you can add or modify the filters for an even more specific search.

You can use the dashboards for events, tasks, and alarms, to help identify additional event or tasks types that can be filtered through interactive analytics.  All in all, it provides a really powerful method of doing forensics investigation to discover why a certain thing did or didn’t happen.