I've had something odd occur twice now in our environment, each occurrence in a different datacenter. I'm a bit stumped, and a little concerned that I can't determine the root cause for this.
What has happened is:
1. Datastore on FC attached EMC VMAX250f LUN becomes unmounted from all attached hosts
2. Datastore cannot be re-mounted
3. Rescan of storage results in the Datastore being removed entirely from the list of Datastores. The underlying LUN for this datastore is now listed as available for creation of a new Datastore, and vSphere does not recognize that there is any existing Datastores on the LUN.
4. VOMA recognizes that there is a VMFS file system detected on the lun, and shows 0 errors
5. "partedUtil getptbl" shows that there are no partitions defined on the LUN
6. Following KB2046610, I was able to recover both Datastores by re-creating it's entry in the Partition Table, and I'm then able to remount the LUN on all the hosts
So, this has happened twice now, each in a different datacenter and thus on different storage frames. The commonality is however, that these LUNS were both provisioned on the same model of storage frame, an EMC VMAX250F. These frames are roughly 6 months old and are the first of this model placed into service in our datacenters.
In datacenter 1, our vsphere hosts are ESXi 6.5U1 with vCenter 6.5U1. In datacenter 2, our hosts are ESXi 6.0d with vCenter 6.5U1.
I'm at a loss for trying to determine why this may be happening. Is there anything configuration wise on the SAN side that would possibly explain this issue?