This post discusses datastore sizing concerns in the production design of many virtual infrastructures. During VMware View Health Checks, an issue may be identified regarding VMFS datastore sizes. The priority 1 recommendation suggested that datastore sizes be reduced, accessed by fewer hosts, allocating no more than 64VMs per LUN/VMFS datastore. The implementation of the smaller datastore has become pain point support teams. This document explains the rationale behind that decision, as well as a path going forward.
In previous versions of vSphere, VMFS used SCSI reservations on storage devices that did not support hardware acceleration. In addition, previous versions of Data ONTAP, SVC, or other storage array did not support hardware acceleration. When SCSI reservations happen, they lock an entire storage device during any operation that requires metadata protection, up until the time the operation completes. Since this lock is exclusive, excessive SCSI reservations can cause performance degradation on any host accessing the same VMFS volume.
SCSI reservations can be caused by certain activities performed on disk. Many of these activities are listed in the section below.
Things that cause SCSI reservations:
- Creating, resignaturing, or expanding a VMFS datastore
- Powering on a virtual machine
- Creating or deleting a file
- Creating a template
- Deploying a virtual machine from template
- Creating a new virtual machine
- Migrating a virtual machine with vMotion
- Growing a file, such as a thin provisioned virtual disk
A VMware View environment is ripe for SCSI reservation issues, given the following:
- View desktops experience far more power cycles than a server
- View desktops and their associated disks are created and deleted on a regular basis
- View desktops are thin provisioned, and grow daily
- View desktops are typically smaller than servers, and have more virtual disk files, resulting in more VMs and VMDKs per datastore than a server infrastructure
- View linked clones are based on a snapshot of the parent, and typically generate a significant amount of SCSI reservations
A few approaches to reduce SCSI reservation conflicts are listed below:
- Limit the number of operations on different hosts that require SCSI reservations at the same time
- Thick provision disks
- Limit the number of hosts that can access a specific LUN
- Reduce the number of snapshots
- Reduce the number of virtual machines per LUN
Many of the approaches listed to reduce SCSI reservations introduce additional cost and administrative overhead, some are simply not possible. For example, using thick provisioned disks greatly increases the amount of SAN storage required by the virtual infrastructure. Dealing with the design considerations for addressing SCSI reservation issues is not a trivial task and requires design updates as new product improvements are released.
The New Approach
Introduction of Atomic Test and Set (ATS) in vSphere 5.0, using VAAI helps resolve SCSI-2 reservation issues. ATS is available on storage devices that support hardware acceleration. ATS is sometimes referred to as hardware assisted locking. Unlike SCSI reservations, ATS supports discrete locking per disk sector. In these cases where hardware acceleration is available on the storage array VMFS uses ATS rather than SCSI reservations. In order to ensure all ATS capabilities are available, the VMFS datastore must be at version 5. Also note that VMFS datastores that were upgraded from VMFS-3 to VMFS-5 may revert back to using SCSI reservations to handle locking.
Storage devices that support hardware acceleration perform the following tasks more efficiently:
- Migration of virtual machines with Storage vMotion
- Deploy virtual machines from templates
- Cloning virtual machines and templates
- VMFS clustered locking and metadata operations for virtual machine files
- Writes to thin provisioned and thick virtual disks
- Creating fault-tolerant virtual machines
- Creating and cloning thick disks on NFS datastores
The current storage design can be updated to reflect new features introduced as a part of the Hypervisor. The implementation of the recommendation from a View health check is causing pain in that the growth of View persistent disks has begun to cause space issues on VMFS datastores. Space issues are compounded by the inability to leverage storage vMotion for Linked clones, as a rebalance operation is the only supported method for balancing out datastore usage. Storage vMotion is supported on full clones, so putting datastores in datastore clusters and enabling SDRS would helpful. Given that, in many environments, full clones and linked clones reside on the same datastores. This option is not viable without some logical changes to the infrastructure. Finally, some may be consumed unnecessarily for old replicas.
All this being said, the solution must be two-fold:
- The solution must continue to ensure that SCSI reservation issues do not occur (or are minimized to a great extent)
- The solution must address space concerned expressed by View support teams.
Ensure SCSI reservation issues remain a non-issue
- Confirm Hardware Acceleration is supported on the storage array
- Ensure that all primitives are available on each host
- Provision new storage volumes as new VMFS-5 datastores.
Address datastores running out of space
- Separate datastores for full clones and linked clones
- Configure datastore clusters and SDRS for full clone datastores
- Regularly rebalance linked clones across linked clone datastores
- Clean up old replicas
- (Optional) Increase size of datastores, and rely on the fact that ATS will handle disk sector locks, rather than having SCSI-2 reservation issues.
To implement the above solution, the approach to be used will likely follow, at a high level, the steps listed below:
Dedicate datastores either as a full clones or for linked clones
- Migrate full clones to their appropriate datastores
- Migrate linked clones to their appropriate datastores by performing a rebalance operation
Combine full clone datastores into a datastore cluster
- Enable SDRS
- Resize LUNs, increasing them to more palatable size
- Rebalance linked clones
Enjoy a balanced, automated storage infrastructure
These steps require testing in a non-production environment, as the approach may vary depending on the environment.
This document addressed the rationale for leveraging smaller VMFS datastores, as well as the implications of SCSI reservations. Additionally, recommendations were provided, as a path forward, to alleviate the pains resulting in smaller LUNs.
VMFS Locking Mechanisms – p157 for SCSI Reservations; p215 for hardware assistance
vSphere 5.1 Excessive SCSI Reservations
Discussion Regarding Reservations and the new ATS
How to detect datastores upgraded to VMFS-5
Differences between Upgraded and newly created VMFS-5 datastores:
Atomic Test and Set