Storage Latency is the #1 Suspect
Building a VMware-based private cloud comes with a specific set of storage challenges resulting in prevalent storage latency issues. Approximately 70% of performance issues in VMware environments are driven by storage latency. This due to three reasons:
Limited visibility
From the storage side, it is hard to distinguish storage IO from different VMs. From the VMware side, however, the resource competition that different VMs are creating when they hit the same datastore is hard to see. Therefore placement decisions are frequently made based on available disk space not based on the actual usage patterns and available IOPS or storage throughput. "We placed the VM on this datastore because it had free space ..."
VMware's focus on CPU and memory
Using VMware it is easy to precisely tailor the amount of CPU and memory resources allocated to each VM; storage is a different matter. While the required disk space is easy to provision, it is impractical to properly allocate each VM the correct IOPS (Input/Ouput Operations per Second) and storage throughput (in MB/s).
High impact of resource competition
The technical nature of disk IO results in high impact on performance and utilization. If storage latency peaks at hundreds of milliseconds or even seconds, performance becomes visibly sluggish and billions of CPU cycle can be wasted.