VMware ESXi and vSphere Cluster Management

Monitor Disk Latency and Storage Performance in VMware ESXi

Learn how to use vCenter Advanced performance charts to compare ESXi kernel and physical-device disk latency and isolate storage bottlenecks.

Why Monitor Disk Latency?

Storage-heavy virtual machines and applications can consume more I/O capacity than a host, controller, path, or storage system can process. When demand exceeds available processing capacity, storage requests wait. This condition is called storage saturation.

Capacity and performance are different measurements. A datastore can have plenty of free space and still deliver poor performance because its disks, array controllers, SAN paths, network storage links, or host queues are overloaded.

Disk latency is the time required to complete storage I/O, usually measured in milliseconds. Elevated latency can cause slow application responses, delayed virtual machine operations, and increased I/O wait inside guest operating systems.

Terms Used in This Lesson

  • ESXi: VMware's bare-metal hypervisor. It runs virtual machines and processes storage I/O through its host stack.
  • vCenter Server: The centralized management platform used to manage ESXi inventory and view performance charts.
  • vSphere Web Client: The browser-based vSphere management interface used in older releases. Current interfaces may use different labels for comparable monitoring views.
  • VMkernel: The ESXi kernel layer that handles host functions, including storage I/O processing.
  • SCSI command: A storage request sent through the SCSI command model, which ESXi uses across many storage types.
  • Disk controller: A host storage adapter or controller through which ESXi reaches storage devices and paths.
  • Real-time chart: A chart interval intended for observing current or very recent behavior during active troubleshooting.

The Two Host Disk-Latency Counters

vCenter provides two primary host-level counters for examining storage-command delays. Both are averages expressed in milliseconds per SCSI command. View them together: comparing the two values helps indicate whether delay is concentrated in the ESXi host or occurs farther downstream in physical storage.

ESXi Disk Latency Counters and Investigation Guidance

Kernel command latency — Average VMkernel processing time per SCSI command. A sustained value above approximately 2 ms warrants investigation. Examine the ESXi storage stack, queueing, adapters, paths, and host-side contention.

Physical device command latency — Average time required by the underlying physical storage device to complete a SCSI command. A sustained value above approximately 15 ms can indicate a storage-device or storage-fabric performance issue. Examine the array, disks, SAN or networked storage, controllers, paths, and concurrent workload.

Kernel Command Latency

Kernel command latency is the average time a storage command spends being processed within VMkernel. A sustained value above approximately 2 ms is an investigation signal.

Possible causes include host-side storage-stack overhead, queue pressure, contention, path problems, or configuration and compatibility issues. The threshold is not automatic proof of a failure. Compare it with the physical-device counter, workload timing, and the behavior of other hosts or controllers.

Physical Device Command Latency

Physical device command latency is the average time taken by the underlying physical storage device to complete a SCSI command. A sustained value above approximately 15 ms can indicate that the storage device or the storage path is slow.

This delay may originate in an array, disk media, storage controller, SAN fabric, networked-storage link, or a path serving the device. It can also result from heavy concurrent I/O. Validate the result with storage-platform and connectivity metrics rather than assuming the physical disk itself has failed.

Open Host Performance Monitoring in vCenter

Select the ESXi host when investigating latency that affects several virtual machines, datastores, controllers, or storage paths through that host. The host view provides a useful common point for comparing storage behavior across workloads.

  1. Select the target ESXi host in the vCenter inventory.
  2. Open Monitor.
  3. Open Performance.
  4. Switch to the Advanced chart view.

Interface labels can vary between vSphere releases. Look for the host's performance-monitoring area and its advanced chart or chart-configuration controls.

Before starting, confirm that your account can view host performance metrics. If the host is not visible, review inventory and permission assignments; the guide on assigning permissions covers related access concepts.

Configure an Advanced Disk Performance Chart

Use a real-time line chart during an active incident. A line chart makes short spikes, sustained elevation, and changes between the two counters easier to compare.

  1. Open Chart Options or the equivalent chart-configuration control.
  2. Set the performance metric category to Disk.
  3. Set the time range to Real-time.
  4. Choose Line Graph as the visualization.
  5. Select the applicable host.
  6. Select the relevant disk controller or storage adapter object.
  7. Add Kernel command latency.
  8. Add Physical device command latency.
  9. Apply the configuration.
  10. Verify that both series appear on the same chart and share the same time axis.
Metric category: Disk
Time range: Real-time
Chart type: Line Graph
Object scope: Target ESXi host and applicable disk controller
Counters: Kernel command latency
           Physical device command latency

If the counters do not appear, confirm that Disk is selected, that the correct host is in scope, and that a disk controller or storage adapter object is selected. An incorrect chart scope can hide the expected counters.

Interpret the Combined Chart

Compare both lines over the same interval. First establish whether the elevation is sustained or consists of isolated spikes. Then compare the timestamps with application peaks, backup windows, snapshots, replication, storage maintenance, or batch jobs.

Latency Pattern Interpretation Matrix

Kernel normal; physical device normal: No obvious host-level latency condition appears in the selected period. Maintain the baseline and correlate the chart with reported symptoms.

Kernel elevated; physical device normal or low: Delay may be concentrated in host-side processing, queues, adapters, or paths. Review ESXi host and adapter conditions.

Kernel elevated; physical device elevated: A broad storage delay or downstream device slowdown may be affecting the host. Review storage paths, array metrics, and workload timing.

Kernel normal; physical device elevated: Physical storage response is slow even though host overhead is limited. Investigate the storage device, fabric, network, controller, or array.

Example: Storage Array or Physical-Device Delay

Physical device command latency remains above approximately 15 ms during an application peak, and kernel command latency rises at the same time. The host is likely waiting for storage completion. Investigate the array, device media, SAN or network path, controller load, and concurrent workload activity.

Example: Host-Side or Path-Related Delay

Kernel command latency stays above approximately 2 ms while physical device command latency remains comparatively low. Focus on ESXi-side storage processing, adapter or path conditions, queueing, driver and firmware compatibility, and host contention.

Example: Brief Backup-Related Spike

Both lines spike briefly during a scheduled backup and return to the established baseline. Record the correlation and assess backup I/O behavior, but distinguish a temporary expected burst from persistent or incident-correlated latency.

Example: Controller-Specific Problem

Only one disk controller shows elevated latency while other controllers on the same host remain normal. Narrow the investigation to storage served through that controller, including its paths, attached LUNs, cabling or network links, and target-side configuration.

Monitoring and Escalation Workflow

  1. Establish a baseline. Record normal kernel and physical-device latency during representative workload periods.
  2. Capture incident times. Note when users report slow applications, delayed virtual machine operations, or guest I/O wait.
  3. Compare chart data. Determine whether latency was sustained, isolated, workload-related, or absent during the reported incident.
  4. Scope the condition. Check whether it affects one controller, one host, multiple hosts, or the wider storage environment.
  5. Correlate layers. Compare vCenter results with adapter, path, fabric, array, controller, and workload data.
  6. Choose the investigation owner. Host-only elevation points toward the ESXi or adapter layer; physical-device elevation points toward storage connectivity or the storage platform; elevation across both counters may require all layers to be reviewed.
  7. Collect supporting evidence. Save chart timestamps, selected objects, counter values, affected virtual machines, paths, controllers, workload events, and related storage-system metrics before escalating.

Troubleshooting Checklist

  • Virtual machines are slow but datastore free space is healthy: Use host advanced disk charts. The issue may be I/O saturation or delayed command completion rather than insufficient capacity.
  • Kernel command latency is persistently high: Compare it with physical-device latency and determine whether the behavior is limited to one controller or host. If physical-device latency is not similarly elevated, examine host-side processing, queueing, adapter paths, and compatibility.
  • Physical device command latency is persistently high: Check whether it coincides with storage-intensive jobs and validate storage-system and connectivity health.
  • Values show occasional spikes with no user impact: Compare timestamps with scheduled tasks and the normal baseline. Prioritize sustained or incident-correlated latency.
  • Expected counters are missing: Confirm the Disk category, correct host, and correct disk controller or storage adapter object in chart options.

Exam-Relevant Notes

  • Kernel command latency measures average VMkernel processing time, not the physical device's completion time.
  • Physical device command latency measures average physical-device completion time, not datastore free capacity.
  • The approximate investigation levels are above 2 ms for kernel command latency and above 15 ms for physical device command latency.
  • Thresholds indicate that investigation is warranted; they do not independently prove failure.
  • Both counters use milliseconds per SCSI command and should be compared over the same time interval.
  • Host-level monitoring is appropriate when symptoms may involve several workloads or storage paths through one ESXi host.

For broader administration practice, review the VMware ESXi online course, and use the material on communication between vCenter Server and ESXi when validating management connectivity and inventory behavior.