VMware ESXi and vSphere Cluster Management

Storage Technologies for VMware ESXi

Learn how ESXi abstracts storage, compare DAS, Fibre Channel, FCoE, iSCSI, and NFS NAS, and understand datastores, virtual disks, and shared-storage requirements.

VMware ESXi presents physical storage to virtual machines through several abstraction layers. An administrator may connect an ESXi host to local disks, a Fibre Channel SAN, an iSCSI target, or an NFS NAS system. The virtual machine usually sees none of these underlying details. Instead, it sees a virtual disk attached to a virtual SCSI controller.

This lesson explains the main storage technologies supported by ESXi, how they reach a datastore, and how storage design affects capabilities such as vSphere High Availability and vMotion.

ESXi Host-Level Storage Virtualization

ESXi host-level storage virtualization is the logical abstraction of physical storage resources for virtual machines. ESXi receives storage from disks, arrays, and storage networks, organizes that storage into datastores, and presents virtual machine disk devices to guests.

The storage path can be understood as a series of layers:

Physical storage: Disks or an external storage array provide the actual capacity. Examples include local SATA disks, SAS disks, and shared SAN or NAS storage.

Host adapter or network connection: ESXi reaches the physical storage through storage controllers, Fibre Channel host bus adapters, Ethernet adapters, or other appropriate connectivity hardware.

Datastore: A datastore is an ESXi storage container created on available physical storage. It holds virtual machine configuration files, virtual disks, snapshots, and other VM files.

Virtual disk: A virtual disk is a file stored on a datastore. ESXi presents it to a virtual machine as a disk device.

Guest SCSI disk: The guest operating system normally sees the virtual disk as a SCSI disk attached to a virtual SCSI controller.

The guest operating system and its applications generally do not need to know whether the datastore is backed by local SAS, a Fibre Channel array, iSCSI, FCoE, or NFS. ESXi manages the underlying storage path through its storage adapters and networking components.

Block Storage and File Storage

Block storage presents storage as blocks that a host can use as a disk device. Fibre Channel, FCoE, and iSCSI are networked block-storage approaches. Direct-attached disks also provide block-style disk access to the host.

File storage presents files and directories through a file-sharing protocol. NAS commonly provides file storage to ESXi through the Network File System (NFS) protocol. With NFS, ESXi mounts a remote file system and creates or uses virtual machine files within that file system.

Both models can hold virtual machine files, but they differ in where the storage interface is implemented:

  • With block storage, ESXi accesses storage devices or logical units and creates or uses a datastore on that storage.
  • With NFS, the NAS system provides a file system that ESXi accesses over the network.
  • In both cases, the VM usually receives a virtual SCSI disk rather than direct visibility into the storage protocol.

Storage Technology Categories Supported by ESXi

Direct-attached storage (DAS): Local or externally attached disks connected directly to one ESXi host, commonly through SAS or SATA.

Fibre Channel (FC): A high-speed storage-area network technology that provides shared block storage through Fibre Channel adapters and a SAN.

Fibre Channel over Ethernet (FCoE): Fibre Channel frames transported across Ethernet links.

iSCSI: SCSI commands and data encapsulated in TCP/IP packets and delivered across an IP network.

NAS using NFS: File-level shared storage delivered over a TCP/IP network through the Network File System protocol.

DAS is normally local to one host. FC, FCoE, iSCSI, and NFS NAS are normally used to make storage available through a network, potentially to multiple ESXi hosts. Whether storage is truly shared depends on its configuration and presentation to the hosts, not merely on the name of the technology.

Direct-Attached Storage

Direct-attached storage (DAS) is storage connected directly to an ESXi host rather than through a separate storage network. It includes internal disks installed in the server and external disk shelves or arrays directly connected to that server.

Common DAS connection technologies include:

  • SAS: A storage connection technology commonly used for server disks, disk shelves, and directly attached arrays.
  • SATA: A disk connection technology often used for directly attached storage, including local laboratory or entry-level server disks.

DAS does not require a separate Fibre Channel fabric or storage IP network. An ESXi host can detect the directly connected disks, create a datastore on suitable storage, and place virtual machine files there.

DAS example: a single-host lab

Suppose a laboratory ESXi host contains two SATA disks or a SAS-connected disk shelf. The administrator creates a datastore on that local storage and places a virtual machine's virtual disk on it. The VM can run normally, but another ESXi host cannot access the datastore unless the storage is separately designed and presented as shared storage.

This host-local limitation prevents the datastore from providing the shared storage required for features such as vSphere High Availability and vMotion in the context of this lesson. A VM whose files exist only on one host's local disks cannot simply be restarted or migrated by another host that cannot see those files.

Fibre Channel Storage

Fibre Channel (FC) is a high-speed storage-area network, or SAN, technology. A SAN is a dedicated network architecture used to provide shared block storage to servers.

FC transports encapsulated SCSI commands between Fibre Channel nodes. An ESXi host sends storage requests through its Fibre Channel connectivity, and the storage array responds through the SAN. The host generally accesses a logical storage unit supplied by the array as block storage.

ESXi hosts require suitable Fibre Channel host bus adapters (HBAs) to connect to FC storage. An HBA is a host adapter that provides the server-side connection to Fibre Channel storage. The environment also needs compatible SAN connectivity and storage-array configuration.

FC is commonly selected when multiple ESXi hosts need access to the same block-storage system. If the same datastore is presented correctly to multiple hosts, those hosts can use the shared storage needed by capabilities such as High Availability and vMotion.

FC example

In a shared FC design, each ESXi host has one or more FC HBAs connected through the Fibre Channel fabric to a storage array. The array presents shared block storage to the hosts. ESXi creates or mounts a datastore on that shared storage, and virtual machines on the datastore can be accessed by the hosts that have the required storage visibility.

Fibre Channel over Ethernet

Fibre Channel over Ethernet (FCoE) carries Fibre Channel frames over Ethernet links. It allows Fibre Channel traffic and ordinary Ethernet traffic to share the same Ethernet infrastructure or physical link when the environment is designed to support both traffic types.

FCoE is not the same as native Fibre Channel. Native FC uses Fibre Channel-specific connectivity, while FCoE uses Ethernet to transport the Fibre Channel frames. FCoE is also different from iSCSI: iSCSI encapsulates SCSI commands and data in TCP/IP packets, whereas FCoE transports Fibre Channel frames and does not use TCP/IP as its storage protocol.

FCoE can reduce the number of separate physical adapters and cables in some data-center designs, but it requires compatible converged network adapters, switches, and storage infrastructure. Its shared-storage behavior is based on the Fibre Channel storage presented through the Ethernet transport.

iSCSI Storage

iSCSI carries SCSI control and data over TCP/IP networks. SCSI is a family of storage command interfaces; iSCSI packages those commands into TCP/IP traffic so an ESXi host can reach a remote storage target through an IP network.

An iSCSI design commonly includes an ESXi initiator, an IP network, and a storage target. The initiator is the ESXi-side identity or component that requests storage access. The target is the storage-side endpoint that provides the storage device or logical unit.

Because iSCSI uses standard TCP/IP networking, it is a networked block-storage option. It does not require a Fibre Channel SAN, although administrators should still design appropriate network connectivity, performance, redundancy, and isolation for storage traffic.

iSCSI example

An ESXi host uses an Ethernet adapter and an IP storage network to connect to an iSCSI target on a remote array. The target presents a block-storage device. ESXi uses that device for a datastore, and virtual machine virtual disks are stored there. If multiple hosts are given compatible access to the same storage, the datastore can serve as shared storage.

Network-Attached Storage

Network-attached storage (NAS) is file-level shared storage delivered over a standard TCP/IP network. Instead of presenting a raw block device to ESXi, a NAS system exposes a file system or file share.

ESXi commonly accesses NAS through Network File System (NFS). NFS is a file-sharing protocol that lets ESXi mount a remote file system supplied by a NAS system. The resulting NFS datastore contains virtual machine files, including virtual disks.

NFS is different from FC, FCoE, and iSCSI because it is file-based rather than block-based:

  • FC provides SCSI-based block storage through a Fibre Channel SAN.
  • FCoE provides Fibre Channel frames over Ethernet and remains a Fibre Channel storage approach.
  • iSCSI provides SCSI-based block storage over TCP/IP.
  • NFS provides file-level access to a NAS file system over TCP/IP.

NFS example

A NAS appliance exports an NFS file system over an IP network. An ESXi administrator mounts that export as an NFS datastore. Virtual machine configuration files and virtual disk files are stored as files on the NAS. Other ESXi hosts can mount the same NFS datastore when the NAS export and permissions are configured for them.

Datastores and Virtual Disks

A virtual machine's operating system, applications, and data are stored in its virtual disks. A virtual disk is generally a large file held on a datastore. ESXi presents that file-backed disk to the VM as a SCSI disk attached to a virtual SCSI controller.

Because a virtual disk is a file, administrators can manage it using file-oriented operations supported by the virtualization platform and storage design. It can be copied, moved, deleted, or archived as a virtual machine disk file, subject to application state, snapshots, permissions, and other operational considerations.

The virtual disk does not normally expose the physical disk or network protocol to the guest. For example, the same guest configuration can use a virtual SCSI disk backed by local SATA, FC, FCoE, iSCSI, or NFS storage. The guest sees the virtual device; ESXi handles the path to the datastore.

Guest storage view

Guest operating system
        |
Virtual SCSI controller
        |
Virtual disk file on a datastore
        |
Physical disk, SAN block device, or NFS file system

The guest may be able to report details about its virtual controller and virtual disk, but it normally cannot identify the complete underlying path from the VM to the physical storage system.

Shared Storage and vSphere Features

The most important architecture question is whether the storage is available to one ESXi host or shared among the hosts that need to run the VM.

  • Host-local storage: A datastore backed only by local DAS is normally visible to one host. It is suitable for workloads that do not require another host to access the same VM files.
  • Shared storage: A datastore presented to multiple hosts through FC, FCoE, iSCSI, or NFS can support coordinated access to VM files when correctly configured.

Shared storage is necessary for vSphere High Availability and vMotion in the context of this lesson. High Availability needs another host to access the VM's files if the original host fails. vMotion needs the destination host to access the VM's storage while the workload is moved between hosts.

ESXi Storage Technology Comparison

Technology: Direct-attached storage. Storage model: Block-style local disk access. Connection or transport: Direct SAS or SATA connection. Requirement: Local storage controller or direct disk connection. Shared across hosts: Usually no. Consideration: No separate storage network, but host-local storage cannot provide the shared storage required by High Availability and vMotion.

Technology: Fibre Channel. Storage model: Shared block storage. Connection or transport: Fibre Channel SAN carrying encapsulated SCSI commands. Requirement: FC HBA, SAN connectivity, and compatible storage array. Shared across hosts: Yes, when presented to multiple hosts. Consideration: Requires FC infrastructure and correct SAN configuration.

Technology: Fibre Channel over Ethernet. Storage model: Shared block storage. Connection or transport: Fibre Channel frames over Ethernet. Requirement: Compatible converged adapters, Ethernet infrastructure, and storage system. Shared across hosts: Yes, when presented to multiple hosts. Consideration: Not native FC and not TCP/IP-based iSCSI; requires compatible converged networking.

Technology: iSCSI. Storage model: Shared block storage. Connection or transport: SCSI control and data in TCP/IP packets. Requirement: Ethernet adapters, IP network, and iSCSI target. Shared across hosts: Yes, when configured for multiple hosts. Consideration: Storage performance and resilience depend on the IP network and target design.

Technology: NAS using NFS. Storage model: Shared file storage. Connection or transport: NFS over TCP/IP. Requirement: Ethernet network and NAS system exporting an NFS file system. Shared across hosts: Yes, when the export is available to multiple hosts. Consideration: ESXi accesses files rather than a raw block device.

Choosing a Storage Architecture

  1. Determine whether the VM must move between hosts or restart on another host after a host failure.
  2. If those capabilities are required, plan storage that the relevant ESXi hosts can share.
  3. Choose a transport based on existing infrastructure, required performance, operational skills, and resilience goals.
  4. Provide the correct host adapters or network adapters for the selected technology.
  5. Create or mount datastores on the available physical storage.
  6. Place the VM's virtual disks and other required VM files on the appropriate datastore.

Storage paths through host adapters or network adapters are normally transparent to VM applications and guest operating systems. The important administrative distinction is not what the guest sees, but how ESXi reaches the datastore and whether the datastore is available to the hosts that need it.

Troubleshooting Storage Design Problems

vMotion or High Availability is unavailable

Symptom: A VM stored on host-local disks cannot use vMotion or High Availability.

Likely cause: The datastore is backed by DAS that only one ESXi host can access.

Resolution: Use a datastore backed by storage shared among the relevant hosts, such as correctly presented FC, FCoE, iSCSI, or NFS storage.

Fibre Channel storage is not visible

Symptom: An ESXi host cannot access the expected FC storage.

Likely cause: The required Fibre Channel HBA or SAN connectivity is absent, unavailable, or incorrectly configured.

Resolution: Verify that the host has suitable FC HBAs and that the SAN and storage array present the intended storage to that host.

Storage is called shared, but only one host can access it

Symptom: An administrator expects a datastore to be shared, but other hosts cannot see it.

Likely cause: The storage is direct-attached to one host, or the networked storage has not been presented and mounted for the other hosts.

Resolution: Distinguish local DAS from SAN and NAS architectures, then verify access from every host that must use the datastore.

The guest is expected to identify the storage protocol

Symptom: An administrator expects the VM guest to report whether its disk comes from FC, FCoE, iSCSI, NFS, or local storage.

Likely cause: The ESXi storage abstraction has been misunderstood.

Resolution: Remember that the guest normally sees a virtual SCSI disk. ESXi manages the underlying datastore and storage path outside the guest's normal view.

Key Takeaways

  • ESXi abstracts physical storage and presents virtual machines with virtual disks.
  • A datastore is the ESXi storage container that holds VM files and virtual disk files.
  • DAS uses local or directly connected SAS and SATA storage and normally belongs to one host.
  • FC, FCoE, and iSCSI provide networked block storage.
  • FCoE transports Fibre Channel frames over Ethernet, while iSCSI transports SCSI over TCP/IP.
  • NAS provides file-level shared storage, commonly through NFS.
  • A virtual disk is a file on a datastore and appears to the guest as a SCSI disk.
  • Shared storage is required for vSphere High Availability and vMotion in the architecture described here.
  • The guest operating system normally does not need to know the underlying storage technology.

For a concise reference, see Storage Technologies for ESXi.