VMware ESXi and vSphere Cluster Management
VMware vSphere HA Datastore Heartbeating Explained
Learn how vSphere HA datastore heartbeating distinguishes ESXi host failure from management-network isolation and how to configure heartbeat datastore selection.
Datastore heartbeating is a vSphere High Availability (HA) mechanism that gives HA another source of host-liveness information when management-network communication is unavailable. It works alongside management-network heartbeats rather than replacing them.
This distinction matters because an ESXi host can lose management connectivity while continuing to run virtual machines and access shared storage. Without additional evidence, HA could incorrectly treat that host as failed and attempt unnecessary VM restarts.
Prerequisites and Key Terms
You should understand ESXi hosts, vCenter Server, clusters, shared datastores, VMkernel networking, and basic vSphere HA operation. The following terms are central to this lesson:
- vSphere HA: The VMware availability service that detects host failures and restarts affected virtual machines on other eligible cluster hosts.
- Management-network heartbeat: Network-based communication used by HA hosts to report connectivity and liveness within the cluster.
- Datastore heartbeating: Liveness evidence recorded through shared datastore access and used when management-network heartbeats are unavailable.
- Heartbeat datastore: A datastore selected by vSphere HA as a location through which datastore heartbeat information can be observed.
- HA coordinator host: The host that coordinates HA state and recovery decisions. Older material may call this the master host.
- HA agent host: A cluster member participating in HA monitoring. Older material may call this a slave host.
- Host failure: A condition in which an ESXi host is no longer operating sufficiently for HA to preserve its running workloads.
- Host isolation: A condition in which an ESXi host loses relevant management-network connectivity but may still be running VMs and accessing storage.
- Host isolation response: The configured action an isolated host takes for its virtual machines.
Purpose of Datastore Heartbeating
HA hosts normally exchange management-network heartbeats. These messages help the HA coordinator host determine whether other HA agent hosts remain reachable. A lost network heartbeat is an important warning, but it does not by itself prove that the remote host has powered off.
When the coordinator stops receiving management-network heartbeats from a host, HA gathers additional evidence through configured heartbeat datastores. The affected host can record heartbeat information through shared storage, and the coordinator can observe that activity.
If datastore heartbeat activity continues, the host may still be powered on and functioning even though management connectivity is unavailable. This evidence supports an interpretation of host isolation rather than an immediate conclusion that the host has failed.
If HA finds neither management-network heartbeat evidence nor datastore heartbeat evidence, that absence can contribute to a host failure classification. HA uses multiple signals and configured policies, so datastore heartbeating is one part of the decision rather than an independent recovery policy.
How HA Evaluates Host State
- Normal operation: HA hosts exchange management-network heartbeats and maintain access to their configured storage.
- Network heartbeat loss: The HA coordinator stops receiving management heartbeats from an HA agent host.
- Additional evidence gathering: HA evaluates the configured heartbeat datastores and other available host-state information.
- Isolation interpretation: Continued datastore heartbeat activity indicates that the host may still be running despite management-network loss.
- Failure interpretation: If the host provides no usable management or datastore heartbeat evidence, HA can determine that the host has failed, subject to its policies and other signals.
- Recovery action: If HA determines that a host failed, it initiates recovery of affected VMs on surviving eligible hosts.
| Condition | Management-network heartbeats | Datastore heartbeat evidence | Likely HA interpretation | Operational consequence |
|---|---|---|---|---|
| Normal connectivity | Present | May be present | Host is reachable and participating | No host-failure recovery is required |
| Management network interruption with storage access | Absent or unavailable | Present | Host may be isolated but still running | Isolation response and VM behavior become important |
| Total host outage | Absent | Absent | Host may have failed | HA can begin VM recovery on remaining eligible hosts |
| Insufficient or unavailable storage evidence | Absent | Unavailable or inconclusive | HA has less evidence for classification | Storage and network conditions must be investigated |
The precise result depends on HA implementation details, host state, storage accessibility, timing, and configured policies. Do not interpret datastore heartbeat evidence as a guarantee that every VM on the host is healthy.
Effect on Virtual Machine Recovery
When HA determines that an ESXi host has failed, it attempts to restart affected virtual machines on surviving hosts that are eligible under cluster configuration and admission control. Recovery can be delayed or prevented if there is insufficient capacity, inaccessible VM storage, incompatible host configuration, or another placement constraint.
An isolation scenario is different. The isolated host may still be running its VMs. The outcome is influenced by the configured host isolation response and by the behavior of the VMs themselves. Depending on the configuration, the isolated host may leave VMs running, power them off, or shut them down so that another host can restart them.
Accurate classification matters in both directions:
- An incorrect failure interpretation can cause unnecessary VM restart attempts while the original host is still running workloads. This can create conflicts, service disruption, or operational uncertainty.
- An incorrect isolation interpretation can delay recovery when the host is genuinely unavailable, extending application downtime.
Datastore heartbeating does not replace review of HA admission control, host isolation response, VM restart priority, or the broader vSphere HA configuration.
Heartbeat Datastore Requirements
A suitable heartbeat datastore must be accessible to the relevant hosts in the HA cluster. Shared access is important because the host being evaluated and the HA coordinator must be able to use or observe the same storage location through their respective storage paths.
Good candidates generally have these properties:
- Consistent host access: The datastore is mounted and accessible from the HA hosts that need to use it.
- Resilient connectivity: Storage paths, fabrics, network storage connections, and host adapters do not depend on one fragile component.
- Reliable availability: The datastore is not routinely removed, unmounted, or placed behind unstable storage access.
- Appropriate redundancy: The design includes storage and path redundancy suitable for the cluster’s failure requirements.
- Operational independence: The selected storage does not share a single failure point with the management network or with every other heartbeat candidate.
HA generally uses a limited set of candidate datastores rather than every datastore visible in the environment. The available candidate set can change when a datastore becomes inaccessible, a host is added or removed, storage presentation changes, or a datastore is retired.
Do not assume that a datastore visible from one host is automatically suitable for the whole cluster. Validate access from all intended HA hosts.
Datastore Heartbeating Selection Policies
vSphere HA provides policy choices that balance administrative control with automatic resilience. The exact labels can vary by vCenter Server version, but the concepts are consistent.
| Policy | Who selects datastores | Fallback behavior | Administrative control | Resilience considerations | Best-fit use case |
|---|---|---|---|---|---|
| Automatic selection | HA selects appropriate accessible datastores | HA can select from eligible accessible candidates as availability changes | Lowest manual control | Usually provides broadest ability to adapt to storage changes | General clusters where accessible shared storage is already well designed |
| Specified datastore list only | Administrators define the allowed list | No automatic replacement outside the specified list if selected storage becomes unavailable | Highest control over the storage locations used | Can reduce resilience if the list is too small or a selected datastore has a common failure point | Environments with strict storage governance and a deliberately maintained approved set |
| Preferred list with automatic complement | Administrators nominate preferred datastores and HA can select additional accessible datastores when needed | HA can use suitable alternatives if preferred choices disappear | Balances preference with automatic adaptation | Retains fallback capability while favoring resilient or preferred storage | Clusters that want control over primary candidates without sacrificing availability |
Automatic Datastore Selection
With automatic selection, HA chooses appropriate datastores that are accessible from the cluster hosts. This is often the simplest operational choice because HA can adapt when storage availability changes.
Automatic selection does not remove the need for storage design. If host access is inconsistent or all candidates share one storage failure domain, automatic selection cannot create redundancy that the infrastructure does not provide.
Specified Datastore List Only
This policy restricts HA to administrator-selected datastores. It is useful when organizational or architectural requirements demand a controlled storage set.
The trade-off is important: if a selected datastore becomes inaccessible, HA cannot freely choose an unlisted datastore under this policy. A small list can therefore become a single point of reduced evidence, especially when the remaining cluster storage is healthy but excluded.
Preferred List with Automatic Complement
This policy lets administrators nominate preferred datastores while permitting HA to select other accessible cluster datastores when the preferred choices are unavailable. It provides a practical balance between control and resilience.
For example, an administrator might prefer two highly resilient shared datastores but allow automatic fallback to other suitable datastores during maintenance or a storage failure.
Configuration Workflow in the vSphere Client
Use the following conceptual workflow. Menu labels can vary slightly between vCenter Server and vSphere Client versions.
- Open the vSphere Client and navigate to the target cluster in the inventory.
- Open the cluster Configure or Settings area. Some versions expose this through a Manage or Settings view.
- Open the vSphere HA configuration and choose Edit.
- Locate the Datastore Heartbeating section.
- Choose the required policy: automatic selection, specified datastore list only, or preferred list with automatic complement.
- If the selected policy requires a list, choose the preferred or allowed datastores and confirm that they are shared by the intended HA hosts.
- Save the configuration.
- Return to the cluster HA configuration and verify the resulting policy and selected heartbeat datastores.
The configuration path can be summarized as:
Cluster inventory object > Configure or Manage/Settings > vSphere HA > Edit > Datastore HeartbeatingNo command-line configuration is required for the core workflow. Use the vSphere Client so that the policy and datastore selections are visible and reviewable with the rest of the cluster HA configuration.
Monitoring and Validation
After configuring datastore heartbeating, review the cluster’s vSphere HA settings and confirm which policy is active and which datastores are selected or eligible.
Validate each candidate datastore from the perspective of the intended HA hosts:
- Confirm that the datastore is mounted and accessible on each relevant host.
- Check storage adapters, paths, zoning, network storage connectivity, and datastore health.
- Confirm that the datastore remains eligible under the selected heartbeat policy.
- Check for storage alarms, path failures, unmount operations, and recent access-control changes.
- Review HA events and host connectivity states after storage or management-network changes.
Review datastore heartbeating after cluster expansion, datastore retirement, storage migration, zoning changes, host replacement, or changes to host storage presentation. A configuration that was adequate for an earlier cluster layout may become too restrictive after infrastructure changes.
Practical Scenarios
Management Network Interruption with Continued Storage Access
An ESXi host loses management-network communication with the HA coordinator but remains powered on and continues writing datastore heartbeat information. The continued datastore evidence supports the conclusion that the host may be isolated rather than failed.
Administrators should still investigate the management path and review the configured host isolation response. Datastore activity is evidence of host liveness, not proof that every management or application function is healthy.
Actual Host Outage
An ESXi host powers off unexpectedly. It produces neither management-network heartbeats nor datastore heartbeat activity. HA can classify the host as failed and begin VM recovery on remaining eligible hosts, assuming storage and capacity requirements are satisfied.
Specified List Loses a Selected Datastore
A cluster is restricted to a small approved heartbeat datastore list, and one selected datastore becomes inaccessible. Under a specified-list-only policy, HA cannot freely use an unlisted datastore. This can reduce the amount of liveness evidence available even if another shared datastore is healthy.
Restore access or revise the allowed list after validating that replacement storage is shared and accessible across the intended hosts.
Preferred Datastores with Automatic Fallback
An administrator prefers two highly resilient shared datastores but permits automatic use of other accessible cluster datastores if those preferred choices disappear. This approach preserves administrative preference while retaining a fallback path during storage maintenance or failure.
Storage Maintenance Review
Before retiring a datastore or changing host storage presentation, determine whether it is used or eligible for HA datastore heartbeating. Confirm that remaining shared datastores are accessible to all relevant hosts before completing the change.
Troubleshooting
HA Cannot Use an Expected Datastore
- Confirm that the datastore is mounted and accessible from the affected cluster hosts.
- Check host storage adapters, paths, zoning, network storage connectivity, and datastore health.
- Confirm that the datastore remains eligible under the selected heartbeat datastore policy.
- Review recent storage maintenance, unmount operations, host additions, and access-control changes.
A Manually Selected Datastore Becomes Unavailable
- Determine whether the active policy permits automatic fallback.
- With specified-list-only selection, restore access or revise the allowed datastore list.
- Validate that replacement datastores are shared and accessible across the intended HA hosts.
A Host Is Reported as Disconnected or Isolated
- Separate management-network connectivity loss from a true ESXi host outage.
- Review HA events and host status alongside datastore accessibility.
- Check management VMkernel connectivity, physical switching, VLAN configuration, uplinks, and redundant paths.
- Review the configured host isolation response before taking recovery actions.
Unexpected VM Restart Behavior During a Network Event
- Review HA failure and isolation events to determine the classification made by HA.
- Verify datastore heartbeat availability at the time of the event.
- Review host isolation response, admission control capacity, and VM restart priority.
- Assess whether restrictive heartbeat datastore selection contributed to insufficient evidence.
Datastore Heartbeating Compared with Related HA Features
| Feature | What it monitors | Primary purpose | Typical trigger | Relationship to datastore heartbeating |
|---|---|---|---|---|
| Management-network heartbeat | Network connectivity and host liveness | Normal HA host communication | Periodic HA communication | Datastore heartbeating provides additional evidence when this signal is unavailable |
| Datastore heartbeating | Host liveness through shared datastore access | Help distinguish isolation from host failure | Loss of management-network heartbeat | Focuses on ESXi host liveness, not guest application health |
| Host isolation response | Behavior of VMs on an isolated host | Define what the host does when management connectivity is lost | Host isolation condition | Influences VM behavior when HA interprets the event as isolation |
| VM Monitoring | VM and guest operating system activity | Respond to an unresponsive guest or application | Guest-level heartbeat or responsiveness failure | Separate feature; it does not determine whether an ESXi host is isolated or failed |
| Admission control | Available cluster failover capacity | Ensure the cluster can reserve resources for HA recovery | Cluster configuration or capacity evaluation | Influences whether recovered VMs can be placed after a host failure |
VM Monitoring is not datastore heartbeating. Datastore heartbeating evaluates evidence that an ESXi host is still active through shared storage. VM Monitoring addresses guest operating system or application-level responsiveness.
Design and Operational Recommendations
- Use resilient shared storage that is visible to the appropriate HA hosts.
- Prefer storage connectivity with redundant paths and carefully reviewed failure domains.
- Avoid manually restricting the heartbeat datastore list so tightly that one datastore failure removes most or all useful alternatives.
- Use specified lists only when the operational reason is clear and the selected datastores are actively maintained.
- Consider preferred-list-with-automatic-complement when you need to favor particular datastores but still want fallback.
- Review heartbeat selection after cluster expansion, datastore retirement, storage migration, zoning changes, or host access changes.
- Coordinate storage maintenance with HA configuration review and event monitoring.
- Evaluate datastore heartbeating together with admission control policy, isolation response, restart priority, and storage redundancy.
Exam-Relevant Summary
For broader cluster administration context, review communication between vCenter Server and ESXi, vSphere HA admission control, and browsing VMFS datastores.