VMware ESXi and vSphere Cluster Management

Webhooks: Event-Driven HTTP Notifications

Learn how webhooks work, how to receive and send them, and how to build secure, reliable, idempotent integrations with HTTP.

A webhook is an HTTP callback triggered by an event. An event producer detects something that happened, such as a successful payment or a new repository commit, and sends an HTTP request to an endpoint configured by a subscriber. The receiving application is the webhook consumer.

Webhooks provide push-based notifications: the producer contacts your application when an event occurs. With polling, your application repeatedly sends API requests asking whether anything has changed. Push notification usually reduces unnecessary requests and can deliver changes sooner, but it requires a reachable endpoint and careful handling of retries, security, and duplicates.

A webhook payload commonly contains event metadata and resource data. The payload may be minimal, stale, or intentionally limited. When the consumer needs authoritative or complete details, it can use the event's identifier to make a follow-up request to the provider's API.

How Webhooks Work

The main components are the producer or provider, the event, a webhook subscription, a delivery endpoint, the payload, the consumer, an HTTP response, and a retry mechanism.

1. An event occurs in the producer, such as payment.succeeded.
2. A webhook subscription matches the event and identifies the subscriber's endpoint.
3. The producer sends an HTTP POST request, usually containing JSON.
4. The endpoint checks transport security, request size, headers, timestamp, and signature before trusting the body.
5. The consumer records the delivery and places slow work on a durable queue.
6. The endpoint promptly returns a 2xx response to acknowledge acceptance.
7. If delivery fails, times out, or receives a retryable response, the provider may try again according to its own policy.

An inbound webhook is an event received by your application. An outbound webhook is an event your application sends to another system. In both cases, the receiving endpoint should authenticate the sender, validate the message, and process duplicate deliveries safely.

HTTP Request and Response Mechanics

Most webhook deliveries use HTTP POST with a JSON body. A request has a URL, method, headers, and body. Common headers or payload fields identify the event type, delivery ID, timestamp, content type, and signature.

POST /webhooks/provider HTTP/1.1
Host: app.example.com
Content-Type: application/json
X-Event-Type: payment.succeeded
X-Delivery-Id: evt_12345
X-Timestamp: 2026-08-25T12:00:00Z
X-Signature: sha256=<signature>

{"id":"evt_12345","type":"payment.succeeded","data":{"payment_id":"pay_987","status":"succeeded"}}

A prompt 2xx response means the endpoint accepted the delivery. Some consumers return 200 OK; others return 202 Accepted when work has been durably queued for asynchronous processing. A non-2xx response usually indicates failure and may cause redelivery. A timeout means the provider did not receive a response in time and may also retry.

Redirects are risky for webhooks. A provider may not follow them, or a redirected request may lose authentication context. Configure the exact HTTPS endpoint and return the response directly. Never assume that a 2xx response means the business operation has finished; it should mean that the event has been safely accepted.

Designing a Webhook Endpoint

1. Expose a publicly reachable HTTPS URL. DNS, the certificate, firewall rules, and ingress configuration must allow the provider to connect.
2. Route the URL to a dedicated webhook handler rather than a general-purpose form or API route.
3. Read the unmodified raw request body with a maximum size. Signature verification often depends on the exact bytes sent by the provider.
4. Obtain the required signature and timestamp headers, reject stale requests, and verify the signature before JSON parsing or transformation.
5. After authenticity checks, validate the content type, event type, schema, required fields, and tenant or account context.
6. Atomically record the delivery ID if it has not been seen, then enqueue durable background work.
7. Return a fast 2xx response after durable acceptance. Do not perform slow email, network, deployment, or complex database work in the request path.
8. Store delivery and processing state so operators can inspect failures, retry work, and recover from incidents.

read raw request body with a size limit
obtain timestamp and signature headers
reject stale timestamps
verify HMAC over the provider-required signed content
parse JSON
validate event type and schema
atomically record the delivery ID if unseen
enqueue durable background work
return HTTP 202 or 200
make worker actions idempotent

Configuration should keep endpoint paths, size limits, queue names, and secrets outside source code:

WEBHOOK_ENDPOINT_PATH=/webhooks/provider
WEBHOOK_SIGNING_SECRET=<stored-securely>
WEBHOOK_TIMESTAMP_TOLERANCE_SECONDS=300
WEBHOOK_MAX_BODY_BYTES=1048576
WEBHOOK_QUEUE_NAME=webhook-events

Event Payloads and Schemas

An event payload often has an envelope surrounding the resource data. The envelope describes the notification; the data object describes the affected resource.

• Event ID: a unique identifier for deduplication and tracing.
• Event type: a label such as order.created or payment.succeeded.
• Creation time: when the provider created the event, which may differ from delivery time.
• API or schema version: identifies the interpretation of fields.
• Account or tenant context: identifies the customer, workspace, or account affected.
• Data object: the resource snapshot or resource identifiers associated with the event.

Minimal payloads are smaller and expose less data, but they may require a follow-up API request. Expanded payloads are convenient for consumers but can become large and may represent a snapshot rather than the resource's current state. Design consumers to tolerate either choice when the provider allows it.

Schema evolution is normal. Treat optional fields as absent unless present, ignore unknown fields unless they are security-sensitive, and avoid assuming a fixed field order. Versioned event formats should be handled explicitly. Deploy readers that understand old and new formats before changing the producer, and use backward-compatible additions where possible.

Webhook Security

HTTPS encrypts the connection in transit and helps the consumer verify the server certificate. It does not, by itself, prove that the request came from the intended provider. Authentication and message validation are still required.

A common method uses a shared secret and HMAC, a keyed cryptographic hash. The provider computes a signature over specified content, often including a timestamp and the raw body. The consumer computes the same value and compares signatures using a constant-time comparison. Verify the exact raw bytes before JSON parsing, whitespace normalization, character decoding changes, or re-serialization.

• Reject timestamps outside a small tolerance to limit reuse of captured requests.
• Track event IDs or nonces so a previously valid request cannot be accepted repeatedly.
• Make the business operation idempotent as a second replay defense.
• Validate the HTTP method, content type, request size, expected event types, and payload structure.
• Keep secrets out of URLs, source control, client-side code, logs, and error responses.
• Use secret rotation procedures that allow an orderly overlap between old and new secrets when supported.

Other controls include bearer tokens, mutual TLS, and IP allowlists. Bearer tokens must be protected because possession grants access. Mutual TLS provides strong client certificate authentication but requires certificate lifecycle management. IP allowlists can reduce unwanted traffic, but addresses may change and filtering alone does not prove message integrity. Do not rely on an obscure URL or source IP instead of signature verification.

Reliability and Delivery Semantics

Most webhooks provide at-least-once delivery: an event should not be silently lost under normal retry behavior, but it may be delivered more than once. A timeout can occur after your server completed the operation but before the provider saw the response, so the provider may send the same event again.

Use idempotency keyed by an event ID or delivery ID. Store that identifier in a durable database table with a uniqueness constraint, and make downstream actions safe to repeat. Domain-level constraints are also useful: for example, an order should transition to paid once even if the payment event is delivered twice.

Providers commonly retry with exponential backoff, meaning the delay grows between attempts. Policies differ in delay, maximum attempts, and retry window. A durable queue and worker let your application retry internal processing separately from provider delivery. Events that remain unsuccessful can be placed in a dead-letter queue for investigation, correction, and manual replay when supported.

Delivery order is not always guaranteed. Parallel requests, retries, and network timing can make a newer event arrive before an older one. Use resource versions, event timestamps, conditional state transitions, or a follow-up API request. Periodic reconciliation or state synchronization with the provider API can find missing events and repair drift.

For important workflows, accept the event and update the delivery record transactionally before acknowledging it. A worker can then process the event from durable storage. This avoids losing work between sending the response and creating a queue message.

Operational Practices

Record enough metadata to diagnose delivery without leaking private data. Useful fields include delivery ID, event type, verification result, response status, processing outcome, queue identifier, and correlation ID. Redact tokens, signatures, payment details, credentials, and unnecessary payload fields.

Monitor delivery success rate, endpoint latency, retry volume, signature failures, duplicate rate, queue backlog, worker failures, and dead-letter volume. Alert on sustained non-2xx responses, unusual traffic, rising latency, abnormal signature failures, or a growing backlog.

• Restrict who can create, edit, disable, or replay webhook subscriptions.
• Maintain audit trails for endpoint changes, secret rotation, event selection, and manual replays.
• Use separate test and production endpoints, credentials, queues, and data.
• Migrate endpoints with an overlap period when the provider supports multiple destinations.
• Upgrade event or API versions deliberately, test fixtures against both formats, and deploy backward-compatible readers first.

Testing and Local Development

A request inspector can show headers, bodies, status codes, and timing. For local development, run the handler on a local port and use a secure HTTPS tunneling utility to forward a temporary public URL to that port. Configure that URL only on a test subscription and use test secrets.

Send signed fixture requests that match the provider's signing algorithm. Test the complete lifecycle, not just the successful path:

• Valid signature and valid JSON
• Invalid, missing, and stale signatures
• Malformed JSON and unexpected content types
• Oversized bodies and unsupported event types
• Duplicate event IDs and duplicate delivery attempts
• Slow workers, delayed responses, timeouts, and provider retries
• Out-of-order events and missing events requiring reconciliation
• Secret rotation and endpoint migration

Provider dashboards or command-line tools may show delivery attempts and allow supported events to be replayed. Replay tests should verify that the consumer produces no duplicate business effect.

Practical Integration Examples

Payment confirmation

A payment provider posts payment.succeeded to an order service. The service verifies the signature, records the delivery ID, marks the order paid once, queues receipt generation, and returns a 2xx response promptly.

Source control deployment trigger

A repository hosting service sends a push event to a deployment service. The handler validates the signature and event type, checks the target branch, and queues a build instead of running the deployment during the HTTP request.

Inventory synchronization

An e-commerce system receives product-update events from an inventory provider. It deduplicates event IDs, uses resource versions where available, and fetches current data from the provider API when events arrive out of order.

Customer relationship automation

A form platform posts a new-lead event to a CRM integration. The handler validates the request, normalizes fields, creates or updates a contact idempotently, and records failures for later review.

Webhooks Compared With Other Patterns

Choose webhooks when one system needs to notify another across an HTTP boundary without requiring a persistent connection. Choose polling when simplicity or provider limitations matter more than immediacy. Use WebSockets or server-sent events for continuously connected clients. Use a message queue or event stream for durable internal communication, high throughput, replay, consumer groups, or stronger control over delivery.

Troubleshooting Common Failures

Endpoint timeouts

Slow database, network, email, or deployment work may run before the response. The endpoint may also be unreachable because of DNS, firewall, certificate, or TLS problems. Validate reachability, inspect ingress logs, measure handler latency against the provider's timeout, and acknowledge after durable queuing.

Signature verification failures

The framework may have parsed or reformatted the body before verification. Other causes include the wrong secret, header, algorithm, timestamp format, or an incomplete secret rotation. Verify the exact raw byte body, confirm the provider's signing rules, and log only safe diagnostic metadata.

Duplicate business effects

A retry after a timeout or non-2xx response is normal. Store delivery IDs durably with a uniqueness constraint, return success after acceptance, and make downstream operations safe to repeat.

Unexpected event order

Retries and parallel delivery can delay older events. Use versions or timestamps, make transitions conditional and idempotent, fetch current resource state when needed, and run periodic reconciliation.

Unexpected or malicious traffic

Public endpoints are scanned. Reject requests without valid signatures before processing, enforce size and rate limits, validate schemas, and redact logs. IP filtering can supplement these controls but should not be the sole defense.

Exam-Relevant Notes

• A webhook is a push-based HTTP callback; polling is a consumer-initiated request pattern.
• At-least-once delivery means duplicate events are expected. Exactly-once business effects require idempotent consumer design.
• Verify signatures over the unmodified raw body, before JSON parsing or transformation.
• Return a prompt 2xx response after durable acceptance; move slow work to a queue and worker.
• Do not assume delivery order, current payload data, or provider-specific retry behavior.
• Use timestamps, event or nonce tracking, and idempotency to reduce replay risk.
• Monitor both delivery health and asynchronous processing health.

For a concise implementation checklist, configure a dedicated HTTPS endpoint, authenticate the raw request, validate its schema, atomically record its delivery ID, enqueue durable work, acknowledge quickly, process idempotently, monitor outcomes, and reconcile against the provider API when events may be missing or out of order.

Continue exploring webhook integrations as part of an event-driven architecture.