APIs: Concepts, Design, Requests, Responses, and Integration
Webhooks: Building Secure Event-Driven API Integrations
Learn how webhooks work, compare them with polling, and build secure, reliable, testable webhook receivers with retries, signatures, queues, and observability.
A webhook is an event-triggered HTTP request sent from one system to another. It is a callback: instead of repeatedly asking whether something changed, an application receives a notification when the change occurs. Webhooks are a common way to connect payment systems, source-control platforms, commerce applications, and other APIs.
This lesson explains the complete webhook lifecycle, from subscription and signature verification to asynchronous processing, retries, testing, monitoring, and safe endpoint changes. It assumes familiarity with HTTP requests and responses, JSON, backend routes, environment variables, and basic queues.
Webhook Concepts and Roles
The system that emits an event is the provider. The application that receives and processes it is the consumer or subscriber. An event is a recorded occurrence, such as an order being created or a payment succeeding. The event data is carried in the request payload, usually as JSON.
- Provider: Detects an event and sends the webhook request.
- Consumer: Receives, authenticates, validates, and processes the delivery.
- Event type: A label such as
payment.succeededororder.updated. - Callback URL: The public endpoint address registered with the provider.
- Delivery ID: A unique identifier for one attempted delivery. It supports tracing and deduplication.
Webhooks are an event-driven integration mechanism. The provider initiates communication when an event occurs, so the consumer does not need to request the same state repeatedly.
How Webhook Delivery Works
- The consumer creates or configures a publicly reachable HTTPS callback URL.
- The consumer registers that URL with the provider and selects event types to receive.
- The provider detects a subscribed event and creates an HTTP request.
- The provider adds headers and a payload, often including an event ID, event type, timestamp, delivery ID, and signature.
- The consumer receives the request, verifies its authenticity, validates the payload, and records the delivery.
- The consumer durably accepts the work, commonly by writing to a database or queue.
- The consumer returns a successful HTTP status promptly.
- If the request times out or receives a failure status, the provider may retry according to its documented policy.
A successful response means the consumer has accepted responsibility for the delivery. It should not mean that every downstream task has already finished.
Provider detects event
|
| signed HTTPS POST
v
Consumer endpoint -> verify -> record delivery -> enqueue work -> 202 Accepted
|
v
Worker processes eventWebhooks Versus Polling
Polling means that a client repeatedly sends requests to ask whether new data or state changes exist. A webhook reverses the direction: the provider sends a request when it has an event.
Use webhooks when low-latency notification matters and the provider offers dependable delivery. Use polling when no webhook exists, when periodic reconciliation is sufficient, or when the consumer needs an authoritative full-state scan. A hybrid design is often strongest: use webhooks for prompt updates and polling for reconciliation after outages or suspected gaps.
Webhook Request Structure
A typical delivery is an HTTP POST to a callback URL:
POST /webhooks/provider-event
Content-Type: application/json
Provider-Signature: <signature>
Provider-Timestamp: <timestamp>
Provider-Delivery-Id: <delivery-id>
{
"id": "evt_123",
"type": "payment.succeeded",
"created_at": "2026-08-25T12:00:00Z",
"data": { "order_id": "order_42", "amount": 2500 }
}Providers use different header names, signature formats, and JSON schemas. Treat every field and header as untrusted external input. Validate types, required fields, allowed event types, identifiers, timestamps, and maximum sizes according to the provider's documentation.
Creating a Secure Webhook Receiver
- Choose a dedicated route and accept only the expected HTTP method.
- Apply transport and request controls before expensive work: HTTPS, body-size limits, rate limits, and authorization rules.
- Read the raw request body, meaning the exact bytes received. Signature schemes often sign those bytes, not a parsed and reserialized JSON object.
- Verify the provider signature before trusting or acting on payload contents.
- Check the timestamp and replay tolerance when the provider supplies one.
- Parse JSON after verification, then validate its schema and event type.
- Use the event-type field to route the event to the appropriate handler.
- Record the delivery ID with a uniqueness constraint and enqueue nontrivial work.
- Return a success response promptly after durable acceptance.
rawBody = readRawBody(request)
verifySignature(rawBody, request.headers, signingSecret)
event = parseJson(rawBody)
validateEventSchema(event)
if deliveryAlreadyRecorded(event.deliveryId): return 200
persistDelivery(event.deliveryId, event.type)
enqueue(event)
return 202Event Processing Design
Webhook delivery is commonly at least once: an event may be delivered more than once. Exactly-once behavior should not be assumed. A provider may retry because a response was lost even though the consumer completed the work.
- Store the delivery ID and processing state durably.
- Make handlers idempotent: processing the same delivery again must not produce an unintended second charge, shipment, email, or deployment.
- Use a queue and background worker for slow or failure-prone work.
- Return success only after the event has been durably recorded or queued. Returning success before durable acceptance can lose the event.
- Expect out-of-order events. Compare event timestamps or sequence numbers, ignore stale updates, or retrieve current authoritative state from the provider API.
- Partition related work by resource when ordering matters, while still handling missing or duplicate events.
For example, if an order.updated event arrives before order.created, the consumer can inspect the event version, place the update in a waiting state, or fetch the current order before applying it.
Webhook Security
For HMAC, compute the message authentication code over the provider-defined signed message, often a combination of timestamp and raw body. Compare the received and calculated values using a constant-time comparison. For asymmetric signatures, validate the signature using the provider's trusted public key and its documented key-rotation process.
IP allowlists can provide supplemental protection, but they are not a replacement for signatures. Provider IP ranges can change, and requests may pass through proxies. Use authorization controls, network filtering, and WAF rules as additional layers.
WEBHOOK_SIGNING_SECRET=<stored-secret>
WEBHOOK_REPLAY_TOLERANCE_SECONDS=300
WEBHOOK_MAX_BODY_BYTES=1048576
WEBHOOK_QUEUE_NAME=webhook-eventsHTTP Responses and Provider Behavior
Common acknowledgment responses are 200 OK, 202 Accepted, and 204 No Content. Follow the provider's documented timeout, status-code, and retry behavior rather than assuming all providers interpret responses identically.
Retries, Failures, and Dead-Letter Handling
Providers often use a limited retry schedule with exponential backoff, which increases the delay between attempts. A timeout, connection failure, 429, or 5xx response is generally transient. A malformed payload, invalid signature, or unsupported event may be permanent and should not be blindly retried forever.
- Record delivery ID, event type, attempt number, timestamps, response status, and a safe failure reason.
- Use bounded retries with backoff in your own worker.
- Move repeatedly failing messages to a dead-letter queue, a durable holding location for manual review.
- Provide a controlled replay tool after correcting the root cause.
- Ensure replay remains idempotent and is authorized.
Testing and Local Development
Begin with provider sample payloads and test events. Use a staging endpoint and a staging signing secret separate from production. When a provider requires a public callback, run the local server on a chosen port and expose it through a secure HTTPS tunneling tool. Register the temporary HTTPS URL as a test callback and send provider test events.
- Verify a known-valid signature.
- Modify one byte of the body and confirm verification fails.
- Test an incorrect secret, stale timestamp, malformed JSON, oversized body, and unsupported event type.
- Simulate timeouts, 429 responses, 5xx responses, retries, and duplicate delivery IDs.
- Send out-of-order events and concurrent deliveries for the same resource.
- Confirm that sensitive fields and secrets are absent from logs.
Observability and Operations
Useful logs include delivery IDs, event types, correlation IDs, verification outcomes, response statuses, attempt numbers, and processing results. Do not log signing secrets, authorization headers, or unnecessary sensitive payload fields.
- Track delivery volume and failure rate.
- Measure retry rate, end-to-end processing latency, and queue depth.
- Measure duplicate-event rate and dead-letter volume.
- Alert on sustained signature failures, rising 4xx or 5xx rates, unusual timeouts, and growing backlogs.
- Document endpoint ownership, subscribed event types, environment-specific URLs, secret rotation, replay controls, and recovery procedures.
Webhook Lifecycle Management
When the provider offers subscription-management endpoints, support the full lifecycle: create, list, update, disable, and delete subscriptions. Keep a record of the provider subscription ID, endpoint URL, environment, event types, schema version, owner, and current status.
Prefer versioned payload schemas or event versions when available. During endpoint migration, register the new URL, verify it with test deliveries, and use overlapping subscriptions or dual acceptance only when duplicate handling is already safe. Monitor both endpoints before disabling the old one.
For secret rotation, temporarily accept signatures made with both the old and new secrets when supported. Record which key verified each delivery, update the provider, confirm successful deliveries with the new key, and remove the old secret after the overlap period.
Practical Examples
Payment Confirmation
A payment service sends payment.succeeded to an order system. The receiver verifies the signature, stores the delivery ID, queues fulfillment, and returns 202 Accepted. If the provider retries after a timeout, the unique delivery record prevents a second shipment or charge.
Source-Control Notification
A repository host sends a push event when commits reach a branch. The deployment service verifies the request, checks the branch name, stores the delivery ID, and queues one build for that delivery.
Duplicate Delivery
If a provider times out after the consumer has stored and queued an event, the retry is expected. The consumer recognizes the existing delivery ID and returns success without repeating the business action.
Troubleshooting Webhooks
Repeated Timeout Failures
Common causes include slow work before the response, an unreachable endpoint, or a proxy timeout that is shorter than the provider's limit. Inspect access logs and provider delivery records. Return a 2xx response after durable queueing, move long-running work to a worker, and verify routing, load-balancer, and proxy timeouts.
Every Request Fails Signature Verification
Check the secret, environment, signed-message format, and exact raw bytes. Parsed or reserialized JSON may differ from what the provider signed. Compare the implementation with provider documentation, preserve raw bytes, and never log the secret.
A Business Action Happens Twice
The provider may have retried after a timeout or 5xx response, or a worker retry may not be idempotent. Enforce durable uniqueness for delivery IDs and make downstream operations idempotent. Separate acknowledgment from background processing.
Valid Events Are Rejected as Too Old
Check server clock synchronization, replay-window configuration, and provider delivery delay. Use an appropriate documented tolerance without removing replay protection.
Events Arrive Out of Order
Do not assume provider ordering or sequential worker execution. Use timestamps, sequence numbers, or resource versions; partition related jobs when necessary; or retrieve authoritative current state from the provider.
Local Works but Production Does Not
Check that the production URL is publicly reachable, its TLS certificate is valid, DNS and routing are correct, and firewalls or WAF rules allow the provider. Inspect edge logs and verify the production event subscription and environment.
Exam- and Interview-Relevant Notes
- Webhooks are provider-initiated; polling is client-initiated.
- At-least-once delivery means duplicate deliveries are normal, not exceptional.
- Verify the signature against the raw request body before parsing or acting on data.
- Acknowledge quickly after durable acceptance; use a queue for slow work.
- Use idempotency, delivery-ID deduplication, and stale-event handling.
- Use 5xx or timeout behavior carefully because it can trigger retries.
- HTTPS, signatures, timestamp checks, secret protection, validation, limits, and redacted logs work together as layered defenses.
For related API integration concepts, see the API reference and return to this webhook guide when designing a receiver.