• Early attempts are dense because most failures are transient and clear in seconds.
• The window is bounded because an event nobody accepted in two days is stale anyway, and because unbounded retention of undelivered events is an unbounded storage and worker-capacity commitment.
• Retry only what's retryable. A 500, 502, 503, 429, or a connection timeout is retryable. A 400 or 422 means the receiver rejected the content — retrying identical bytes will fail identically, so it burns nine attempts to no purpose. A 410 Gone should disable the endpoint. Module F-8's rule about error shapes carrying a retryability signal is the same rule pointed outwards.
• Honour Retry-After on a 429. A receiver telling you their rate limit (Module P-18) is giving you information your backoff curve doesn't have.
• A timeout is ambiguous, not a failure. If you time out at 10 seconds and the receiver processed the event at 12, you will retry something that already happened. This is not fixable from your side — it is the precise reason the receiver must be idempotent, and why your docs must say so.

What retries cost: every retry is a duplicate delivery from the receiver's perspective. At-least-once is a choice you make on the receiver's behalf, and the honest thing is to document it loudly and give them the tool to deal with it — a stable event ID that does not change across retries of the same event. An event ID that is regenerated per attempt makes deduplication impossible and is a surprisingly common bug.

---

One Dead Endpoint Must Not Consume Your Delivery Capacity

Here is the outage shape that turns a webhook feature into a pager rotation. A single customer's endpoint starts black-holing connections — no reset, no 503, just silence until your timeout fires at 10 seconds. That customer generates a lot of events. Your delivery workers pull from one shared queue.

By Little's Law (Module F-4), with a 10-second hold time, each worker sustains 0.1 deliveries per second. Fifty workers give you five deliveries per second, total, for the entire platform. Events for every other customer sit in the queue behind a customer who isn't listening. Your delivery lag graph, which is the metric customers actually notice, goes vertical — and nothing is broken on your side.

This is head-of-line blocking, and it needs two mechanisms.

Per-endpoint circuit breaking (Module A-7). Track failures per endpoint. After a threshold of consecutive failures, open the breaker: stop attempting delivery, queue events for that endpoint only, and probe periodically with a single request. A dead endpoint then costs one in-flight request instead of an unbounded share of your workers.

Isolation of the queue itself. A circuit breaker limits time wasted per endpoint but doesn't stop one enormous customer from monopolising a shared queue. The pattern is queue-per-endpoint, or — because thousands of physical queues is its own operational problem — a partitioned queue where the partition key is the endpoint ID, with workers assigned to partitions. Kafka's partitioning (Module P-15) does exactly this, at the cost that a single hot partition can still lag while others are idle, and that the number of partitions bounds your parallelism. It is Module A-7's bulkhead pattern, applied to delivery capacity.

For fan-out at scale — one event that 5,000 subscribers want — the delivery is not one job, it is 5,000 jobs. Materialising them as 5,000 rows or messages per event is what makes the outbound tail visible and controllable. Storing the event once and letting a scanner discover subscribers at delivery time seems cheaper until you need per-subscriber retry state, which you always do.

Why this matters in production: the metric to alert on is not delivery failure rate, which a broken customer dominates and which is therefore permanently amber. It is per-endpoint queue depth and age of the oldest undelivered event. A single global lag number hides the case where 99% of endpoints are healthy and one is 400,000 events behind — and hides the reverse, which is the one that pages you.

---

The Ordering You Must Refuse to Promise

Retries and parallel workers together make ordering unachievable. Event A fails and is retried in 10 seconds. Event B, created a second later, succeeds immediately. The receiver sees B then A. Nothing is wrong; the system is working as designed.

Even without retries, N workers pulling from a queue reorder by definition, and two events created 5 ms apart can traverse different TCP connections with different retransmission luck. The only way to offer ordering is a single in-flight delivery per endpoint with strict head-of-line blocking — meaning one slow event stalls that endpoint's entire stream, which is usually worse than out-of-order arrival.

So the sender's obligation is to make out-of-order arrival survivable rather than to prevent it:

• A monotonic sequence or version per resource, so the receiver can detect and discard a regression.
• The event's own occurred_at timestamp, distinct from delivery time.
• Enough state in the payload that a receiver processing only the newest event per resource reaches the correct final state. An event saying "status is now shipped, version 7" is recoverable from out of order. An event saying "status changed" that requires a follow-up API read is a race against the next event.

Documenting "webhooks may arrive out of order" is not a disclaimer, it is a specification. The corollary Module P-11 makes precise: you are offering the receiver eventual consistency, so give them the version marker that makes convergence decidable.

---

Receiving: Verify, Then Queue-Then-Ack in Milliseconds

The single most consequential decision on the receiving side: your handler does no work. It verifies the signature, writes the event to durable storage or a queue, and returns 200. Everything else happens in a background worker (Module P-19).

Why inline work is a trap: your processing latency becomes the sender's timeout, and their timeout becomes your duplicate storm. Suppose the handler charges a card, writes three tables, and sends an email — 8 seconds on a good day. The sender's timeout is 10 seconds. On a bad day you take 11. The sender records a failure and retries. Your second handler starts the same work while the first is still running, so now you have two concurrent charge attempts and 16 seconds of work in flight. Retry three fires. Load rises, latency rises, more retries fire. This is a self-amplifying spiral driven entirely by the gap between your work and someone else's patience — and the sender is doing exactly what their documentation promised.

There is a second, quieter reason. A Node.js instance handles 200–1,000 req/s with one database query, and far less with eight. Webhook traffic is spiky by nature — a batch job on the sender's side emits 50,000 events in a minute. An endpoint that returns in 20 ms absorbs that with a handful of instances; one that takes 8 seconds needs Little's Law's worth of concurrency (400,000 request-seconds in that minute) and does not have it. Serverless makes this worse: Module F-10's rule that no work happens after the response returns means "return 200 then process" is not even available to you — the enqueue must complete before the response, or the work silently vanishes. Note the ordering in the snippet above: the durable INSERT happens before the 200, so the enqueue is a re-drivable pointer rather than the only record.

The 200 is a receipt for custody, not for processing. That distinction has to be in your own head, because it changes what a failure later means: a poisoned event now fails in your worker where you own the retry policy and the alert, instead of failing in the sender's delivery system where you can only watch.

---

Redelivery Is Normal, So Handling Must Be Idempotent

Every webhook receiver is a Module P-16 problem. Duplicates arrive for reasons you cannot influence: the sender's ambiguous timeout, their at-least-once queue, their operator manually replaying a batch, and their perfectly reasonable retry after your load balancer dropped a connection during a deploy.

The mechanism is the one from Module P-16 — a uniqueness constraint on the sender's event ID, enforced by the database rather than by a SELECT first (Module F-13's concurrency argument applies verbatim). The ON CONFLICT (event_id) DO NOTHING above is the whole trick, and it works only because the sender keeps the event ID stable across retries. If a sender regenerates IDs per attempt, you have to fall back to a natural key — (resource_id, version) or a content hash — and say so in your integration notes.

Two refinements specific to webhooks:

Deduplicate at ingest, and make the processing step idempotent too. The inbox insert stops duplicate storage. It does not stop a worker crashing after charging a card but before marking the row processed, so the worker's own effects need idempotency keys against downstream services.

Out-of-order arrival is decided by the event's version, not by arrival order. Arrival order is meaningless, as the sender's section established, so a "last write wins by arrival" update applies a stale event over a fresh one:

That predicate is the entire out-of-order defence, and it is one line. Without it, the sequence "shipped(v7) then delivered(v8)" arriving reversed leaves the order permanently shipped — a data bug with no error, no log line, and a customer support ticket six days later.

If you genuinely need to process a resource's events in order, buffer: hold event v8 until v7 has been applied, with a timeout after which you stop waiting and reconcile from the API instead. The cost is per-resource state and a stall risk, which is why the version predicate is the right default and buffering is the exception.

---

Polling Versus Webhooks, and Why You Need Reconciliation Anyway

Webhooks are not strictly better than polling; they trade a different set of costs.

The decisive line is the last one. Polling's failure mode is lateness, which is self-healing and easy to alarm on. A webhook's failure mode is absence, which produces no error anywhere: the sender exhausted its nine attempts during your two-hour deploy incident, dead-lettered the event, and moved on. Your system is simply, quietly wrong, and stays wrong.

So a webhook integration is only complete with a reconciliation path, in one of two forms:

1. You poll the sender's API periodically for resources changed since your last successful sync, and repair anything you missed. Low frequency — hourly, nightly — because it only has to catch gaps, not carry the load.
2. The sender exposes a listing endpoint for events in a time range, so you can replay a window after an incident: "give me all events between 14:00 and 16:00." If you are the sender, build this. It converts an incident from "we lost your events, sorry" into a customer-runnable repair, and it is the single most valuable thing in a webhook API after the signature.

The cost of reconciliation is real work: a scheduled job, a watermark to track, and code that must be idempotent — which it already is, from the previous section. Skipping it is a decision to accept silent divergence, and it should be written down as one rather than arrived at by omission.

---

User-Supplied Callback URLs Are an SSRF Primitive

The moment you let a customer type a URL that your servers will fetch, you have built a request proxy inside your network perimeter. This is server-side request forgery, and Module F-16 met it as the "shortened link resolves to an internal address" problem. Webhook endpoints are the same primitive with a worse blast radius, because delivery is authenticated, repeated, and carries data.

What an attacker points a callback URL at:

• http://169.254.169.254/… — cloud instance metadata, historically a path to credentials.
• http://127.0.0.1:6379/ or an internal hostname — Redis, Elasticsearch, an admin panel, anything reachable from your delivery workers and not from the internet.
• A hostname whose DNS resolves publicly at validation time and to 10.0.0.5 at delivery time — DNS rebinding, which defeats validate-then-fetch.
• A public URL that 302-redirects to an internal one, if your HTTP client follows redirects by default. It does.

The defences, and their costs:

• Scheme and port allowlist. HTTPS only, port 443. Cheap; blocks customers with unusual setups, which is usually acceptable.
• Resolve the hostname yourself, reject private ranges, then connect to the resolved IP — pinning the address you validated so the second lookup can't differ. This is the only real answer to rebinding, and it costs you a custom socket/agent instead of a plain fetch.
• Do not follow redirects. A webhook target that redirects is a misconfiguration; treat a 3xx as a delivery failure.
• Egress from an isolated network with no route to internal services and no metadata endpoint. This is the defence that holds when the application-level checks have a bug, and it is the one worth spending on.
• Verify domain ownership before enabling an endpoint — a challenge response at a known path. Costs onboarding friction; prevents a customer pointing deliveries containing their data at somebody else's server, which is a data-exfiltration route dressed as a configuration field.

Also treat the response as hostile: cap the body you read, cap the time you wait, and never surface the response body back to the customer in delivery logs. A verbose error message that echoes what the internal service replied turns a blind SSRF into a readable one. Module A-13 develops the general principle; the specific instance is that your delivery worker's network position is a capability you are lending out.

---

Where This Shows Up in Your Stack

• PostgreSQL: the inbox table with UNIQUE (event_id) and a partial index on unprocessed rows (WHERE processed_at IS NULL) is the whole receiver design — Module F-13's partial index earning its keep, since the unprocessed set stays tiny while the table grows to millions. On the sender side, delivery attempts are a high-churn table: rows updated on every retry generate dead versions fast, so keep attempt history append-only and partition it by day for cheap dropping rather than DELETE.
• Node.js: express.json() before your webhook route destroys the raw body — mount express.raw() on the webhook path specifically, or keep a verify callback that stashes Buffer on the request. undici/fetch follows redirects by default, so pass redirect: "error" on outbound delivery, and always set an explicit timeout since the default is effectively none.
• Redis: the natural home for per-endpoint circuit breaker state and per-endpoint delivery rate limits — small keys, high write frequency, no durability requirement, comfortably inside the 100K+ ops/s a single instance handles. Also a reasonable short-term dedup cache in front of the database check, though it must never be the only dedup layer, because an eviction becomes a duplicate charge.
• Kafka / SQS: partition by endpoint ID for delivery isolation; a FIFO queue or single-partition-per-endpoint buys ordering at the cost of head-of-line blocking. SQS's visibility timeout is the same at-least-once contract one level down — it will redeliver, so your consumer needs the same idempotency you asked your customers for.

---

Summary

• A webhook is an unreliable, unordered, at-least-once channel between two systems with different availability targets. Both ends must be designed for that; the happy path is not the design.
• Sign timestamp + "." + rawBody with HMAC-SHA256, and verify against raw bytes — a re-serialised parse changes key order, whitespace, unicode escaping, and number formatting, producing content-dependent intermittent failures that pass every test. The timestamp plus a tolerance window is what stops replay; TLS proves nothing about who sent it.
• Rotate secrets with overlapping keys, sending multiple signatures and accepting any match. Cost: a window in which a leaked secret still works — so keep immediate revocation available for real compromise, and accept it breaks slow receivers.
• Retries are an exponential, jittered schedule with a bounded total window ending in a dead-letter store. Retry 5xx/429/timeouts, never 4xx-content rejections, honour Retry-After, and keep the event ID stable across attempts — regenerating it makes deduplication impossible.
• Circuit-break per endpoint and isolate the queue per endpoint or partition (Module A-7), or one silent customer consumes your delivery workers at Little's Law's exchange rate and every other customer's lag goes vertical. Alert on per-endpoint queue age, not a global failure rate.
• Refuse to promise ordering. Retries and parallel workers reorder by construction; offer a monotonic version and a self-contained payload instead, since the alternative — one in-flight delivery per endpoint — trades reordering for head-of-line stalls.
• As receiver: verify, then queue-then-ack. Return 200 in tens of milliseconds and process asynchronously, because inline work makes your latency the sender's timeout and their retries a self-amplifying duplicate storm. The 200 acknowledges custody, not processing.
• Deduplicate on the sender's event ID with a database constraint, and gate updates on version < $new — that one predicate is the entire defence against a stale event overwriting a fresh one, a bug that produces no error at all.
• Webhooks trade polling's visible lateness for silent absence, so a reconciliation endpoint or periodic catch-up sync is mandatory, not optional. If you are the sender, an event-replay-by-time-range endpoint is the most valuable thing in your webhook API after the signature.
• A user-supplied callback URL is an SSRF primitive (Module F-16): allowlist scheme and port, resolve-and-pin the IP against rebinding, refuse redirects, and run delivery from an isolated egress network with no route to internal services or metadata.

The next module, Rate Limiting and Quotas, comes at the same relationship from the other direction: this module's 429 and Retry-After were something your delivery system had to obey, and P-18 is where you build the limiter that emits them — how to choose an algorithm, where to keep the counter, and how to shed load without punishing the well-behaved caller.

---