RFC-014: Queue-backed webhook delivery with retries

Summary

Dispatch sends customer webhooks synchronously inside the request that produced the event, with a single attempt and a 5 s timeout. In August 2026, 2.3% of deliveries failed and were never retried, and one slow customer endpoint added up to 5 s to unrelated API calls. This RFC proposes a webhook_deliveries table as a durable outbox, a dispatcher worker pool that drains it with exponential retries over 24 hours, a per-endpoint circuit breaker, and an Idempotency-Key header so customers can deduplicate. The change is behind a per-workspace flag and rolls out over four weeks. Reviewers are asked to approve the design and answer the four open questions by 2026-09-19.

Context

Customers register up to ten HTTPS endpoints per workspace and subscribe to events (job.created, job.assigned, job.completed, technician.location and 14 others). When an API request or a background job produces an event, WebhookSender.send() is called inline: it serialises the payload, signs it with the workspace secret, and issues one HTTP POST with a 5 s timeout. The result is logged and discarded.

Failures cluster: 61% of failed deliveries in August went to 23 endpoints, mostly during customer maintenance windows or deploys. The remaining 39% are spread across timeouts, TLS errors, and 5xx responses from otherwise healthy endpoints.

Problem

1. Lost events. A failed delivery is gone. Customers reconcile by polling GET /v2/jobs?updated_since=, which is 18% of read traffic and defeats the purpose of webhooks. Support tagged 112 tickets "missing webhook" in the last quarter.
2. Coupled latency. The sending request waits for the customer's endpoint. During INC-2026-0716 one endpoint answered in 4.8 s and POST /v2/jobs/{id}/complete p95 rose from 120 ms to 4.9 s for every workspace sharing the worker pool.
3. No visibility. Customers cannot see delivery attempts or replay an event; Support reads raw logs on their behalf.

Goals and non-goals

Goals

• G1: No event is lost while the customer endpoint is unavailable for up to 24 hours.
• G2: Webhook delivery adds no more than 2 ms p99 to the producing request.
• G3: At-least-once delivery with a stable idempotency key per event so customers can deduplicate.
• G4: Per-event delivery history and a replay action visible in the API and the console.
• G5: One misbehaving endpoint cannot degrade delivery to other endpoints in the same or other workspaces.
• G6: Sustain 25 M deliveries per day on the current worker tier with headroom of 2x.

Non-goals

• Ordering guarantees across events. Events carry occurred_at and a monotonic sequence per workspace; consumers order.
• Exactly-once delivery. Not achievable over HTTP; idempotency keys are the contract.
• A customer-facing retry policy editor. Fixed schedule in this iteration.
• Replacing the event producers. The outbox write happens where WebhookSender.send() is called today.

Proposal

Components

 producer (API / job)                         dispatcher workers (N=8 per region)
 ┌──────────────────────┐   same txn   ┌────────────────────┐   claim batch   ┌───────────────┐
 │ domain write         │────────────▶ │ webhook_deliveries │◀────────────────│ worker        │
 │ + INSERT delivery    │              │ (outbox, Postgres) │  FOR UPDATE     │  sign + POST  │──▶ customer
 └──────────────────────┘              └────────────────────┘  SKIP LOCKED    │  record result│    endpoint
                                                 ▲                            └───────┬───────┘
                                                 │ next_attempt_at = now + backoff     │
                                                 └─────────────────────────────────────┘
                                       endpoint_state (circuit breaker, per endpoint)

1. Outbox table webhook_deliveries: id, workspace_id, endpoint_id, event_id, payload_hash, attempt, status (pending, delivered, failed, dead), next_attempt_at, last_error, created_at. Payloads live in the existing events table; the delivery row references them. Partial index on (next_attempt_at) WHERE status = 'pending'.
2. Dispatcher: a worker pool per region claims up to 200 pending rows per poll with SELECT ... FOR UPDATE SKIP LOCKED, groups by endpoint, and sends with a per-endpoint concurrency of 4 and a 10 s timeout. Results update the row; failures schedule the next attempt.
3. Circuit breaker endpoint_state: after 20 consecutive failures an endpoint opens for 5 minutes; deliveries to it stay pending without consuming attempts. A single probe closes it. Open breakers surface in the console and via GET /v2/webhooks/endpoints/{id}.
4. Idempotency: header Idempotency-Key: <event_id> and X-Larkspur-Attempt: <n> on every POST. The signing string includes the event ID so replays verify identically.
5. Replay: POST /v2/webhooks/deliveries/{id}/replay inserts a new delivery row with the same event; console button calls it.

Retry schedule

Failure handling

• Worker crash mid-batch: claimed rows are locked in an open transaction; the lock releases on crash and another worker claims them. Duplicate sends are possible and covered by the idempotency key.
• Database unavailable: producers fail their own write (unchanged behaviour); no delivery row is created without the domain change.
• Backlog growth: alert when pending rows older than 5 minutes exceed 10,000 or when the oldest pending row is over 15 minutes old. Workers scale horizontally; the claim query is bounded by the partial index.

Observability

Metrics: webhook_delivery_attempts_total{result}, webhook_delivery_latency_seconds, webhook_backlog_age_seconds, webhook_breaker_open{endpoint}. Every attempt logs event_id, endpoint_id, attempt, status, and duration. The console delivery view reads webhook_deliveries directly.

Alternatives considered

Risks and mitigations

Rollout plan

Everything is behind workspace.flags.webhook_outbox. When the flag is off, WebhookSender.send() behaves exactly as today. Rollback at any phase is flipping the flag; pending rows are drained by the dispatcher regardless, so no events are lost on rollback.

1. Phase 0 (week of 2026-09-22): schema and shadow writesMigration adds webhook_deliveries and endpoint_state. Producers write outbox rows for all workspaces; the dispatcher runs in dry-run mode and marks rows delivered without sending. Gate: outbox insert adds under 2 ms p99 to producers; dry-run backlog age stays under 5 s for 72 hours.
2. Phase 1 (2026-09-29): internal workspacesFlag on for Larkspur's own three workspaces and the integration test suite. Gate: zero lost events versus the synchronous path over one week, verified by comparing event IDs received by the test receiver.
3. Phase 2 (2026-10-06): opt-in beta, 20 customersSupport invites the 23 endpoints with the highest failure rates. Console shows delivery history. Gate: failed-and-never-retried rate below 0.05% for beta workspaces; no P1 support tickets attributable to duplicates.
4. Phase 3 (2026-10-20): general availabilityFlag on for all workspaces in 25% daily increments. Synchronous path remains for two release cycles, then is removed. Gate: backlog age p99 under 15 s at 100%; alert volume unchanged for one week.

Security and privacy

• No new data classes. Delivery rows store IDs, hashes, timestamps, and the response status; payloads remain in events under existing retention (90 days).
• last_error stores the HTTP status and the first 256 bytes of the response body with credentials redacted by the existing log scrubber.
• Signing is unchanged (HMAC-SHA256 over timestamp, event ID, and body). Replays re-sign with the current timestamp so customers' 5-minute replay window still applies.
• The replay endpoint requires the webhooks:write scope and is rate-limited to 100 per workspace per hour.

Open questions

• Q1. Should technician.location events skip retries entirely?Dana Okafor · by 2026-09-16Location updates are superseded every 30 s; retrying stale ones wastes attempts. Proposal: per-event-type policy with max_attempts = 1 for location.
• Q2. Retain dead deliveries for 30 days or align with the 90-day event retention?Marcus Bell · by 2026-09-1630 days covers every replay request Support has seen; 90 days triples table size at target volume.
• Q3. Do we email endpoint owners after attempt 5, or only when a breaker opens?Ines Fontaine · by 2026-09-19Attempt-5 emails may be noisy for endpoints with brief maintenance windows.
• Q4. Regional workers or a global pool?Priya Natarajan · by 2026-09-19Regional keeps data residency simple and matches the outbox's database; global evens out load. Proposal assumes regional.

Decision log

2026-09-04

Retry ceiling is 24 hours, not 72. Decided by Ines Fontaine with Support. Beyond 24 hours customers reconcile from the API anyway; a longer window mostly grows the table.

2026-09-08

Idempotency key is the event ID, not a per-delivery UUID. Decided in review with Tomasz Wierzbicki and Priya Natarajan. Customers deduplicate on the thing they care about (the event), and replays reuse it by design.

2026-09-10

Alternative D deferred rather than rejected. Revisit criterion recorded in the alternatives section.