Webhook Incident Recovery Flow

A webhook replay runbook is a controlled procedure for resending an event that failed delivery or failed processing. It should exist before the incident, because the most dangerous replay decisions are usually made when a customer, payment, shipment, or integration is already stuck.

The first checkpoint is payload capture. The runbook should say where the raw body, headers, delivery id, event id, source timestamp, response status, and failure reason are stored. Pretty-printed JSON is not enough if the receiver verifies signatures over the exact body. If the raw body is not available, the runbook should switch to reconstruction mode instead of pretending the replay is exact.

The second checkpoint is idempotency. Before replaying, confirm the consumer has a replay key, event id dedupe table, natural unique constraint, or side-effect guard. If the receiver can charge twice, ship twice, email twice, or decrement inventory twice, production replay should stop until a compensating guard is in place. A retry without idempotency is not recovery; it is a new risk.

The third checkpoint is target selection. Diagnostic replay should go to a sandbox, local endpoint, or dry-run route when possible. Production replay should name the exact endpoint, tenant, object id, and side effects expected. If the endpoint has changed since the event was created, the team should decide whether they are testing old behavior, current behavior, or a migration path.

The fourth checkpoint is authorization and signature handling. Some webhooks include timestamp windows or signatures that cannot be reused. In that case, the sender platform may need to provide an official replay mechanism, or the team may need to reconstruct a new signed request. Manually copying an old signature can produce misleading failures.

The fifth checkpoint is verification. After replay, query the receiving system for the expected state and check logs for duplicate side effects. Record the replay id, actor, time, payload source, and result. The runbook is complete only when the stuck object has a verified state, not when the HTTP request returns 200.

Webhook Replay Runbook Checklist

// COMMENTS

ON THIS PAGE