Payment Webhook Processing Service — On-Call Runbook

0. 3 AM Quick Path

The three most likely causes

1. Queue backlog: Queue depth and oldest-message age are rising while worker throughput is below arrival rate. Check worker health, errors, and downstream saturation; scale only if dependencies can absorb it.
2. Provider timeout: Requests to or from one provider are timing out. Check provider status and timeout metrics; keep events durable and reduce retry pressure.
3. Signature failure: One provider suddenly returns 400, 401, or 403, often after a secret rotation or deployment. Compare the configured secret version and raw-body handling with the provider’s documented requirements; never bypass validation.

Five-minute decision path

1. Acknowledge the alert and open the incident channel.
2. Record the current time and earliest known failure time.
3. Check these metrics:
   • Webhook requests by provider and status
   • Signature failures by provider and reason
   • Queue depth and oldest-message age
   • Worker processing and failure rates
   • Provider timeout rate
   • Dead-letter queue depth
4. Follow the matching section:
   • Queue age rising → Queue BacklogQueue Backlog
   • Provider timeout rate rising → Provider TimeoutsProvider Timeouts
   • Signature failures rising → Signature FailuresSignature Failures
5. Declare SEV-1 and escalate immediately if events may be lost, payment state is uncertain, duplicate financial actions are possible, or signature verification was bypassed.

Do not purge queues, disable signature validation, manually mark payments successful, or bulk-replay events during initial triage.

1. Purpose and Scope

Use this runbook when payment provider webhooks are delayed, failing, rejected, or not reflected in downstream systems.

Primary failure modes:

• Queue backlog
• Payment provider timeouts
• Webhook signature failures

Primary object