Retry and replay

Webhook delivery retries happen outside trade and payout DB transactions.

Default schedule

• attempt 1: immediate after event commit
• attempt 2: +1 minute
• attempt 3: +5 minutes
• attempt 4: +15 minutes
• attempt 5: +1 hour
• attempt 6: +6 hours
• after max attempts: dead_letter

Each delay is measured from the preceding failed attempt's scheduling time, not from original event occurrence. Dispatch runs asynchronously, so these are due times, not delivery-time guarantees. With an endpoint policy above six attempts, further retry delays remain six hours; the endpoint maximum is ten total attempts.

What causes retry

• request timeout
• DNS failure
• TLS failure
• network error
• HTTP 429
• HTTP 5xx
• other non-2xx except terminal 410

What ends delivery

• any 2xx response -> sent
• HTTP 410 -> dead_letter immediately
• max attempts reached -> dead_letter

Historical skips and missing history require explicit resend

The monitor's event-only no_endpoint, endpoint_disabled, and not_subscribed statuses record the dispatcher's actual observation that no callback could be scheduled: respectively zero configured endpoints, no active endpoints, or no matching active subscription. They have no retained HTTP attempts or automatic retry. Creating/enabling an endpoint later does not retroactively send them; explicitly use Resend to backfill to currently active, subscribed endpoints.

history_unavailable is a terminal dispatched event without retained delivery history, including historical baseline records and gaps after endpoint deletion. It does not establish whether a callback was previously sent or received, nor infer historical endpoint configuration. No missing attempts, HTTP responses, or numeric delivery IDs are fabricated. disposition_recorded_at is the observation/baseline recording time, separate from the original event occurrence time. Explicit Resend creates new real deliveries and may duplicate an earlier callback; receivers must deduplicate by the original event_id.

Event-only pending means the outbox is pending/dispatching, not definitively skipped; event-only failed/dead_letter reflect the actual outbox state. Event-only Dead letter records can be selected for single or bulk Resend: it creates real deliveries to current active subscribers and sends the normal signed callback with the original event identity and occurrence time. Opinion order data is reconstructed from current durable records; Polyflight data retains its stored event snapshot. Event-only pending/failed work cannot be manually backfilled while automatic processing owns it.

Timeout behavior

• default endpoint timeout: 5 seconds
• operator can set timeout per endpoint record
• the last receiver response is trimmed and limited to 500 characters for debugging, not retained as a full response archive or comprehensively secret-redacted; receivers must not echo credentials or sensitive diagnostics

Replay semantics

• replay targets an existing retained delivery record whose endpoint still exists and is active
• replay creates another delivery attempt for same outbox event
• replay does not create new business event
• replay does not mutate order, ticket, result, wallet operation, payout, refund, or redemption state
• replay stays operator-scoped
• replay of any previously delivered 2xx response may produce another accepted callback, not only replay after HTTP 200; dedupe by the original event_id
• an event already marked dispatching cannot be claimed by another dispatcher; replay returns 409 WEBHOOK_DELIVERY_NOT_REPLAYABLE if no eligible attempt can start, rather than reporting a queued callback
• the dashboard's single/bulk Resend queues 1–100 records atomically through POST /api/operator/webhooks/deliveries/resend; retained records use the original active endpoint, while terminal historical event-only records (including dead letter) use current active subscribers
• queuing is not receipt confirmation; the dispatcher sends pending or due failed deliveries asynchronously and does not repeat successful/unselected terminal sibling deliveries
• automatic retries use already materialized deliveries and do not fan out to newly configured endpoints; delivery order is not guaranteed

Retained replay/resend targets the same endpoint record, using its current URL, secret and delivery policy, irrespective of today's subscriptions. It is not a historical-URL replay. Review Setup History after URL or secret changes. Every send uses a fresh timestamp/signature and normal callback reconstruction, not archived signed bytes.

The Callback Settings recent-deliveries tab and POST /deliveries/{id}/replay use retained-delivery dispatch during the request. The full monitor's Resend uses the asynchronous queue route and also supports historical backfill. The monitor's replayable flag describes retained replay only; false on a terminal event-only row does not mean the historical Resend action is unavailable.

Manual resend preserves lifetime attempt counts. It permits one explicitly requested attempt even if earlier retries exhausted the endpoint limit; failure at/above that limit returns to dead letter without a new automatic retry budget. Already-pending deliveries cannot be queued again.

If dispatch throws before finalization, it releases only its matching in-flight claim into failed outbox processing with a one-minute retry delay when the database is available. This does not fabricate a delivery attempt or HTTP result, and cannot reset a committed terminal skip or a newer claim.

Receiver design rules

• return 2xx only after durable commit on operator side
• return 5xx for transient failure that should retry
• use 410 only when event should never be retried for that endpoint
• do not rely on exactly-once delivery
• retain the event_id dedupe record for as long as dashboard replay is possible; permanent inbox retention is safest
• acknowledge an already-committed duplicate with 2xx; 409 is retried like any other non-2xx
• never use webhook replay to retry a Polyflight wallet effect; reconcile the REST v1 operation by its original financial key

Dashboard workflow

1. Inspect the row kind, factual status/reason, event time and delivery time. For real deliveries, inspect attempts and last response sample; event-only rows do not represent HTTP attempts.
2. Fix receiver issue or secret mismatch; do not infer a wallet debit/credit from callback delivery status.
3. Use Resend on an eligible delivery or historical event-only record. For several records, tick the leftmost checkboxes and confirm Bulk Resend. Configure an active, subscribed endpoint before historical backfill. If any item is ineligible, the batch queues nothing. Confirm the receiver durably deduplicates event_id, including when prior history is unavailable.
4. Refresh after asynchronous dispatch to inspect whether attempts become sent; reconcile monetary effects separately using the original financial idempotency key. After a queue-request timeout, inspect retained state before resubmitting.

Endpoint deletion

Deleting an endpoint removes its delivery history, so those deliveries can no longer be inspected or replayed. Its configuration setup history is retained separately, including the deletion record and before/after settings. Inspect Callback Settings → Setup History or use GET /api/operator/webhooks/endpoints/history with webhooks:read; see endpoint setup history for actor provenance, secret safety, and baseline limitations.

For audit review, the delivery API and dashboard can filter by event or delivery time, row kind, status, HTTP class and business identifiers. Their CSV export uses the same applied filters and includes reconstructed callback data, not raw signed request bytes or response samples. It exports at most 5000 matching rows and rejects larger selections with 422 EXPORT_TOO_LARGE; narrow the filters rather than treating a partial file as complete. Exports can contain customer and financial data and require sensitive-data handling.