Webhook Dispatcher: Retries, Ordering and Pausing

Webhook Dispatcher: Retries, Ordering and Pausing

You've joined the platform team that owns the webhook dispatcher: the service that delivers event notifications to customers' HTTP endpoints, retrying failures with backoff and giving up ("dead-lettering") when an endpoint stays broken. Support has two retry-policy complaints, and the roadmap has two features behind them. Time is simulated: every operation carries an integer ts in seconds, and ts never decreases.

Codebase

How your code is run

run(operations: list[list]) -> list executes the operations in order against a fresh dispatcher and returns one result per operation. Every operation is [name, ts, ...args]. Errors are returned as "error: <code>" strings, never raised.

• An unknown operation name returns "error: unknown operation". A wrong number of arguments or a non-integer ts returns "error: invalid arguments".
• Event ids are unique across all endpoints. An event is delivered to the one endpoint it was enqueued for. outcomes scripts the simulated HTTP status of each attempt: attempt *n* gets outcomes[n-1], and the last entry repeats once the list runs out. It must be a non-empty list of integers, each 0 (a timeout) or in 100..599; otherwise invalid outcomes.
• enqueue makes the event due at its own ts. Nothing is attempted until process runs.
• process at time ts attempts every event whose due_at <= ts (earliest due_at first, ties by enqueue order). An attempt increments the event's attempt count and then:
• status 200..299: the event is delivered; record "evt-1 #3 200 delivered".
• status 500..599, 408, 429 or 0 is retryable. If attempts remain, the next attempt is due at ts + delay and the record is "evt-1 #1 503 retry at 130".
• any other status (e.g. 404, 410) is permanent: the event is dead immediately; record "evt-1 #1 404 dead".
• a retryable failure on the last allowed attempt also ends in "... dead". A retry is always due in the future, so an event is attempted at most once per process call.
• Retry policy (see config.py): an event gets at most 6 attempts in total. The wait after the first failed attempt is 30 seconds, and it doubles after every further failure, never exceeding 300 seconds: waits of 30, 60, 120, 240, 300.
• status returns the event's state ("pending", "delivered" or "dead"), the number of attempts made so far, and next_attempt_at: its due_at while pending, otherwise null.

Worked example

[["register", 0, "shop", "https://shop.example/hooks"],
 ["enqueue", 0, "shop", "evt-1", [503, 503, 200]],
 ["process", 0],
 ["process", 30],
 ["process", 89],
 ["process", 90],
 ["status", 90, "evt-1"]]

returns

[true, "queued",
 ["evt-1 #1 503 retry at 30"],
 ["evt-1 #2 503 retry at 90"],
 [],
 ["evt-1 #3 200 delivered"],
 {"state": "delivered", "attempts": 3, "next_attempt_at": null}]

Part 1 — Retries are off

Goal: make retries follow the policy above.

Customers and support have reported:

1. "Our server returns a 500 once and the retry comes a full minute later, not 30 seconds."
2. "An endpoint that was down for hours received seven attempts per event. We promised six."

Rules

1. The wait after the first failed attempt is 30 seconds, then 60, 120, 240 and 300 (the cap).
2. An event is dead-lettered after its 6th attempt fails; there is no 7th attempt.
3. Everything else in the table above keeps working as described.

Part 2 — Per-endpoint ordering

Goal: a customer's endpoint must see its events in the order they were enqueued, even when an earlier event is being retried.

Rules

1. For one endpoint, an event is not attempted while any earlier-enqueued event for the same endpoint is still pending (not yet delivered or dead). It waits behind that event, even if its own due_at has passed.
2. As soon as the earlier event is delivered or dead-lettered, the next one is eligible. If it is due at the current ts it is attempted in the same process call.
3. Endpoints do not block each other.
4. In one process call, repeatedly pick, among each endpoint's first pending event whose due_at <= ts, the one with the smallest (due_at, enqueue order) and attempt it, until none is left.
5. A waiting event reports state: "pending", attempts: 0 and its own due_at as next_attempt_at (blocking is not reflected in status).

Example: evt-1 (outcomes [500, 200]) and evt-2 ([200]) are enqueued for shop at ts 0. process at 0 returns ["evt-1 #1 500 retry at 30"] (evt-2 waits). process at 30 returns ["evt-1 #2 200 delivered", "evt-2 #1 200 delivered"].

Part 3 — Pause and resume an endpoint

Goal: customers need to pause an endpoint during maintenance, and later resume it, optionally re-sending events that had already given up.

Rules

1. disable(endpoint_id) pauses the endpoint: process makes no attempts for it. Errors: unknown endpoint, already disabled.
2. While disabled, enqueue still accepts events for it. Its pending events (new ones and ones waiting for a retry) keep their attempt counts, and status reports them as state: "held" with next_attempt_at: null. Delivered and dead events are unaffected.
3. enable(endpoint_id, replay_dead) resumes the endpoint; replay_dead must be true or false (anything else is invalid arguments, checked before the other errors). Errors: unknown endpoint, not disabled.
4. On enable, every pending event of the endpoint becomes due at the enable ts (it keeps its attempt count, so the retry schedule continues where it stopped).
5. With replay_dead: true, the endpoint's dead-lettered events are also revived: state back to pending, attempts reset to 0, due at the enable ts, and removed from dead_letters. With false they stay dead.
6. Ordering from Part 2 holds across all of this: a revived event goes back to its original position (by enqueue order) among the endpoint's pending events and is attempted before later events.
7. Other endpoints are completely unaffected.

Example: evt-1 ([404]) is dead-lettered for shop, then shop is disabled, evt-2 is enqueued, and at ts 50 ["enable", 50, "shop", true] is called. evt-1 and evt-2 are both due at 50; process at 50 attempts evt-1 (again, as attempt #1) and, since it dies again with its 404, evt-2 right after it in the same call.

Notes

• config.py, transport.py, registry.py and models.py are known-good. The Part 1 bugs are in the retry logic.
• You may use the AI assistant freely. Later parts must not break earlier ones: hidden tests re-check earlier behaviour, including exact attempt records and retry at times.
• Parts 1 and 2 are the baseline; finishing all three is a strong result.