## Make a Courier Pay Workflow Resilient to Downstream Failures A coding exercise has produced a simplified courier-pay calculator. The calculation rules themselves are already tested and are out of scope here. For each completed delivery, the calculator emits a `PayInstruction` containing `delivery_id`, `courier_id`, `amount_minor_units`, `currency`, and `rules_version`. A downstream ledger must record each instruction exactly once from the business perspective, even though requests, acknowledgments, and workers can fail. Design the production workflow around that calculator. Explain what happens when the downstream ledger is slow, unavailable, accepts a request but loses the response, or rejects invalid data. ### Constraints & Assumptions - A delivery must never be paid twice because a caller retried after a timeout. - A transient dependency failure must not silently lose an accepted pay instruction. - Invalid instructions require a terminal, inspectable state rather than endless retry. - Operators must be able to reconcile completed deliveries with ledger records. - Do not assume a distributed transaction across the calculator's database and the downstream ledger. ### Clarifying Questions to Ask - Which event establishes that a delivery is final and payable? - Can the ledger accept a caller-supplied idempotency key and return the prior result? - Which rejection classes are retryable, and which require correction or manual review? - Can an instruction be canceled or superseded after it has reached the ledger? ### What a Strong Answer Covers - A durable boundary between committing the pay instruction and attempting delivery. - Stable idempotency keyed to the business action, not to a worker attempt. - Bounded retries with backoff, jitter, timeout classification, and a terminal failure path. - Protection against the “accepted but response lost” ambiguity. - Reconciliation, observability, replay safety, and tests for partial failure. ```hint Separate creation from delivery First make the intent durable in the calculator's own transaction; then reason about repeated delivery attempts. ``` ### Follow-up Questions 1. Why is an in-memory retry loop insufficient after a process crash? 2. What should a worker do after a timeout if the ledger may already have committed the instruction? 3. How would a corrected amount be represented without reusing the original business action incorrectly? 4. Which metrics distinguish a ledger outage from permanently invalid instructions?