Fixing a flaky async Python call path with timeouts and exponential-backoff retries
Company: Sierra
Role: Product Manager
Category: Software Engineering Fundamentals
Difficulty: medium
Interview Round: Other
A Python service awaits a call to an unreliable dependency, and the call sometimes fails with an error and sometimes stalls. Debug the existing call path, then extend it with an async retry mechanism: give each attempt a timeout, wait an exponentially growing delay after each failure, and keep retrying until a call succeeds.
The original code is not reproduced here: assume you can run it and see how it fails, and walk through how you would find what is broken before you build on it.
```hint Fresh attempts, real waits
Ask what happens if the same coroutine object is awaited a second time, whether a timeout should bound one attempt or the whole loop, and what the event loop is doing while your code waits between attempts.
```
### Clarifying Questions
- What does the failure look like today: does the caller crash on the first error, hang forever, or slow down other requests while it waits?
- Which failures should be retried: every exception, or only timeouts and transient errors such as dropped connections, with bad-input errors raised at once?
- Does "until success" mean no limit at all, or is there an overall deadline the caller would accept, and should the delay between attempts be capped?
- Is the call safe to repeat if a timed-out attempt actually reached the dependency?
### What a Strong Answer Covers
- A method for finding the existing fault: reproducing it, reading the traceback, and checking for common asyncio mistakes
- A correct retry helper: a fresh call per attempt, a timeout per attempt, and non-blocking exponential backoff with a cap
- Clear choices on which errors to retry, how cancellation behaves, and how to avoid retrying forever when that is not wanted
- Tests that run fast and deterministically by faking the sleep
- Operational concerns: idempotency, jitter so clients do not retry in lockstep, and logging and metrics
### Follow-up Questions
- Many service instances call the same dependency and it goes down. What happens when it comes back, and how do you keep retries from knocking it over again?
- How would you test the backoff schedule without the test suite actually waiting?
- A customer is waiting on this call in a live conversation. How does that change your timeout and retry settings?
Overview: A Python coding exercise from a Sierra product manager interview: debug an async code path that calls an unreliable dependency, then add retries with a timeout on each attempt and exponential backoff until the call succeeds. It tests asyncio fundamentals, error handling, cancellation and safe retry design.