Design Robust Large-Input Processing
Company: Attentive
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Technical Screen
The string-reduction algorithm above is now part of a backend service.
Each job reads a text object from S3-compatible storage, applies the transformation, and writes the result back. A worker has 1 GB of memory and a 10-minute timeout, but some input objects are several GB in size. In production, some jobs crash.
Describe how you would:
1. Triage and diagnose the crash.
2. Determine whether the failure is caused by malformed input, an application bug, memory pressure, timeout, or a dependency issue.
3. Redesign the service so it can handle very large inputs without first loading the entire object into memory.
4. Preserve correctness when character runs span chunk boundaries.
5. Add observability, testing, and recovery mechanisms so similar failures are easier to detect and fix in the future.
Quick Answer: This question evaluates debugging and root-cause analysis, memory management, streaming/chunked data processing (including correctness across chunk boundaries), and system reliability aspects such as observability, testing, and recovery for large-input backends.