## Problem Statement Design an **inference service that converts PDF files into Markdown**. Assume the following building blocks already exist and you do not need to implement them — your job is to compose them into a production service: - **`split_pdf(pdf) -> List[np.ndarray]`** — a **CPU-intensive** function that splits a PDF into individual pages and rasterizes each page into a NumPy array (image). - **`ocr(page_array) -> str` (or batched `ocr(batch) -> List[str]`)** — a **GPU-intensive** OCR engine that extracts text/layout from a page image. - **`to_markdown(ocr_output) -> str`** — a **memory-intensive** post-processing step that converts a page's OCR output into Markdown and/or assembles per-page results into a final document. The work is therefore a three-stage pipeline with **three different resource profiles** (CPU → GPU → memory). The central design tension is that these stages contend for different resources and must scale independently. You must design for **two scenarios** (Part 1 and Part 2 below). ### Constraints & Assumptions State these explicitly (or ask — see below) and design against them: - **Scale:** pages range from 1 to a few thousand; documents up to ~1,000 pages on the sync path. Many concurrent jobs on the async path (assume bursty traffic). - **Resource asymmetry:** CPU split, GPU OCR, and memory-heavy markdown each saturate a different resource; they must scale independently. - **GPU is the cost driver:** OCR throughput is gated by GPU utilization and batch size; idle GPUs are wasted money. - **Correctness:** the final Markdown must be assembled in **original page order** regardless of completion order. - **Artifacts are large:** a rasterized 1,000-page PDF is far too large to hold entirely in a single process's memory. - **Sync SLA:** a synchronous request must complete (or stream) within a bounded connection lifetime; assume a hard ceiling (e.g. minutes, not hours). ### Clarifying Questions to Ask - What is the **size distribution** of PDFs (median vs. p99 page count), and the expected **request rate / concurrency**? - Is the synchronous response required to be **one final ordered document**, or may the API **stream pages** as they complete? - What are the **latency SLAs** — interactive (sync) vs. "result within N minutes" (async)? - Is **multi-tenancy / fairness** a requirement (do we need per-tenant quotas and isolation)? - What are the **durability and retention** requirements for source PDFs and outputs (and any compliance constraints on storing customer documents)? - Do we need **partial results / progress**, exactly-once delivery, or is at-least-once with idempotency acceptable? ### Part 1 — Synchronous API for one very large document A single user submits **one very large PDF (e.g. 1,000 pages)** and wants the **full converted output as fast as possible**, over a single request/response interaction. Optimize **end-to-end latency for this one document**. ```hint Where to start Don't treat the document as one monolithic job that runs split → OCR → markdown sequentially over all 1,000 pages. The three stages have different bottlenecks (CPU, GPU, memory). What is the smallest unit you could process independently, and what would that buy you? ``` ```hint Pipelining Think about whether a later stage really has to wait for the previous stage to finish for *every* page before it can start. Once you can process work at a finer granularity, what scheduling structure lets the three resources be busy at the same time, and what would you need to do to keep the GPU from sitting idle? ``` ```hint Delivering the result A 1,000-page conversion can exceed normal HTTP timeouts, and pages won't necessarily finish in order. Think about how a long-running request can give the client useful output before the whole document is done, versus returning one ordered document. Whichever you pick, how do you reconstruct the correct final page order? ``` #### What This Part Should Cover - A **sync API contract** and how the result is delivered (one ordered body vs. an incrementally streamed response), with a stated reason for the choice. - A **page-level decomposition** that overlaps CPU split, GPU OCR, and memory markdown into a pipeline rather than running them serially over all pages, plus a sense of *why* this is the dominant latency win. - An **intra-job GPU batching** scheme (micro-batching pages of the one document) and the throughput-vs-latency trade-off it implies, plus how the GPU is kept from idling. - A **result-ordering** mechanism that reassembles out-of-order page completions into correctly ordered Markdown. - Awareness that a single large document can exceed the connection ceiling, and what the contract does about it. ### Part 2 — Asynchronous API for many concurrent requests **Many clients** submit conversion requests concurrently, and each client is willing to **receive the result later** (poll, webhook, or download link). Here you optimize **global throughput and resource utilization**, not the latency of any single document. ```hint Decoupling With many jobs in flight, think about what should sit *between* the CPU, GPU, and memory stages so a burst of uploads doesn't overwhelm the GPU pool, and so each stage can autoscale on its own metric. What component gives you that? ``` ```hint GPU efficiency The GPU is the scarce, expensive resource, and it's most efficient when it processes full OCR batches — but a single small job may not have enough pages to fill one. Given that, where could the pages in a batch come from, and what scheduling policy stops one 1,000-page document from monopolizing the GPU and starving small jobs? ``` #### Clarifying Questions for this Part - What delivery mechanisms must we support — polling, webhook callback, signed download URL, or all three? - Is there a **per-tenant SLA or priority tier** (e.g. paid jobs jump ahead), or is best-effort fair sharing acceptable? #### What This Part Should Cover - An **async API contract**: submit → `job_id`, status/progress, and result retrieval (poll / webhook / signed download), with idempotent submit. - **Decoupling via durable queues** between the three stages so bursts are absorbed and each stage autoscales on its own signal. - **Cross-job GPU batching** — filling OCR batches from a shared queue across many jobs so small jobs ride along and the GPU stays saturated. - **Fairness / scheduling** so one giant document cannot starve small jobs (fair queuing, per-tenant caps, optional priority class). - **Independent autoscaling** of CPU / GPU / markdown pools, each keyed to its own bottleneck metric. ### What a Strong Answer Covers These dimensions span **both** parts — a strong candidate addresses them once for the shared engine and then specializes per part: - **Intermediate storage:** passing pointers to object storage rather than large payloads through queues; spilling per-page output instead of holding the whole document in RAM. - **Fault tolerance:** idempotent per-page tasks, per-page retries, checkpointing finished pages, dead-letter handling so one bad page doesn't block the job forever. - **Backpressure & admission control:** bounded queues, file-size/page limits, rejecting or shedding under saturation rather than silently buffering. - **Observability:** per-stage queue depth and latency, GPU utilization and average batch size, memory headroom, failure/retry rates. - A reasoned **sync-vs-async trade-off**: how the two modes share one engine and differ only in admission and delivery, and when a sync job should be redirected to the async path. ### Follow-up Questions - The sync path receives a 1,000-page PDF but the configured connection ceiling will be exceeded. What does your service do — reject, degrade, or transparently fall back to async — and what does the client see? - How do you guarantee **fairness** so that one tenant submitting many huge documents cannot monopolize the GPU pool and starve everyone else? - OCR quality is sometimes poor on certain pages (skew, low DPI). How would you add a **re-process / retry-with-different-settings** path (e.g. re-rasterize at higher DPI, deskew, swap OCR model) without reprocessing the whole document? - Suppose GPU OCR becomes the dominant cost. What levers (batch size, dynamic batching, quantization, autoscaling on queue depth, spot/preemptible GPUs) would you pull, and what are their risks?