The work candidates describe is not product CRUD. It is the machinery that runs model evaluations and feeds training: pipelines that move large datasets asynchronously, harnesses that fan out thousands of evaluation jobs, reward signals for reinforcement learning experiments, and platform APIs that researchers call. The fourteen reported questions follow that shape. Four are technical and domain-specific (agent evaluations at scale, a training-data pipeline, model failure modes, reward signals), four are system design (task orchestration, fault tolerance with message queues, AWS versus GCP, internal platform APIs), three are coding and fundamentals (concurrent background tasks, Linux and DevOps, memory and throughput on synthetic datasets), and three are behavioral.
One mental model carries most of the technical questions: the life of a job. A job is enqueued, leased by a worker, run, acknowledged or failed, retried with backoff, parked in a dead-letter queue if it keeps failing, and its result is stored where someone can query it. Agent evaluations, dataset-generation tasks and reward verification all follow that cycle, and each question becomes a matter of choosing delivery semantics, idempotency keys, timeouts, concurrency limits and the metrics that show where jobs are stuck. Rehearse that vocabulary until you can draw it without notes.
The ML side is lighter than the infrastructure side, but you need working vocabulary. Know the difference between a verifiable reward (unit tests pass, an answer matches a checker) and a human-preference signal (ratings, rankings), why noisy labels and reward hacking matter, and how a failure-mode taxonomy turns raw model outputs into something an engineer can act on. The requirements candidates describe treat RLHF and RLVR familiarity as a bonus and say pure research profiles without strong engineering output are not a fit, so lead with systems you built and operated.
Logistics are part of the loop too. Candidates report that the role is fully onsite in San Francisco every day, and the final stage is a hands-on work trial whose format is not detailed. Settle location questions at the recruiter screen and ask what the work trial involves before you commit your preparation time.
Recruiter Screen
reportedCandidates describe the Recruiter Screen as an opening conversation about your background, your motivation and logistics. Logistics matter here because the role is reported as fully onsite in San Francisco. Treat it as the place to connect your experience to the stack the role lists (Python or JavaScript, message queues, AWS or GCP, async pipelines) and to learn the shape of the remaining rounds.
What to demonstrate
- Fit between your background and the backend and distributed-systems experience the role lists
- Motivation for evaluation and training infrastructure work specifically
- Logistics, including the reported daily onsite commitment
How to prepare
- Prepare a short career walkthrough that highlights two systems you owned in production, with one number for each (throughput, latency, cost or error rate).
- Write one honest sentence on why evaluation, reward or training-data infrastructure interests you, tied to something you have built.
- Confirm location and start-date constraints, and ask for the format of the later rounds and the work trial in writing.
Technical Deep-Dive
reportedThe Technical Deep-Dive is described as in-depth technical discussion with engineering team members. The reported format is conversational, so the work is to take real projects to full depth: architecture, failure history, numbers and the alternatives you rejected. The technical and domain-specific question set (agent evaluations, training-data pipelines, failure modes, reward signals) is good rehearsal material for this kind of discussion.
What to demonstrate
- Depth of backend and distributed-systems knowledge in systems you have actually built
- Ability to explain trade-offs in latency, throughput and fault tolerance
- Working understanding of evaluation and reward concepts at engineer level
How to prepare
- Pick two or three projects and write down, for each, the architecture, one outage or bottleneck, the fix and the metric that moved.
- Answer the question on running agent evaluations at scale aloud, covering job specs, sandboxing, rerun policy and result storage.
- Write a half-page glossary for RLHF, RLVR, reward hacking and failure-mode taxonomies in your own words.
Structured Technical Conversations
reportedStructured Technical Conversations are described as structured discussions focused on technical skills. The reports do not detail the structure, so prepare a repeatable answer shape: clarify scope, state assumptions, give a first design, then walk the failure modes. The coding-and-fundamentals questions (concurrency with rate limiting, Linux and DevOps, memory and throughput) fit a discussion format as well as a coding one.
What to demonstrate
- Core engineering fundamentals discussed in a structured format
- Clear reasoning about assumptions, edge cases and failure handling
- Command of Python or JavaScript concurrency and of Linux production basics
How to prepare
- Practise answering the Linux and DevOps question with a concrete checklist: process and resource inspection, signals and graceful shutdown, log and metric sources, deploy and rollback.
- Explain out loud how you would keep memory flat while processing a very large synthetic dataset: streaming, bounded batches, backpressure.
- Rehearse the same five-step shape on three different questions until it is automatic.
System Design Evaluations
reportedSystem Design Evaluations are described as assessments of both internal and external system design capability. The question types candidates report in this area are task orchestration for thousands of concurrent evaluation jobs, fault tolerance and observability around message queues, AWS versus GCP component choices for low-latency async work, and internal platform APIs. Expect open-ended prompts where you set the requirements yourself.
What to demonstrate
- Ability to design an asynchronous, queue-based system end to end
- Trade-off reasoning on storage, compute and orchestration
- Handling of failure modes, observability and backpressure
How to prepare
- Draw the orchestration design on paper with scheduler, queue, workers, lease or heartbeat, retry policy, dead-letter queue and result store, then break one component on purpose and explain recovery.
- Prepare a one-page comparison of the AWS and GCP queue, compute and storage options you have used, with latency and operational cost trade-offs.
- List what you would instrument on an event-driven pipeline: queue depth, consumer lag, age of the oldest message, retry rate, per-stage latency.
Hands-On Work Trial
reportedThe Hands-On Work Trial is described as a practical assessment of skills through a work trial. Reports do not give its length, location or task, so ask the recruiter. The most useful preparation is to be able to produce working, tested code quickly in a language you choose: a background worker with concurrency limits, rate limiting, retries and timeouts is the kind of task the coding questions point to.
What to demonstrate
- Ability to produce working code in a practical setting
- Production habits: tests, error handling and clear structure
- Scoping and communication while working on an open task
How to prepare
- Build a small queue-polling worker in Python or JavaScript with a concurrency cap, a token-bucket rate limiter, retry with jitter, per-job timeout and graceful shutdown, with tests.
- Write a short README for it covering assumptions, failure behavior and what you would do next.
- Practise stating your scope and assumptions out loud before you start typing.
PracHub editorial advice for the preparation topics above.
Naming Kafka, RabbitMQ or Pub/Sub without saying what delivery guarantee the design relies on
State at-least-once delivery out loud, then make consumers idempotent with a job id or dedupe key, and say where that key is stored. Explain what happens to a message whose consumer crashes after doing the work but before acknowledging.
Designing the evaluation orchestrator as a single scheduler with no answer for a worker or scheduler dying mid-job
Add leases or heartbeats so abandoned jobs return to the queue, a retry cap with backoff, a dead-letter queue for poison jobs, and a way to run the scheduler redundantly. Walk one failure through the whole design before the follow-up asks.
Writing a background worker that retries forever, retries immediately, or has no per-job timeout
Cap attempts, use exponential backoff with jitter, set a timeout on every external call and job, retry only idempotent work, and stop pulling new jobs on shutdown while in-flight work finishes. Add a test that forces a timeout and a failure.
Telling the ownership story as a team effort with no personal decision and no measured result
Write each story with your decision, the metric before and after, and what you left running afterwards (alerts, runbook, load test). Replace 'we improved performance' with the specific number you moved.
Treating reward signals and failure modes as vague ML topics instead of engineering problems
Separate verifiable rewards from human-feedback signals, name reward hacking and label noise as risks, and describe the pipeline stage that detects them, such as sampling outputs, categorizing failures and re-checking a reward on known cases.
Choose a category, try a prompt, then open its approach, worked solution or follow-up when you need it.
Walk through your approach to choosing between AWS and GCP components for low-latency asynchronous workloads.
Walk through your approach to choosing between AWS and GCP components for low-latency asynchronous workloads.
Approach
- Start from the workload, not the vendor: p99 latency budget, throughput and burst shape, ordering and delivery needs, task duration, where the data and models already live, team experience, credits, and cross-cloud egress cost. Colocating compute, queue and data in one region usually matters more than which provider you pick.
- Map the equivalents and their real differences. SQS standard is at-least-once with best-effort ordering; SQS FIFO orders within a message group and deduplicates within a 5-minute window. Pub/Sub is at-least-once by default, with ordering keys and an optional exactly-once mode for pull subscriptions. Kafka needs are covered by MSK on AWS and Managed Service for Apache Kafka on GCP.
- Compute: Lambda caps an invocation at 15 minutes. Cloud Run handles longer requests and concurrent requests per instance. For long evaluation or pipeline tasks, containers on ECS/EKS or GKE avoid those limits. Cold starts are handled with provisioned concurrency on Lambda and minimum instances on Cloud Run.
- Orchestration: Step Functions on AWS; Workflows and Cloud Tasks on GCP, where Cloud Tasks also gives per-queue rate and concurrency limits. Prefer the managed primitive unless a hard requirement rules it out.
- Decide with measurements. Benchmark enqueue-to-dequeue latency at p50 and p99 under your burst pattern in the target region, price the egress, and keep a thin queue interface in your own code so switching later costs less.
Follow-up
- When would you run your own Kafka instead of a managed queue?
- Your data sits in one cloud and the GPUs you can get are in the other. How do you design around that?
- How do you handle a burst ten times normal load without losing messages or breaching latency?
Write a robust implementation for handling concurrent background tasks with rate-limiting and error recovery.
Write a robust implementation for handling concurrent background tasks with rate-limiting and error recovery.
Approach
- Structure in Python asyncio: an asyncio.Queue with a maxsize (for backpressure) feeds N worker coroutines, so N caps in-flight work. Keep the rate limit separate from concurrency: concurrency limits how many calls are in flight, the rate limit how many start per second.
- Token bucket: store tokens, capacity and the last refill time from time.monotonic(). On acquire, refill with tokens = min(capacity, tokens + elapsed * rate). If fewer than 1 token remains, sleep (1 - tokens) / rate and retry. Guard the state with an asyncio.Lock because many coroutines share it.
- Error recovery: separate retryable errors (timeouts, 429, 5xx, connection resets) from permanent ones (validation 4xx). Retry with exponential backoff and full jitter, sleeping uniform(0, min(cap, base * 2**attempt)), honour Retry-After, and wrap each call in asyncio.wait_for, or asyncio.timeout on 3.11+.
- After the last attempt, record the failure (task, error, attempts) in a dead-letter list instead of raising, so one bad task cannot kill the pool. Catch errors inside each worker: TaskGroup cancels its siblings on the first unhandled exception. Never swallow CancelledError, which has derived from BaseException since 3.8, so a plain except Exception does not catch it.
- Shutdown: await queue.join(), then cancel the workers and gather them. Memory stays bounded by the queue size. Test with a fake clock and a fake client to confirm the rate is never exceeded and that retries of tasks with side effects use idempotency keys.
Follow-up
- The rate limit has to be shared across several processes or machines. How do you enforce it, for example with a Redis bucket updated atomically in a Lua script?
- Some tasks are CPU-heavy. What changes, given the GIL?
- A task fails after its side effect succeeded but before it was acknowledged. How do you make the retry safe?
Discuss the basics of Linux and DevOps practices necessary for maintaining production reliability.
Discuss the basics of Linux and DevOps practices necessary for maintaining production reliability.
Approach
- Host triage sequence: uptime and top (load average against core count), free and vmstat (memory pressure, swap), df -h and df -i (space and inodes), iostat (disk wait), ss -tlnp (listening sockets), lsof and ulimit -n (file descriptor exhaustion), journalctl -u for the service, and dmesg for OOM-killer events. Use strace on a hung process to see which syscall it is stuck in.
- Process lifecycle: run services under systemd with Restart=on-failure and resource limits, or in containers. Handle SIGTERM to stop accepting work and drain. Kubernetes sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds, which defaults to 30 seconds.
- Delivery: CI that runs tests on every change, immutable container images tagged with the commit SHA, infrastructure as code (Terraform or the cloud's equivalent), and rolling or canary deploys with automatic rollback when error rate or latency breaches the SLO.
- Health and observability: separate liveness (restart me) from readiness (stop sending me traffic) so a slow dependency does not cause restart loops. Use RED metrics for services and USE metrics for resources, centralised structured logs, and alerts on SLO burn rate rather than raw CPU.
- Hygiene: keep secrets in a secrets manager, not in the repo or images; grant least-privilege IAM; test backups by actually restoring them; write runbooks for the top alerts and blameless postmortems after incidents.
Follow-up
- df says the disk is full but du finds far less. What is going on and how do you confirm it?
- A process disappears with nothing in its own logs. Where do you look?
- How do you roll back a deploy that included a database migration?
Replace offset paging on the resource feed with keyset
resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.
Approach
- Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
- Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
- Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
- Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
Follow-up
- The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
- Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
Stop tag and share joins from fanning out a page
resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.
Approach
- Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
- Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
- Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
- Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
Follow-up
- The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
- A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
How do you approach running agent evaluations at scale?
How do you approach running agent evaluations at scale?
Approach
- Split the harness into three versioned parts: a task set (environment image, tools, starting state, success check), a runner, and a grader. Pin the model version, system prompt, tool versions and sampling settings so any score can be reproduced.
- Run each episode in an isolated sandbox (container or microVM) with CPU and memory limits, a step budget and a wall-clock timeout, and tear it down afterwards. Store the full trajectory (messages, tool calls, tool outputs, final state) in object storage, keyed by run, task and attempt.
- Grade programmatically wherever the outcome can be checked: hidden unit tests, end-state assertions, exact answers. Use an LLM judge or a human rubric only where it cannot, and calibrate the judge against a human-labelled sample before trusting it.
- Agent runs are stochastic, so sample n attempts per task. Report mean pass@1 with a bootstrap confidence interval over tasks, and use the unbiased pass@k estimator 1 - C(n-c,k)/C(n,k). Compare two models with paired tests on the same tasks.
- Keep infrastructure failures (sandbox crash, provider 429, timeout in a tool) separate from model failures so they are retried rather than scored. Cap concurrency to the model API's rate limits, and write results idempotently so a retried attempt is never counted twice.
Follow-up
- Pass rate dropped 3 points after a model update: how do you tell a real regression from sampling noise?
- How would you detect an agent gaming the grader, for example by editing the tests it is checked against?
- A tenth of your environments are flaky. What do you do with those tasks?
Can you explain how you would design a high-throughput asynchronous data pipeline for training datasets?
Can you explain how you would design a high-throughput asynchronous data pipeline for training datasets?
Approach
- Lay out the stages: ingest, schema validation and normalisation, exact and near-duplicate removal, quality filtering or scoring, optional human review, shard writing, then publishing a manifest. Decouple the stages with a durable log or queue (Kafka, Pub/Sub, SQS) or with batch handoffs through object storage, and make each stage idempotent.
- For throughput, batch records and bound each stage's concurrency. Use bounded queues so a slow stage pushes back on the stages before it instead of growing memory. Use asyncio with a semaphore for I/O-bound stages (API calls, downloads) and process pools for CPU-bound ones (tokenisation, hashing).
- Treat delivery as at-least-once and make writes idempotent by keying them on a content hash, so a retry overwrites rather than duplicates. Send records that fail repeatedly to a dead-letter store with the reason, so one bad record cannot stall a partition.
- Partition by a hash of the record or document id so exact dedup can run locally within each partition. Use MinHash with LSH for near-duplicates. Write fixed-size Parquet or JSONL shards, and write the manifest last so a dataset version is either complete or invisible.
- Record lineage (source, transform code version, filter thresholds) per version. Gate publishing on data checks (schema, null rates, length distribution, rejection rate by reason), and track per-stage throughput and lag.
Follow-up
- A quality filter had a bug for two weeks. How do you rebuild the affected dataset versions?
- The same document arrives from two sources in different partitions. How do you dedup it?
- Which stage becomes the bottleneck at ten times the volume, and how would you see it coming?
What strategies do you use to expose and mitigate meaningful model failure modes across domains like finance o
What strategies do you use to expose and mitigate meaningful model failure modes across domains like finance or code?
Approach
- Start from a failure taxonomy for each domain. For code: does not compile, fails hidden tests, passes tests but is insecure or wrong on edge cases, calls APIs that do not exist. For finance: arithmetic slips, wrong units or periods, misread tables, claims the source does not support.
- Build tasks that can be checked. For code, use hidden test suites, and strengthen weak suites with mutation testing so wrong solutions cannot pass. For finance, compute ground truth programmatically from the source data. Add perturbed variants (changed numbers, renamed variables, irrelevant context) to separate memorisation from reasoning.
- Report error rates by slice and category, with counts and confidence intervals rather than a single headline score. Cluster failing outputs, for example by embedding similarity, to surface categories the taxonomy misses, and have domain experts review a sample to confirm the grader is right.
- Mitigate by turning findings into targeted data (corrected examples, preference pairs, verifiable RL tasks), then re-measure on a held-out split that was never used to generate that data. Keep every fixed failure in a regression suite.
- Trade-off: a set tuned too closely to known failures overfits and stops finding new ones. Rotate fresh held-out tasks in and watch for contamination.
Follow-up
- How do you show a rubric-based grader agrees with domain experts? Which agreement statistic would you use, such as Cohen's kappa?
- You have found ten failure modes. How do you decide which to fix first?
- How would you check whether a benchmark's tasks have leaked into training data?
How would you architect reward signals for reinforcement learning pipelines?
How would you architect reward signals for reinforcement learning pipelines?
Approach
- Choose the reward type by whether correctness can be checked. Verifiable rewards (tests pass, exact final answer, valid output format) suit RLVR. Where it cannot be checked, use a reward model trained on human pairwise preferences, typically Bradley-Terry: P(a preferred over b) = sigmoid(r(a) - r(b)).
- Run reward as its own service, separate from the trainer. Rollouts arrive with a prompt id, completion and metadata, and come back with a scalar reward, its component scores and the verifier or reward-model version. Code verifiers run sandboxed with no network and with timeouts, so a run can be reproduced exactly.
- Plan for reward hacking: special-casing visible tests, printing expected outputs, exploiting a reward model's length bias. Counter it with hidden tests, a KL penalty toward the reference policy, length normalisation or penalties, and regular human audits of the highest-reward samples.
- Shaping trade-off: binary pass/fail is clean but sparse, while partial credit (fraction of tests passed) gives a denser signal but can reward partial solutions. Log every component separately even when you combine them into one weighted reward.
- For on-policy RL, reward computation sits on the critical path. Batch verifier calls, autoscale verifier workers on queue depth, and cache by a hash of prompt and completion. A timeout returns a defined reward and a flag, so it never blocks the training batch.
Follow-up
- Mean reward is rising but held-out quality is flat. What do you check first?
- Your verifier is flaky on 2% of runs. What does that noise do to training, and how do you contain it?
- How would you combine a preference reward model with a verifiable reward on the same task?
Design a distributed task orchestration system capable of handling thousands of concurrent model evaluation jo
Design a distributed task orchestration system capable of handling thousands of concurrent model evaluation jobs.
Approach
- Pin down the requirements first. A job is an evaluation run that fans out into many tasks. Thousands of jobs run at once, on mixed resources (CPU sandboxes, GPUs, external model APIs with rate limits). You need priorities, cancellation and live progress.
- Components: a submit API that writes job and task rows to a metadata store such as Postgres, with explicit states (pending, leased, running, succeeded, failed, cancelled); a scheduler that expands jobs and enqueues runnable tasks; stateless workers that lease a task, heartbeat, and write results to object storage with a summary row in the database.
- Correctness under failure: leases with a visibility timeout give at-least-once execution, and a task whose worker dies is re-queued when its lease expires. Make results idempotent per task and commit them with a conditional update on the lease owner or a fencing token, so a stale worker cannot overwrite a newer attempt. Retry with exponential backoff up to a cap, then dead-letter.
- Capacity and fairness: per-job and per-team concurrency quotas, priority queues, and token buckets for each model provider's rate limit, so one large job cannot starve the rest. Autoscale workers on queue depth and age of the oldest task. Workers check for cancellation at each heartbeat.
- Build versus buy: Postgres as a queue with SELECT ... FOR UPDATE SKIP LOCKED is enough at moderate rates. Temporal, Celery, Ray or Kubernetes Jobs give you retries and scheduling for free but are less flexible. Name the throughput at which you would switch.
Follow-up
- A worker loses its heartbeat but keeps running. How do you stop two workers committing results for the same task?
- How would you add dependencies between tasks, for example generation before grading?
- What happens to running jobs if the scheduler process crashes?
How would you ensure fault tolerance and observability in an event-driven architecture utilizing message queue
How would you ensure fault tolerance and observability in an event-driven architecture utilizing message queues?
Approach
- Delivery semantics: most brokers deliver at least once, so consumers must be idempotent, through a processed-event table keyed by event id or through upserts. Acknowledge only after the side effect commits. On the producer side, use the transactional outbox pattern to avoid the dual write between the database and the broker.
- Failure handling: retry with exponential backoff and jitter up to a fixed number of attempts, then route to a dead-letter queue that keeps the original payload and the error. Kafka orders messages only within a partition, so key by entity id when ordering matters.
- Backpressure: bound each consumer's concurrency and scale on consumer lag. In a Kafka consumer group each partition is assigned to one consumer, so consumers beyond the partition count sit idle. Partition count caps parallelism.
- Observability: carry trace context (W3C traceparent) and a correlation id in message headers, and write structured logs keyed by event id. Track publish and consume rates, lag, age of the oldest message, processing latency percentiles, retry counts and DLQ depth. Alert on growing lag and DLQ depth, not on individual errors.
- Recovery and evolution: keep enough retention to replay from an offset after a bad deploy. Use a schema registry with backward-compatible changes so old consumers keep working during rollouts.
Follow-up
- Consumer lag has grown steadily for an hour. Walk through how you diagnose it.
- How do you replay a dead-letter queue safely once the bug is fixed?
- Where does Kafka's exactly-once support stop applying, for example on writes to an external database?
How do you architect internal platform APIs for high-velocity engineering and research teams?
How do you architect internal platform APIs for high-velocity engineering and research teams?
Approach
- Start from the callers. Researchers mostly call from notebooks and Python scripts and want a thin typed SDK. Services want a stable network contract. Model resources (datasets, jobs, runs, artifacts) with consistent naming, cursor-based pagination and filtering.
- Long-running work returns immediately with an operation or job id (HTTP 202) plus a status endpoint, a webhook or a stream. Accept idempotency keys on create calls so a retried submit never launches a duplicate job.
- Define contracts in OpenAPI or protobuf and generate the clients from them. Make only additive changes within a version, publish a deprecation policy, and in protobuf never reuse or renumber a field.
- REST/JSON versus gRPC: gRPC gives typed contracts and streaming for service-to-service calls. REST is easier from notebooks and curl. A gateway can serve both from one definition.
- Operability: service-account authentication, per-team quotas and rate limits, structured error codes with request ids, and per-client usage metrics so you know who a breaking change would hit. Ship docs with runnable examples.
Follow-up
- How do you change a field's type without breaking the teams already calling it?
- One team's batch script is saturating the API. What protects everyone else?
- When does an internal SDK become a liability rather than a help?
How do you optimize memory and throughput when processing large-scale synthetic datasets?
How do you optimize memory and throughput when processing large-scale synthetic datasets?
Approach
- Stream rather than load. Read in batches (a pyarrow dataset scanner, pandas chunksize, JSONL line by line), process with generators and write output shards incrementally, so memory grows with batch size, not dataset size.
- Choose the format deliberately. Columnar Parquet with zstd compression lets you read only the columns you need and skip row groups with predicate pushdown. Keep data in Arrow or NumPy buffers instead of one Python object per row, and use dictionary encoding for repeated strings and the narrowest dtypes that fit.
- Match parallelism to the bottleneck. CPU-bound Python work is limited by the GIL, so use process pools per shard and pass shard paths, not large pickled objects. I/O-bound generation calls get asyncio with bounded concurrency and rate limits. Across machines, Ray, Spark or Dask can distribute shards sized to fit each worker's memory.
- Profile before optimising: tracemalloc or memray for memory, py-spy or cProfile for CPU, and per-stage records-per-second, to find whether generation calls, tokenisation, dedup or I/O is the limit.
- Dedup without holding text in memory: store fixed-size hashes for exact duplicates, use MinHash with LSH for near-duplicates, and when the hash set outgrows RAM, partition by hash prefix or external-sort. A Bloom filter gives approximate membership with a false-positive rate you choose and no false negatives.
Follow-up
- A handful of very large records keeps OOM-killing workers. How do you handle the skew?
- Throughput degrades steadily over a long run. What would you suspect and measure?
- The dedup set no longer fits on one machine. What is your next design?
One customer endpoint stalls deliveries to every other destination
The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.
Approach
- Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
- Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
- Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
- Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.
Follow-up
- The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
- Where should the 10-second timeout number actually come from?
The plan moves from mapping your stories to the loop, through the job-lifecycle design pattern, to a coded worker and a full mock. Each day uses this loop's reported questions or PracHub practice-set exercises that fit the same skills.
Prepare, practise & reflect
One practical outcome each day. Spend longer where you need it.
0 / 7 done01Map the loop and inventory your projects
- Write the five reported rounds in order and, for each, the question types you would prepare for it.
- List three projects you owned in production and, for each, the system diagram, one failure and one metric.
- Check yourself against the listed stack (Python or JavaScript, message queues, AWS or GCP) and mark which items you cannot yet discuss with specifics.
Deliverable: A one-page loop map, a three-project inventory with diagrams and metrics, and a gap list against the stack.
02Evaluations, failure modes and reward signals
- Outline an answer to running agent evaluations at scale: task spec, sandbox, concurrency, reruns for flaky results, result storage.
- Write a failure-mode taxonomy for a code-generation model and the pipeline step that surfaces each category.
- Write down the reward-signal design for a verifiable case and a human-feedback case, including reward-hacking risks.
Deliverable: Three written answer outlines (evaluations at scale, failure modes, reward signals), each with a stated assumption and a risk.
Practice prompt ↗Practice prompt ↗Practice prompt ↗03Data pipelines and memory
- Sketch the training-data pipeline as stages (ingest, validate, deduplicate, shard, write) with a queue or buffer between each.
- Decide the unit of work and batch size, and write how you keep memory flat with streaming and bounded buffers.
- Practise the offset-versus-keyset paging exercise from the PracHub practice set, since large result sets from such a pipeline have to be paged safely.
Deliverable: A labelled pipeline diagram with stated throughput assumptions, a memory plan for large synthetic datasets, and a worked keyset-paging solution.
Practice prompt ↗Practice prompt ↗Practice prompt ↗04Orchestration, queues and fault isolation
- Design the orchestration system for thousands of concurrent evaluation jobs: scheduler, queue, leases, retries, dead-letter queue, result store.
- List the fault-tolerance and observability measures for an event-driven design: idempotency keys, queue depth, consumer lag, oldest-message age, tracing.
- Work through the PracHub debugging exercise where one customer endpoint stalls deliveries to every other destination, then say how the same isolation applies to your design.
Deliverable: An orchestration diagram with one failure walked through end to end, a metrics list, and a written diagnosis of the stalled-deliveries exercise.
Practice prompt ↗Practice prompt ↗Practice prompt ↗05Cloud choices, platform APIs and Linux basics
- Build an AWS versus GCP comparison for queues, compute and storage in low-latency async workloads, using services you have actually used.
- Outline the internal platform API: resource model, versioning, idempotency keys, pagination, error format, auth and quotas.
- Write a Linux and DevOps checklist: inspecting processes and resources, signals and graceful shutdown, logs, deploy and rollback.
- Do the PracHub exercise on joins that fan out a page, to sharpen how your API's list endpoints should aggregate related data.
Deliverable: A one-page cloud comparison, a platform API outline, a production-reliability checklist and a worked fan-out fix.
Practice prompt ↗Practice prompt ↗Practice prompt ↗Practice prompt ↗06Code the resilient worker
- Implement a worker that polls a queue with a concurrency cap, a token-bucket rate limiter, per-job timeouts and retry with jitter.
- Add graceful shutdown, a dead-letter path for poison jobs and idempotent handling of repeated deliveries.
- Write tests for a timeout, a transient failure, a permanent failure and shutdown while jobs are in flight.
Deliverable: A working, tested worker module with a README stating its assumptions and failure behavior.
Practice prompt ↗07Behavioral stories and full mock
- Write the three reported behavioral stories (scaling bottleneck, conflicting requirements with research teams, ambiguity) with a decision, a number and a trade-off in each.
- Give one design answer aloud from a blank page, using your standard shape, and record it.
- Review the recording and list the three weakest moments to fix before the loop.
Deliverable: Three finished story outlines, one recorded design answer, and a short fix list from your review.
Practice prompt ↗Practice prompt ↗Practice prompt ↗Practice prompt ↗Expand any day for tasks and deliverables. Your progress is saved on this device.
The reported behavioral prompts focus on three things for this role: owning a production system through a scaling problem, ranking conflicting infrastructure needs when researchers are the customers, and working without documentation or precedent. Prepare each story with your own decision, a measured result and a trade-off, and keep it tied to pipelines, queues or platform work where you can.
Tell me about a time you owned a complex production system end-to-end through a major scaling bottleneck.
Tell me about a time you owned a complex production system end-to-end through a major scaling bottleneck.
Approach
- Choose a system you personally ran in production, ideally queue-based or pipeline-shaped, and open with the symptom in measurable terms: queue depth growing, p99 latency doubling, a nightly job missing its window, cost per run.
- Spend most of the answer on diagnosis and decision. Say how you located the bottleneck (profiling, tracing, per-stage metrics), which alternatives you weighed (shard, batch, cache, scale out, rewrite) and why you chose one.
- Show end-to-end ownership beyond the fix: the rollout (canary, flag, rollback plan), the alerts and dashboard you added, the load test or runbook you left behind.
- Close with a before and after number and the trade-off you accepted, such as added complexity or eventual consistency. Use I for your decisions and name teammates' parts separately.
- The usual sinking mistake is a team story with no personal choice or no measurement; the second is stopping at the fix without saying how you kept it fixed.
Follow-up
- How did you know your change, and not something else, produced the improvement?
- What would have broken first if load had grown another ten times?
- What did you give up by choosing that solution over the alternatives?
How do you prioritize conflicting infrastructure requirements when partnering directly with research teams?
How do you prioritize conflicting infrastructure requirements when partnering directly with research teams?
Approach
- Answer the 'how' with a short method, then prove it with one real case. A workable method: collect each request with its deadline and what it blocks, translate each into the experiment or run it enables, and look for the shared need underneath.
- Rank by blocking impact, cost of delay and reversibility, not by who asks loudest. Write the trade-off down in one place so research and infrastructure see the same reasoning.
- In your example, name the conflict concretely, for instance one team wanting low-latency single-run results while another wanted bulk throughput, or a researcher wanting one-off scripts while the platform needed a stable interface. Say what you shipped first and why.
- Show the communication step: telling the team whose request waited what the date is and what interim workaround they have. Mention any interface (a platform API, a config surface) that let you serve both groups.
- Avoid two answers: 'I tell everyone yes and work late' and 'I follow whatever the founders say' with no reasoning.
Follow-up
- Two requests are equally urgent and you can only do one. How do you decide?
- How did you handle the researcher whose request you deferred?
- When have you said no to a request, and what happened afterwards?
Describe a situation where you had to operate in high ambiguity without clear documentation or precedent.
Describe a situation where you had to operate in high ambiguity without clear documentation or precedent.
Approach
- Pick a case where the documentation or precedent was genuinely absent: an undocumented legacy system, a new kind of workload, or a vague goal from a stakeholder. State the situation in two sentences.
- Show how you converted unknowns into facts: list what you knew and what you assumed, ask the two or three questions that unblock most decisions, then run a small time-boxed spike or experiment to test the riskiest assumption.
- Explain how you kept the first step reversible, such as a flag, a shadow run or a small slice of traffic, and how you checked in with the stakeholder before committing further.
- Say what you wrote down so the next person had precedent: a design note, a runbook or an example. Give the outcome and one assumption that turned out wrong.
- The mistake that sinks it is a story where you waited for clarity, or one where you guessed and got lucky with no reasoning shown.
Follow-up
- How did you decide you had enough information to commit?
- Which of your early assumptions was wrong, and how did you find out?
- What did you leave behind so the next engineer would not face the same gap?
- 01
Tell me about a time you owned a complex production system end-to-end through a major scaling bottleneck.
- 02
How do you prioritize conflicting infrastructure requirements when partnering directly with research teams?
- 03
Describe a situation where you had to operate in high ambiguity without clear documentation or precedent.
- 04
Tell me about a time a failure mode in a queue or pipeline you owned surprised you, and what you changed afterwards.
- 05
Describe a time you shipped a pragmatic first version under time pressure and how you improved it later.
How many rounds are there and how long does the process take?
Candidates report five rounds over about 4-6 weeks: Recruiter Screen, Technical Deep-Dive, Structured Technical Conversations, System Design Evaluations and a Hands-On Work Trial. Reports vary by team and seniority, so ask the recruiter for your exact sequence.
PracHub Software Engineer practice ↗Is the role remote or hybrid?
Candidates report that the role is fully onsite in San Francisco, with daily office attendance. Confirm that you can meet this at the recruiter screen rather than at the end of the process.
PracHub Software Engineer practice ↗Which language and cloud should I prepare in?
The requirements candidates describe list Python and/or JavaScript (Node.js, Next.js) and production experience with AWS or GCP. Use the language you are strongest in for coding practice, and be ready to discuss queue and compute services from whichever cloud you have used.
PracHub Software Engineer practice ↗Do I need reinforcement learning experience?
Candidates describe backend and distributed-systems depth as the core requirement and RLHF or RLVR familiarity as a bonus. Learn the working vocabulary: verifiable versus human-feedback rewards, reward hacking, label noise and failure-mode taxonomies, so the evaluation and reward questions are not unfamiliar.
PracHub Software Engineer practice ↗What does the Hands-On Work Trial involve?
It is described as a practical assessment through a work trial, and its length and task are not detailed in reports. Ask the recruiter for the format, then prepare by building a tested background worker with concurrency limits, rate limiting, retries and timeouts.
PracHub Software Engineer practice ↗How should I prepare for the system design questions?
Practise the job lifecycle (enqueue, lease, run, acknowledge, retry, dead-letter, store result) until you can draw it quickly, then apply it to task orchestration, event-driven fault tolerance and platform APIs. For each design, walk through one failure and name the metrics that would show it.
PracHub Software Engineer practice ↗Sources & methodology 3 sources ↗
No official company page is cited. Rounds and questions come from candidate reports and PracHub editorial material; each source shows the date it was read.
- 01Afterquery Software Engineer candidate reports ↗
Company-reported rounds, questions and FAQ.
Candidate reports · Accessed 2026-09-22 - 02PracHub Software Engineer practice ↗
PracHub practice material, not company-reported.
PracHub page · Accessed 2026-09-22 - 03PracHub preparation framework ↗
PracHub preparation guidance.
PracHub page · Accessed 2026-09-22
Explore related engineering environments and compare the preparation focus.
Zelis Software Engineer Interview Guide
Zelis software engineer preparation through three original worked cases: a tested tenant-scoped reconciliation function, uncertain payment outcomes and a remittance incident. Includes sample records, recovery decisions, self-review criteria
Aerospike Software Engineer Interview Questions & Guide 2026
Aerospike database-engine preparation through an original single-owner Python key-value store with bounded logical index accounting, disk-offset reads, conditional writes, tombstones, process-restart recovery and 3,240 checks. Existing 14 p
Roku Software Engineer Interview Guide
Prepare for Roku software interviews with tested ring-buffer and LRU models, invariants, role-specific fundamentals and clear concurrency follow-ups.
Zetes Software Engineer Interview Guide
What Zetes asks Software Engineer candidates, from reported rounds and questions.