Candidates report two broad tracks in Thinking Machines Lab's Software Engineer role. Data Infrastructure work centres on large-scale data for model training: distributed compute in Spark or Ray, storage formats such as Parquet and Delta Lake, and orchestration and streaming with Airflow or Kafka. Distributed deduplication and scalable web crawlers are listed as advanced skill areas for this track. The role description also mentions working with machine learning researchers on training data catalogs, traceability and reproducibility. Full Stack work covers APIs and user interfaces, with Python or Rust on the backend and React with TypeScript on the frontend, plus developer tooling, CI/CD and production observability.
Candidates describe four stages. First comes an initial screening, sometimes a recorded video interview with three foundational questions. Next is a take-home backend exam in any programming language, usually with a couple of days to complete, that also includes written technical and behavioral questions and a slide deck on the technical and business sides of your solution. Then comes a panel where you present that deck, defend your architecture and answer deep-dive questions from the engineering team. The last stage is a culture fit interview. Candidates also report anti-cheating mechanisms on the coding assessment.
For preparation, the take-home matters most because the panel questions candidates report are about your own submission: why you chose the stack, how the solution addresses the business problem in the prompt, where it breaks at ten times the traffic, and which metrics would show it meeting its SLAs. Candidate reports do not tie the design and data-infrastructure questions to a specific round. Those cover ingestion pipelines, rate limiting, a fault-tolerant academic paper crawler, Kafka and Airflow orchestration, distributed key-value stores and dataset versioning schemas, so prepare them as a category.
Initial Screening
reportedCandidates describe this stage as an initial screen, sometimes a recorded video interview with three foundational questions. A recording has no interviewer to rephrase a question or ask what you meant, so each answer has to stand on its own: give the direct answer first, then one concrete example, then stop. Candidates do not report what the three questions are. Prepare the ground any foundational screen for this role can cover: your background, which track you are aiming at, and one piece of technical work you can explain end to end.
What to demonstrate
- Check your own recordings: each answer should stand alone, with the direct answer before the context, since a recorded format leaves no room for a follow-up
- Check that you can name your target track, Data Infrastructure or Full Stack, in one sentence, with evidence from your own work
- Check that you can explain one project end to end: the problem, your part, the stack and the result
How to prepare
- Write three foundational questions for this role, record your answers, play them back, and cut any answer that opens with context instead of the answer
- Write one sentence naming your track and the evidence for it: Spark, Ray and data lake formats for Data Infrastructure; Python or Rust APIs with React and TypeScript for Full Stack
- Choose the one project you will describe and fix its figures (data size, users, team size, your part) so they match what you later say on the panel
Take-Home Coding Exam
reportedCandidates report a backend take-home in any programming language, usually with a couple of days to finish. It comes with written technical and behavioral questions and a slide deck explaining the technical and business sides of your solution. The example scenarios candidates report are production-style services. One aggregates several public APIs, caches the results and exposes a REST interface. Another is a containerized service that accepts file uploads, processes them asynchronously and updates a relational database. Candidates also report anti-cheating mechanisms on the assessment. Treat the code, the written answers and the deck as three equally important deliverables, and write every line yourself, because the panel reportedly asks about specific lines.
What to demonstrate
- Check that your codebase is organised logically, that any design pattern you apply fits the problem as stated, and that names make the code readable without comments
- Check that your endpoints are clean, that requests are handled asynchronously where the prompt calls for it, and that database connections are managed deliberately
- Check that the submission includes unit and integration tests, instructions for running the service locally, and a container setup
- Check that errors are handled explicitly, queries are efficient, and any custom middleware the prompt calls for is in place
How to prepare
- Build one practice service from a reported example scenario in the language you will use, with tests, a Dockerfile and a README that explains how to run it
- Write your answers to the written questions in full sentences as soon as the code is finished, while the trade-offs are fresh, and review them as carefully as the code
- Keep a decisions log while you build: each choice, the alternative you rejected and why. It becomes the backbone of your deck
- Work this guide's design exercise on making run creation safe when the client retries, to practise retry and failure handling in the service you submit
Panel Interview
reportedCandidates report presenting their take-home solution from their own slides, defending their architectural choices and answering deep-dive questions from the engineering team. The questions reported for this stage are about your own submission: walk through the architecture and why you chose that stack, explain how the solution addresses the business problem in the prompt, name the bottlenecks at ten times the current traffic and how you would resolve them, and say which monitoring and alerting metrics would show the service meeting its SLAs. Candidates also describe being pointed at specific lines of their submitted code and asked to explain them, optimise them or refactor them for a new requirement. If you wrote the code and the decisions log yourself, none of this should catch you off guard.
What to demonstrate
- Check that your walkthrough of data flow, component boundaries and technology choices matches the code the panel has read
- Check that you can link each design choice to the business problem stated in the prompt
- Check that you can defend why you chose this database, framework or architectural style over the alternatives, and say what would change your mind
- Check that you can reason about scaling, cost and availability for your own service, not for a generic system
How to prepare
- Open the deck with the business problem and the requirement each component serves, then cover the architecture, then the trade-offs. Present it aloud to someone who has not seen the code
- For the ten-times-traffic question, find the first component in your service that saturates, the measurement that would show it, and the change you would make
- Define a small set of SLA-facing metrics for your service, such as latency percentiles, error rate and queue depth, with an alert condition for each
- Have a friend pick random lines from your submission and ask why each one is there, how to make it faster, and how to change it for a new constraint
Culture Fit Interview
reportedCandidates describe a dedicated culture fit interview as the final stage. Candidate reports do not tie specific questions to it. They do report behavioral questions for the role about four situations: resolving an infrastructure bottleneck across team boundaries, weighing technical debt against shipping a prototype quickly, working with a cross-functional partner such as an AI researcher whose priorities differed from yours, and starting a project from scratch with little documentation. Prepare one story for each, told through your own decisions, and keep the project facts consistent with what you said on the panel.
What to demonstrate
- Check that your cross-team story names the bottleneck, what you did about it and how you brought the other teams along
- Check that your technical-debt answer says what you deferred, why, and when you paid it back
- Check that your collaboration story states what the partner needed, where your priorities conflicted and how you resolved it
- Check that your ambiguity story shows the specific steps you took to create structure when documentation was thin
How to prepare
- Write one story for each reported behavioral theme, naming the decision you made, the evidence you used and the outcome
- For the technical-debt question, prepare one case where you shipped with debt and one where you refused to, with the reason for each
- Rehearse the researcher-collaboration story from the partner's side too: what they needed, what you gave up and what you held firm on
PracHub editorial advice for the preparation topics above.
Submitting take-home code you cannot explain line by line
Candidates report anti-cheating mechanisms on the coding assessment and panel questions about specific lines of the submission. Write the code yourself and reread every file before you submit. For each non-obvious line, be ready to say why it is there and what you would change under a new constraint. If you copied a library pattern you only half understand, replace it with code you can defend.
Treating the take-home's written questions as an afterthought
The take-home reportedly includes written technical and behavioral questions alongside the code. Set aside time for them instead of writing them in the last minutes. Answer each in full, specific sentences with a concrete example, and check that the design approach you describe there matches what your code actually does.
A deck that tours the code but never connects it to the business problem
One reported panel question asks how your solution addresses the core business problem in the prompt. Start the deck with that problem, map each component to the requirement it serves, and put your trade-offs on a slide with the cost of each. That way the business case is stated, not left for the panel to work out.
No numbers for your own service when asked about ten-times traffic or SLA monitoring
Both are reported questions about the service you built. Before the panel, load-test or reason through where your service saturates first: database connections, a synchronous external call, or a single worker. List the metrics and alert thresholds tied to the SLA you would promise. A generic answer about adding caches and replicas does not show that you understand your own system.
Answering data-infrastructure design questions with boxes and no failure path
Reported design questions include a multimodal ingestion pipeline for LLM training, deduplication over petabytes of text, a fault-tolerant academic paper crawler, and batch orchestration with Kafka and Airflow. For each, state the delivery guarantee, where deduplication happens and over what window, how a failed task retries without writing twice, and how you would re-run one partition. This guide's SQL rollup and late-data coding exercises drill exactly these points.
Choose a category, try a prompt, then open its approach, worked solution or follow-up when you need it.
Seal an hour under late data with bounded memory
Metering ingest reads 256 partitions at 10,000 to 40,000 events/second. Events carry occurred_at and ingested_at, and during a producer replay the gap between them is hours. Seal each UTC hour once no more than 50 parts per million of that hour's eventual quantity can still arrive, using memory that does not grow with the size of the replay. Define the watermark, the lateness parameter and how you measure it, the structure holding open hours, and the write that performs the seal. State what an idle partition does to your watermark.
Approach
- Two clocks, two jobs. Bucket by
occurred_at, because that is the hour the customer is billed for, and advance the watermark oningested_at, because that is what the fold has consumed and whatsource_max_ingested_atrecords. Conflating them is what makes late data invisible. - The global watermark is the min over partitions of each partition's committed
ingested_at, not the max: the fold is trustworthy only as far as the slowest partition. The consequence is that one idle partition pins the watermark forever and nothing seals, so an idle partition must promote its watermark to wall clock after a stated idle timeout, and that timeout becomes a correctness parameter, because a partition that is slow rather than idle gets sealed past. - Choose the lateness L from the measured distribution of
ingested_at - occurred_at, weighted by quantity rather than by event count. The target is 50 ppm of the hour's quantity, and a replay is rare in events while carrying disproportionate mass, so an event-weighted quantile picks an L that is comfortably wrong at exactly the moment it matters. - Measure that quantile in bounded memory. A Greenwald-Khanna summary gives epsilon-approximate quantiles in O((1/epsilon) log(epsilon n)) space; a t-digest costs more per merge but has relative error that tightens at the tails, which is the half of the distribution you are reading at p99.99. Keep a separate summary per tenant class, because one tenant's batch importer is not the population.
- Hold open hours in a min-heap keyed by
hour_start. When the watermark advances, pop every hour withhour_end + L < Wand seal it: O(log H_open) per advance and O(1) amortised per event to touch its bucket. Memory is open hours multiplied by distinct(tenant, workspace, sku)keys, so cap the number of simultaneously open hours and spill the oldest intousage_rollup_hourlyasstatus='open'with arevisionbump. While an hour is open the row is upsertable, so the store is your overflow. - The seal itself is a conditional write:
update ... set status='sealed', sealed_at=now() where status='open' returning .... Two sealers race on every restart, and the loser must see zero rows and stop rather than write a second value. After the seal, an event for that hour is not an upsert but an adjustment, andsource_max_ingested_atis what proves it arrived afterwards.
Worked solution 40 min
- Replay a day of events with a synthetic lateness distribution: 99.9% under two minutes, plus a 0.05% tail at four to six hours that carries 3% of total quantity.
- Compute the p99.99 lateness two ways, event-weighted and quantity-weighted, and put the two numbers side by side.
- Implement the min-heap of open hours with the watermark as the min over 256 partitions, then stall one partition for 20 minutes and observe what seals.
- Set the idle-partition timeout to 60 seconds, repeat the stall, and measure how much quantity arrives after the seal.
- Attempt the seal from two workers at once and confirm the conditional update lets exactly one through.
Follow-up
- A replay starts during the sealing window for a period you are about to close. What do you do, and what is the customer-visible consequence of each option?
- Your measured quantity-weighted p99.99 lateness is six hours and the invoice must be issued at 02:00 UTC on the first. How do you reconcile those two numbers?
- How would you detect that L has drifted before it costs you an hour's quantity?
Locate a billing reconciliation gap without rescanning ninety million events
A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.
Approach
- Run the free query first. Sum raw quantity for the period restricted to
ingested_at <= source_max_ingested_atof the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query. - Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
- Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
- Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is
occurred_atversusingested_atbucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same(tenant_id, idempotency_key)present in twoingested_daypartitions is the dedup horizon losing a retry that crossed midnight. - Make the next bisection cheap by storing the aggregate you keep recomputing. A per-
(tenant_id, ingested_day)count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces. - Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own
source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.
Follow-up
- The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
- How do you distinguish a duplicate from a restatement, given
revisionandrecomputed_aton the rollup? - Ingest is still running while you investigate. What makes your two numbers comparable at all?
Parse and verify a timestamped multi-signature webhook header
An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.
Approach
- Parse in one scan: split on
,, then on the first=only, since a value may itself contain=under a future scheme. Accepttexactly once and treat a secondtas a reject rather than last-wins. Push everyv1onto a short list and ignore any other key, so av2can be introduced later without breaking this verifier. - Say what is signed: HMAC-SHA256 over the exact byte string
<t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp. - Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
- Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate
acc |= a[i] ^ b[i]across the whole length, and testacc == 0at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network. - Apply the tolerance as a two-sided bound, rejecting when
|now - t| > 300seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window. - Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.
Follow-up
- The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
- A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
- How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?
Rebuild an hourly rollup with deduplication and late-arrival accounting
From usage_event (event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at, ingested_at), produce the values usage_rollup_hourly should hold for one tenant over one day: per (workspace_id, sku, hour_start) the deduplicated quantity_sum, event_count and source_max_ingested_at, bucketed by occurred_at. Duplicates share (tenant_id, idempotency_key). Also report, per hour, the running total across the day and the share of quantity that arrived more than two hours after the hour began. Write the query, and state which duplicates a daily unique index cannot catch.
Approach
- Deduplicate in its own CTE before any aggregation, because a SUM cannot be un-summed:
row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id) = 1. Include the tiebreaker. Without it the surviving row is non-deterministic when two duplicates share an ingested_at, and a rollup described as deterministically recomputable then disagrees with itself between runs. - Bucket on occurred_at and nothing else, and pin the timezone explicitly.
date_trunc('hour', timestamptz)truncates in the session's TimeZone setting, so the same query run by a session set to a non-UTC zone buckets differently; use the three-argumentdate_trunc('hour', occurred_at, 'UTC')on PostgreSQL 16 or later, ordate_trunc('hour', occurred_at at time zone 'UTC') at time zone 'UTC'before that. Filterenvironment = 'production'explicitly, since metering covers three environments and billing covers one. - Aggregate to the grain with
sum(quantity),count(*)andmax(ingested_at). The last is not decoration: it is the watermark the row consumed up to, and without it there is no way to prove afterwards what a number did and did not include. - Compute the late share inside the dedup-and-aggregate step as a conditional aggregate,
sum(quantity) filter (where ingested_at > hour_start + interval '2 hours'), then divide by the hour's total. Compute the running total as a window over the already aggregated rows:sum(quantity_sum) over (partition by workspace_id, sku order by hour_start rows between unbounded preceding and current row). Running either over raw rows puts the duplicates back. - Answer the index question exactly. The unique constraint is on (ingested_day, tenant_id, idempotency_key), because a unique index on a partitioned table must contain the partition key. It therefore deduplicates only within one ingest day and admits a duplicate whose retry crosses midnight or whose replay runs a week later. That is why this CTE dedups across the whole window being recomputed, and why the dedup horizon is a correctness parameter rather than a retention cost.
- Keep the numeric type all the way through. quantity is numeric so the sums are exact; a cast to double precision anywhere in this pipeline reintroduces drift that surfaces only as a few unreconcilable cents per tenant per month, long after the query is out of anyone's mind.
Worked solution 40 min
- Generate a day of events for one tenant across three workspaces and two SKUs, inject 2% duplicates sharing an idempotency_key with a later ingested_at, and inject 1% whose ingested_at is six hours past occurred_at.
- Write the dedup CTE alone and confirm its row count drops by exactly the injected duplicate count.
- Aggregate and compare quantity_sum against a ground truth computed outside SQL from the generator's own records.
- Add the two window and filtered aggregates, and confirm the final hour's running total equals the ungrouped day total for each (workspace, sku).
- Run the whole query twice over static data and diff the two outputs.
- Re-run with the session TimeZone set to a non-UTC zone and confirm hour_start does not move.
Follow-up
- A dispute forces the same recompute over 40 days for one tenant. What changes about the dedup CTE's memory use and the chosen plan, and what would you do about it?
- Two runs a minute apart return different quantity_sum values for an hour that is already closed. Give two mechanisms that produce that, and the single query that distinguishes them.
- Express the same rollup incrementally so it does not re-scan the day each time the watermark advances. What does the incremental version stop being able to answer?
Decide which facts an invoice line copies instead of joining
invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.
Approach
- Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
- Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
- Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
- Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason
source_rollup_watermarkis stored per line: without it, nobody can reconstruct what the customer was told. - Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
- Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.
Follow-up
- Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
- A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
- Where does currency live, and what breaks if a tenant's billing currency changes between two periods?
How would you design a high-throughput data ingestion pipeline that pr…
How would you design a high-throughput data ingestion pipeline that processes multimodal data for LLM training?
Approach
- State the consistency you need, and where you are willing to be stale.
- Choose a partition key and say what query it makes expensive.
- Name the failure you are designing for, then the recovery path.
Follow-up
- What breaks first when traffic grows ten times?
- What would you drop to keep the system up under load?
Implement a custom rate-limiting middleware for a Python-based API gat…
Implement a custom rate-limiting middleware for a Python-based API gateway.
Approach
- Name the read and write paths separately; they rarely have the same bottleneck.
- State the consistency you need, and where you are willing to be stale.
- Name the failure you are designing for, then the recovery path.
Follow-up
- What would you drop to keep the system up under load?
- How does this behave when that dependency is down for an hour?
Describe how you would build a fault-tolerant web crawler designed to …
Describe how you would build a fault-tolerant web crawler designed to scrape and catalog academic papers.
Approach
- State the consistency you need, and where you are willing to be stale.
- Fix the scope first: who calls this, how often, and what they do when it fails.
- Name the read and write paths separately; they rarely have the same bottleneck.
Follow-up
- What breaks first when traffic grows ten times?
- What would you drop to keep the system up under load?
Given a stream of real-time data, how would you design a system using …
Given a stream of real-time data, how would you design a system using Kafka and Airflow to orchestrate batch processing?
Approach
- Fix the scope first: who calls this, how often, and what they do when it fails.
- Name the read and write paths separately; they rarely have the same bottleneck.
- Name the failure you are designing for, then the recovery path.
Follow-up
- How does this behave when that dependency is down for an hour?
- What breaks first when traffic grows ten times?
How does your technical solution address the core business problem out…
How does your technical solution address the core business problem outlined in the prompt?
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Walk us through the architecture of the service you built for the codi…
Walk us through the architecture of the service you built for the coding exam. Why did you choose this specific stack?
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Make run creation safe when the client times out and retries
POST /v1/runs inserts a job_run row and dispatches a sandbox. The caller is a customer's CI pipeline with a 10-second timeout and three automatic retries. A run that takes 12 seconds to start returns after the client has already given up, so the retry starts a second sandbox and bills it. Redesign the endpoint: the key the client sends, its scope and storage, what happens when the same key arrives with a different body, what happens when a retry arrives while the first request is still in flight, and how long the key stays honoured.
Approach
- Name the defect before designing: the client's timeout is an unknown outcome, and the naive shape gives it no way to ask whether the earlier call landed, so its only choices are to duplicate the run or to drop it.
- Require an
Idempotency-Keyheader scoped to (tenant_id, key) rather than to the credential, so a rotated or replaced credential still replays. Store the key with a request fingerprint, SHA-256 over method, path and the canonicalised body, plus state, the stored response status and body, and the resulting run_id. - Make the claim a write rather than a read: insert the key row inside the same transaction as the run under
unique (tenant_id, idempotency_key). The loser of a concurrent race takes a unique violation and then reads the winner's row, which is what makes the duplicate case deterministic instead of timing-dependent. - Define three outcomes precisely. Same key, same fingerprint, completed: replay the stored response with a header marking it a replay. Same key, different fingerprint: 422 with an
idempotency_key_reusecode, never the stored response. Same key still in flight: 409 with Retry-After, because blocking the second caller holds a connection for the duration of someone else's request. - Close the dispatch gap with an outbox row committed alongside the run and a relay that enqueues it. Commit-then-publish loses the dispatch if the process dies in between; publish-then-commit hands a worker a run_id that does not exist yet.
- Set retention from the client, not from storage cost: the key must outlive the caller's worst-case total retry window, so publish a number such as 24 hours and state that a key reused after expiry legitimately creates a second run.
Worked solution 30 min
- Draw the timeline of the failing case: commit at t=11s, client abandons at t=10s, retry at t=10.1s, and mark where the second sandbox starts.
- Write the idempotency table DDL with the unique constraint and the fingerprint column, then the single transaction that inserts both rows.
- Enumerate the four arrival cases (first, replay, conflicting body, in-flight) and write the exact status, headers and body for each.
- Add the outbox row to the same transaction and describe the relay's at-least-once behaviour and why the worker must tolerate a duplicate dispatch.
- State the retention horizon and the sentence that goes in the public docs about key reuse after expiry.
Follow-up
- The SDK generates a fresh key on every attempt rather than reusing one. What in your response surface makes that visible instead of silently doubling the bill?
- How does this interact with job_run.parent_run_id: is an automatic retry of a failed run the same object as a client retry of the request?
A rare job-run overwrite that logging makes disappear
About one job run in fifty thousand records billable_seconds matching no observed sandbox lifetime, and a few rows show worker_id changing after finished_at was already set. It does not reproduce: debug logging around the terminal write made it vanish for two weeks before it returned. Runs last from 200 ms to 30 minutes, the lease is 60 seconds and is renewed while a run executes. Give an ordered checklist, the mechanism, and a fix that makes the illegal write impossible rather than merely rarer.
Approach
- Mine the evidence instead of chasing a repro: select rows where updated_at is later than finished_at, or where a terminal status was written twice, and join them to attempt history to recover both writer identities. The defect has already happened tens of times and the rows are the recording.
- State the signature before measuring it. If the cause is a lease that expired while the original worker was stalled, affected runs should cluster where the gap between the last renewal and the terminal write exceeds the lease, and should correlate with worker pause metrics rather than with workload shape.
- Read the disappearance honestly. Logging inside the window changed the timing and lowered the probability; it is evidence about how narrow the window is, not a fix. Reproduce by widening the window on purpose, shortening the lease and injecting a pause between sandbox exit and the terminal write, rather than by adding more instrumentation.
- Name the mechanism precisely: the lease expires during a stall such as a long garbage-collection pause or a brief partition, the run is re-dispatched, and the original worker then wakes and writes its terminal state over the new attempt's row. A lease alone cannot stop this, because the check and the write are separated by the stall.
- Fix by fencing the write itself: UPDATE job_run SET status = $2, finished_at = $3, billable_seconds = $4 WHERE run_id = $1 AND status = 'running' AND lease_token = $5, with zero rows affected interpreted as having been fenced rather than as success. The token lives on the row so the store arbitrates, not the worker's memory.
- Keep the state machine honest: a retry inserts a new row pointing at parent_run_id rather than resetting the old one, and a run whose worker vanished terminates as lost with billable_seconds null, because recording failure asserts an outcome nobody observed and then bills and retries on that assertion.
Follow-up
- The supervisor also emits a usage event on completion. What does the fenced worker do about the event it already emitted, and how does metering absorb it?
- Lease renewal is itself a network call. What happens when a renewal times out, and how does the worker decide whether it still holds the lease?
- Why is lengthening the lease past the longest legitimate run the wrong lever, and what breaks if you do it anyway?
For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.
Prepare, practise & reflect
One practical outcome each day. Spend longer where you need it.
0 / 7 done01Screening answers and your track
- Choose your track, Data Infrastructure or Full Stack, and list evidence from your own work for the stack that track reportedly uses.
- Write three foundational questions for this role (your background, why this track, one project end to end), record your answers, play them back, and cut any opening that is not the answer.
- Fix the figures for the one project you will cite throughout the loop: data size, users, team size and your part.
Deliverable: Three recorded screening answers and a one-page project sheet with fixed figures.
Practice prompt ↗Practice prompt ↗Practice prompt ↗Worked solution ↗02Take-home rehearsal: build the service
- Build a small service from a reported example scenario, such as one that aggregates two public APIs, caches the results and exposes a REST interface, in the language you plan to use.
- Add unit and integration tests, a Dockerfile and a README with run instructions, and handle upstream failures and timeouts explicitly.
- Keep a decisions log: every choice, the alternative you rejected and the reason.
Deliverable: A runnable, tested, containerized service and its decisions log.
Practice prompt ↗Practice prompt ↗Practice prompt ↗03Failure handling and the written questions
- Work this guide's design exercise on making run creation safe when the client retries, then apply its idempotency-key and outbox ideas to your Day 2 service where they fit.
- Answer two technical and two behavioral questions as if they were the take-home's written section, in full sentences, and review them as carefully as code.
- Reread every line of your service, mark any line you could not explain without looking something up, and rewrite those lines.
Deliverable: Written answers, plus a service with no line you cannot explain.
Practice prompt ↗Practice prompt ↗04Panel deck and defense
- Build a deck that opens with the business problem, maps components to requirements, then covers the architecture, stack choice and trade-offs.
- Prepare answers for your own service to the reported panel questions: why this stack, how it fits the business problem, the first bottleneck at ten times the traffic, and SLA metrics with alerts.
- Have a friend point at random lines of your code and ask you to explain them, optimise them or change them for a new constraint.
Deliverable: A recorded run-through of the deck, plus the questions you stumbled on with improved answers.
Practice prompt ↗Practice prompt ↗Worked solution ↗05Design and data-infrastructure questions
- Sketch the reported design questions: the multimodal ingestion pipeline for LLM training, the fault-tolerant academic paper crawler, Kafka plus Airflow batch orchestration, the Python rate-limiting middleware, and the distributed key-value store with your choice of consensus algorithm.
- For each design, state the delivery guarantee, the retry path and the first bottleneck.
- Work the SQL exercise on the deduplicated hourly rollup, then prepare short explanations of Parquet versus Delta Lake, Spark or Ray memory bottlenecks, and a schema for dataset versioning and reproducibility.
Deliverable: One page per design question naming its guarantees, failure paths and first bottleneck.
Practice prompt ↗Practice prompt ↗06Coding and debugging
- Work the coding exercise on sealing an hour under late data, then attempt the webhook signature parser drill without reading its approach.
- Do the job-run overwrite debugging drill as an ordered checklist and write the fenced update that fixes it.
- Warm up on fundamentals from the question bank for this role, such as the first unique character in a string, the lowest common ancestor in a binary tree and a small OOP class design, stating complexity aloud.
Deliverable: Solutions with their complexity stated, and a written debugging checklist.
Practice prompt ↗Practice prompt ↗07Culture fit stories and a full mock
- Write one story for each reported behavioral theme: a bottleneck resolved across teams, technical debt against a prototype, a researcher with different priorities, and ambiguity with minimal documentation.
- Use the code-review drill on isolation levels to practise telling a story about resolving a technical disagreement with evidence.
- Run a full mock: present the deck, answer line-level code questions, then answer behavioral questions, and check that your project figures match across all of them.
Deliverable: Four behavioral stories and notes from a full mock panel.
Practice prompt ↗Practice prompt ↗Worked solution ↗Expand any day for tasks and deliverables. Your progress is saved on this device.
The behavioral questions reported for this role cover taking initiative across team boundaries, balancing technical debt against delivery speed, working with researchers whose priorities differ from yours, and handling ambiguity at the start of a project. For each, pick a story where you made the call. Describe the situation briefly, then spend most of the answer on your reasoning and what changed as a result. Keep project facts identical to what you said on the panel.
Tell me about a time when you had to collaborate with a cross-function…
Tell me about a time when you had to collaborate with a cross-functional partner (such as an AI researcher) who had different priorities than you.
Approach
- Give the blast radius: what could have broken, and what you measured.
- Name the disagreement and how you resolved it with evidence.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
How do you handle ambiguity when starting a project from scratch with …
How do you handle ambiguity when starting a project from scratch with minimal documentation?
Approach
- Pick a story where you made the decision, not one where you watched it.
- State the situation in two sentences and spend the rest on the reasoning.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Resolve a review disagreement over a quota check
A colleague's pull request enforces a per-tenant quota by selecting the current count and then inserting when it is under the limit. You flag it as a race. They reply that the transaction already runs at repeatable read, so the snapshot makes it safe, and the tests pass. Walk through taking that disagreement to a resolution: what you write in the review, what you demonstrate rather than assert, which fix you propose and why, and what you do if they still disagree after all of it.
Approach
- Answer the claim precisely instead of restating your objection, because they have made a specific technical argument. In PostgreSQL, repeatable read is snapshot isolation; this is write skew, which snapshot isolation permits by design. Both transactions read a count that is stable within their own snapshot, insert disjoint rows that the other cannot see, and both commit, so the limit is exceeded by exactly the concurrency.
- Demonstrate rather than cite. Two psql sessions, both BEGIN ISOLATION LEVEL REPEATABLE READ, both select the count, both insert, both commit: it succeeds. Repeat at SERIALIZABLE and the second commit fails with serialization_failure, SQLSTATE 40001. That takes two minutes, ends the argument without anyone conceding a position, and leaves an artefact for the next reviewer.
- Offer the options with their costs rather than a verdict. Serialisable plus a retry loop on 40001 is correct but obliges every caller to retry and degrades under contention. An increment-and-compare on a counter row — update tenant_quota set used = used + 1 where tenant_id = $1 and used < limit returning used — is safe even at read committed, because a blocked updater re-evaluates the WHERE clause against the row version it finally locks, and zero rows returned means full. A unique or exclusion constraint that makes the surplus write fail is the third.
- Name the plausible non-fix explicitly, since it is what usually gets merged instead: folding the count into the insert as insert ... select ... where (select count(*) ...) < limit is still racy under read committed, because the subquery cannot see the other transaction's uncommitted rows. It looks atomic and is not.
- Say what you do if they still disagree: escalate the decision rather than the disagreement. Attach the reproduction, hand it to the service owner or a third reviewer, and state that you will not block the merge if the owner accepts the risk knowingly — and that you want that acceptance written down.
- Close with the general lesson worth leaving in the review thread: a passing suite is weak evidence for a concurrency claim because it runs one request at a time. Ask for a test that runs two.
Follow-up
- Write the counter-row version. Does your answer change if the quota counts child rows rather than a column?
- Under serialisable, who performs the retry, and what does the API client see if the retry also fails?
- This is the third disagreement with the same reviewer this month. What changes in how you review?
- 01
Describe a time when you identified an infrastructure bottleneck and took the initiative to resolve it across multiple team boundaries.
- 02
How do you prioritize technical debt against the need to ship a prototype quickly?
- 03
Tell me about a time when you had to collaborate with a cross-functional partner (such as an AI researcher) who had different priorities than you.
- 04
How do you handle ambiguity when starting a project from scratch with minimal documentation?
- 05
A colleague insists a select-then-insert quota check is safe at repeatable read. How do you take that disagreement to a resolution?
Is this an official Thinking Machines Lab interview guide?
No. It is PracHub's own research and practice material for the Software Engineer role at Thinking Machines Lab. Rounds and questions reflect what candidates have reported, not a process Thinking Machines Lab has published, and they change over time. Confirm the current format and scope with your recruiter.
PracHub interview research ↗How difficult is the technical interview process?
Candidates describe the coding work as practical rather than competitive programming: class-level data structures and clean backend design. The hard part is that the panel has you present and defend your own take-home, so your communication and architectural reasoning are tested on code the interviewers have already read. Prepare for depth on your own submission rather than for puzzle difficulty.
PracHub interview research ↗Can I complete the coding exam in any language?
Candidates report that the take-home can be written in any programming language. The role description mentions Python and Rust on the backend, but what matters more is that you can defend every line on the panel. Use the language you write fastest and most idiomatically, unless you are equally fluent in one of those two.
PracHub interview research ↗What happens in the panel interview?
Candidates report presenting their take-home solution from their own slides, defending their architectural choices and answering deep-dive questions from the engineering team, including questions about specific lines of code. The reported questions cover why you chose your stack, how the solution addresses the business problem, the bottlenecks at ten times the traffic, and the monitoring and alerting you would use to meet SLAs.
PracHub interview research ↗How long does the process take?
Candidates report four rounds over roughly 3-5 weeks, from the initial screening to the culture fit interview. Timing varies, so ask the recruiter for the take-home deadline and the panel scheduling window as soon as you are invited.
PracHub interview research ↗Is the coding assessment monitored?
Candidates report active anti-cheating mechanisms on the coding assessment. Write your solution from scratch, understand every line, and expect to explain or modify specific lines during the panel.
PracHub Software Engineer practice ↗Which technical topics should I prepare?
Reported topics include Python, distributed systems, LLM infrastructure, Rust and scalable fault-tolerant infrastructure. For the Data Infrastructure track, prepare Spark or Ray memory tuning, Parquet versus Delta Lake, deduplication and quality checks, crawlers, and orchestration with Kafka or Airflow. For the Full Stack track, add API design in Python or Rust and frontend work in React with TypeScript. Confirm your track with the recruiter.
PracHub Software Engineer practice ↗Sources & methodology 3 sources ↗
No official company page is cited. Rounds and questions come from candidate reports and PracHub editorial material; each source shows the date it was read.
- 01PracHub interview research ↗
PracHub editorial research into this company and role, maintained with this guide. Candidate-reported, not an employer publication.
PracHub page · Accessed 2026-09-24 - 02PracHub Software Engineer practice ↗
Cross-company practice questions for this role.
PracHub page · Accessed 2026-09-24 - 03PracHub interview preparation framework ↗
The framework the preparation plan follows.
PracHub page · Accessed 2026-09-24
Explore related engineering environments and compare the preparation focus.
Nuro Software Engineer Interview Guide
Debugging practice for Nuro software engineer prep built on a reproduction you trust, narrowing to the smallest failing input, and changing one variable per run so the result stays attributable.
Spacex Software Engineer Interview Guide
Data structures and algorithms practice for Spacex candidates, where stating a bound means stating its precondition: constant-time hash lookup is an average case over keys that spread.
Zoox Software Engineer Interview Guide
A study plan for Zoox software engineer prep built on spaced review: commit to a full answer before opening any solution, and repeat every missed exercise three days later.
Braze Software Engineer Interview Guide
Concurrency exercises for Braze candidates: check-then-act races, deadlock from inconsistent lock ordering, and why two individually atomic operations do not compose into an atomic update.