Harvey Software Engineer Interview Prep Guide

Focus most on spreadsheet-style stateful coding—formula parsing, dependency graphs, cycle detection, and exact text highlighting—because your Coding & Algorithms self-rating is 3/5 and there are no solved coding signals yet. Because no category is rated above 3/5, the review-only lane stays narrow: basic in-memory file-system operations and hashing-based file identity. For Harvey AI, this plan highlights legal-document RAG, secure multitenant SaaS handling, and LLM evaluation/guardrails rather than generic product design drills. With 20 days until the interview, budget roughly an hour for this cheatsheet pass, then spend most practice time implementing the emphasized coding pieces end-to-end.

Technical Screen — 33 min

Coding & Algorithms

Focus area — High-yield Harvey-style stateful coding area; coding is 3/5 and there are no solved coding signals yet.

What's being tested

These problems test stateful data structure design for an in-memory spreadsheet: parsing formulas, resolving cell references, maintaining a dependency graph, and updating values incrementally. Interviewers are looking for clean APIs, correct invalidation order, and robust handling of cycles and stale cached values.

Patterns & templates

• Cell model: store raw input, parsed expr, cached value, dependencies, and dependents; separates user state from computed state.

• Expression parsing: implement recursive descent or token scan for +, -, *, /, parentheses, numbers, and cell refs; define precedence explicitly.

• Dependency graph: on set(cell, formula), remove old edges, add new cell -> referencedCell edges, then update reverse dependents for propagation.

• Cycle detection: run DFS with visiting/visited states before committing formula; reject A1 = B1 + 1, B1 = A1 + 1.

• Incremental recomputation: after a leaf update, traverse reverse dependencies with topological DFS/BFS; recompute only affected cells in dependency-safe order.

• Lazy evaluation alternative: mark dependents dirty and compute on get(cell) via DFS memoization; simpler for sparse reads, worse for repeated hot reads.

• Complexities: set is O(F + A) where F is formula size and A affected cells; get is O(1) eager or O(D) lazy.

Common pitfalls

Pitfall: Forgetting to delete old dependency edges means cells keep depending on references from previous formulas.

Pitfall: Detecting cycles only during get can leave the spreadsheet in an invalid committed state; validate before committing updates.

Pitfall: Recomputing every cell after each update is correct but usually fails the intended incremental-evaluation signal.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

Focus area — Formula engines often fail at parsing edge cases; emphasize tokenizer, precedence, and validation under your 20-day timeline.

What's being tested

These problems test expression parsing plus stateful dependency management: parse formulas, resolve cell references, maintain a dependency graph, and recompute affected cells correctly. Interviewers are probing whether you can turn a spreadsheet-like API into clean data structures with cycle detection and incremental evaluation.

Patterns & templates

• Recursive descent parsing for formulas like =A1+B2*3; implement parseExpr, parseTerm, parseFactor to enforce precedence.

• Tokenization separates numbers, operators, parentheses, and cell refs; keep it O(L) per formula length and reject malformed tokens early.

• Dependency graph maps cell -> dependencies and reverse cell -> dependents; updates need both directions for efficient invalidation.

• DFS cycle detection with visiting/visited states; run before committing a formula to catch A1 -> B1 -> A1.

• Incremental recomputation uses reverse graph traversal from changed cells; topologically evaluate dirty dependents in O(V+E) over impacted subgraph.

• Memoized evaluation via eval(cell) avoids repeated recomputation inside one update; invalidate cache entries reachable from changed cells.

• Stateful API design separates setValue(cell, n), setFormula(cell, expr), and getValue(cell); define behavior for missing cells, self-references, and errors.

Common pitfalls

Pitfall: Only evaluating formulas on getValue without invalidation can return stale values after upstream cells change.

Pitfall: Detecting cycles only during evaluation often leaves the spreadsheet in a partially corrupted state; validate before committing updates.

Pitfall: Forgetting to remove old dependencies when replacing a formula causes phantom updates and incorrect graph edges.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

Focus area — Central to spreadsheet updates; emphasize forward/reverse edges and incremental recomputation because no graph-solving history is visible.

What's being tested

These prompts test dependency graph maintenance for spreadsheet formulas: parsing expressions, tracking cell references, detecting cycles, and incrementally recomputing affected cells. A strong solution separates cell state, formula evaluation, and graph propagation instead of recomputing the whole sheet after every update.

Patterns & templates

• Bidirectional graph: maintain deps[cell] -> referenced_cells and revDeps[cell] -> dependents; update both when setFormula(cell, expr) changes references.

• Expression parsing: tokenize formulas like =A1+B2*3; use recursive descent or shunting-yard for precedence, parentheses, literals, and cell references.

• DFS cycle detection: before committing a formula, run hasPath(ref, cell) or color-state DFS; reject formulas creating back edges.

• Topological recomputation: after setValue or setFormula, traverse revDeps from the changed cell and recompute in dependency order.

• Lazy vs eager evaluation: eager updates make get(cell) O(1) but updates cost O(affected); lazy evaluation shifts work to get.

• Transactional updates: parse and validate first, then mutate deps, revDeps, and stored formula/value; rollback is simpler if validation happens before commit.

• Complexity target: update should be O(size of affected subgraph + expression sizes), not O(total cells) unless constraints are tiny.

Common pitfalls

Pitfall: Only storing forward dependencies makes propagation hard; you need reverse edges to find cells invalidated by an update.

Pitfall: Detecting cycles only during evaluation can leave the spreadsheet in a corrupted state; validate before committing the new formula.

Pitfall: Forgetting to remove old formula edges causes stale dependents and incorrect recomputations after overwriting a formula.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

Focus area — Cycle handling is a common follow-up; emphasize clear DFS state logic and rollback behavior for invalid updates.

What's being tested

These problems test cycle detection in a dynamic dependency graph (cells referencing cells) plus related techniques: expression parsing, incremental evaluation, and state management for updates. Interviewers probe correctness (no false-negatives/positives for cycles), performance (incremental vs full recompute), and clear API/state reasoning.

Patterns & templates

• DFS with coloring (white/gray/black) for online cycle detection — O(V+E) time, stack traces give cycle path for error messages.

• Kahn's algorithm / topological sort to compute evaluation order for acyclic graphs; detect leftover nodes as cycles.

• Adjacency list graph representation: map cell -> set(dependents) and reverse map cell -> set(dependees) for update propagation.

• Incremental evaluation via change propagation: BFS/queue from changed node through dependents, memoize values to avoid re-eval.

• Expression parsing + AST: tokenize formulas, build AST with node types (literal, ref, op), evaluate AST using resolved references. Use parseExpression then evalAST.

• Cycle prevention strategies: pre-insert tentative edge, run DFS/Kahn to validate before committing; rollback on failure.

• Complexity framing: initial parse/build O(F) per formula; full sheet recompute O(V+E); incremental updates average proportional to affected subgraph size.

Common pitfalls

Pitfall: Failing to update reverse dependency map causes stale propagation or missed re-evaluations.

Pitfall: Detecting cycles only at evaluation time (not on update) lets invalid state be visible; validate on write.

Pitfall: Using recursion without recursion-depth safeguards crashes on deep chains — prefer iterative DFS or explicit stack.

Practice these

The practice cards below cover canonical variants — solve all of them and time yourself.

Practice questions

Focus area — Harvey works with legal text; emphasize exact matching, overlaps, and safe highlighting despite no explicit subtopic selection.

What's being tested

Tests exact substring matching across one or more source strings, then converting raw match positions into correctly tagged output. Interviewers are probing for clean interval generation, overlap/adjacency merging, deterministic handling of multiple sources, and bug-free string reconstruction.

Patterns & templates

• Brute-force matching with `find_all(text, pattern)` — loop using `str.find(pattern, start)`; advance by 1 to allow overlapping matches.

• Interval collection — convert every match to half-open [start, end) intervals; half-open bounds simplify merging and output slicing.

• Sort-and-merge intervals — sort by (start, end), then merge when next.start <= cur.end; use < if adjacent matches should stay separate.

• Multi-pattern search — for many source strings, use Aho-Corasick for `O(n + total_matches + total_pattern_length)` instead of nested scans.

• Output reconstruction — append unmatched text, opening tag, matched slice, closing tag; never mutate the string while iterating indices.

• Counting occurrences — keep a map like `source_id -> count` or `pattern -> count`; define whether overlapping occurrences count before coding.

• Edge-case template — handle empty sources, duplicate patterns, empty pattern strings, case sensitivity, punctuation, Unicode, and repeated adjacent matches.

Common pitfalls

Pitfall: Advancing `start += len(pattern)` misses overlapping matches like finding "aa" twice in "aaa".

Pitfall: Inserting tags directly into the original string shifts later indices and corrupts positions; build a new output buffer instead.

Pitfall: Forgetting to merge adjacent intervals can produce `<b>abc</b><b>def</b>` when the expected output is `<b>abcdef</b>`.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

Core technical-screen pattern; your coding rating is 3/5, but default concept confidence keeps this at targeted review.

What's being tested

This tests hierarchical data modeling with a trie/tree of directories and files, plus robust path parsing and name-collision handling. Interviewers are looking for clean object design, edge-case discipline, and ability to reason about `mkdir`, `ls`, `addContentToFile`, `readContentFromFile`, capacity limits, and duplicate names.

Patterns & templates

• Trie/tree node model — each `Node` stores `children: Map[str, Node]`, `isFile`, optional `content`, metadata, and size; operations are O(path components).

• Path normalization — split on /, ignore empty components, handle root /; avoid bugs from trailing slashes like /a/b/.

• Directory traversal helper — implement `getNode(path, create=False)` to centralize validation, auto-creation, and missing-path behavior.

• Lexicographic listing — `ls(path)` returns [filename] for files or sorted child names for directories; sorting costs O(k log k).

• Content accounting — for constrained systems, track total bytes and per-file size deltas before appending; reject writes that exceed capacity.

• Duplicate naming policy — OS-style collisions often require generating `name(1)`, `name(2)`; maintain counters or scan siblings carefully.

• Complexity explanation — use d = path depth, k = directory children, c = content length; traversal O(d), append/read affected by c.

Common pitfalls

Pitfall: Treating files and directories as separate maps makes traversal and collision handling harder than a unified `Node` abstraction.

Pitfall: Forgetting that `ls("/a/file.txt")` should usually return only ["file.txt"], not file content or children.

Pitfall: Capacity checks must account for append delta, not total rewritten content unless the API replaces file contents.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

Technical Screen — 26 min

System Design

Focus area — Company-specific Harvey angle: legal document retrieval, citations, and hallucination mitigation deserve emphasis with your system design engagement.

What's being tested

Interviewers are probing whether you can design a retrieval-augmented generation system as a production backend, not just describe “vector search plus an LLM.” For Harvey, the hard parts are document-scale ingestion, permission-safe retrieval, low-latency serving, citation fidelity, and graceful behavior when model outputs are imperfect. A strong Software Engineer answer should decompose the system into APIs, storage, indexing, retrieval, orchestration, observability, and failure handling. Expect follow-ups on scale, consistency, access control, latency, and how you would debug bad answers.

Core knowledge

• RAG architecture usually has two paths: an offline indexing pipeline and an online query-serving path. Indexing parses documents, chunks text, computes embeddings, stores metadata, and builds search indexes; serving embeds the query, retrieves candidates, constructs context, calls the LLM, and returns citations.

• Chunking strategy strongly affects retrieval quality and system behavior. Typical chunks are 300–1,000 tokens with 10–20% overlap; smaller chunks improve pinpoint citations but lose context, while larger chunks reduce fragmentation but waste prompt budget and can bury relevant facts.

• Embedding retrieval maps chunks and queries into vectors and ranks by cosine similarity:

  $$\cos(\theta)=\frac{q \cdot d}{\|q\|\|d\|}$$

  Use approximate nearest neighbor indexes such as HNSW in pgvector, Pinecone, Weaviate, Milvus, or OpenSearch k-NN when exact scan becomes too slow.

• Hybrid search combines dense vector retrieval with lexical search such as BM25. This matters for legal and enterprise documents because exact terms, party names, clause numbers, citations, and defined terms may be missed by pure embeddings.

• Metadata filtering is not optional. Store tenant ID, matter ID, document ID, source, version, page ranges, timestamps, and ACL information alongside each chunk. For confidential legal data, enforce permission filters before or during retrieval, not after generation.

• Reranking improves top-k quality by applying a more expensive model or scoring function to a smaller candidate set. A common pattern is to retrieve top 100 via vector/BM25, rerank to top 10–20, then pack only the best chunks into the prompt.

• Context assembly is a backend algorithm. Deduplicate overlapping chunks, preserve source ordering where helpful, avoid including contradictory stale versions, and fit within the model’s context window. Use citation IDs like [doc_id:page:chunk_id] so the final answer can be traced back.

• Latency budgeting should be explicit. A typical online path may include query embedding, vector search, lexical search, reranking, prompt construction, LLM first-token latency, and streaming. Optimize for p95/p99, not average latency; slow rerankers or cold LLM calls dominate tail latency.

• Caching can help at multiple layers: parsed documents, embeddings, retrieval results for repeated queries, and LLM responses for deterministic prompts. Be careful caching user-visible answers when permissions, document versions, or prompt templates change.

• Versioning and consistency are central in document systems. If a user uploads a new contract version, decide whether queries require read-after-write indexing or can tolerate eventual consistency. Use document version IDs so citations never point to replaced content accidentally.

• Observability should expose retrieval and generation internals: retrieved chunk IDs, similarity scores, applied filters, prompt token counts, model latency, error rates, empty retrieval rates, and user feedback. Without retrieval traces, “the answer was bad” is nearly impossible to debug.

• Failure modes need designed responses. If retrieval returns no evidence, the system should say it cannot find support rather than hallucinate. If an LLM call times out, return a retryable error, partial streamed response, or fallback result depending on product requirements and correctness risk.

Worked example

For Design a Retrieval-Augmented Generation System, start by clarifying the corpus and serving requirements: “Are we indexing user-uploaded legal documents, public law, or both? What is the expected document count, query QPS, latency target, and permission model?” Then declare assumptions, for example: multi-tenant document Q&A, millions of chunks, strict tenant isolation, citations required, and a p95 target under a few seconds excluding long streamed generation.

Organize the answer around four pillars: ingestion/indexing, retrieval/ranking, generation/citation assembly, and production reliability. For ingestion, describe upload APIs, document parsing, chunking, embedding jobs, metadata storage in Postgres, and vector indexing in something like pgvector, OpenSearch, or a managed vector database. For serving, explain query embedding, ACL-filtered hybrid retrieval, optional reranking, prompt construction with chunk IDs, LLM call, and streamed response.

A specific tradeoff to flag is pre-filtering versus post-filtering for permissions. Pre-filtering by tenant and ACL inside the search query is safer and avoids leaking unauthorized chunks into prompts, but it can reduce recall or complicate index design; post-filtering is simpler but dangerous because unauthorized text may reach the model. Close by saying that if you had more time, you would go deeper on index freshness, backfills, evaluation harnesses using golden Q&A sets, and operational dashboards for retrieval misses and citation failures.

A second angle

For Debug Poor Answer Quality in a RAG System, the same architecture becomes a diagnostic flow rather than a greenfield design. First separate retrieval failure from generation failure: inspect whether the correct source chunks appeared in the top-k results before blaming the LLM. If the right chunks are absent, investigate parsing quality, chunk boundaries, embedding model changes, metadata filters, lexical search gaps, or stale indexes. If the right chunks are present but the answer is wrong, inspect prompt construction, context ordering, token truncation, citation formatting, and model timeout or streaming behavior. The key difference is that the best answer is evidence-driven: show logs, retrieved chunk IDs, scores, and prompt snapshots rather than making broad claims about “improving the model.”

Common pitfalls

Pitfall: Treating RAG as “put documents in a vector DB and ask GPT-4.”

That answer misses the engineering surface area: parsing, versioning, permissions, indexing freshness, reranking, prompt packing, latency, and observability. A better answer names each component and explains the contract between them.

Pitfall: Ignoring access control until the end.

In Harvey-like systems, confidential documents are core data, not incidental data. Say explicitly that ACL and tenant filters must be enforced before retrieved text enters the prompt, and that logs, caches, and traces must avoid leaking sensitive content across tenants.

Pitfall: Over-indexing on ML quality and under-explaining distributed-system behavior.

It is tempting to discuss embedding model choices or hallucination theory at length. For a Software Engineer interview, spend more time on APIs, storage schemas, async indexing, retries, idempotency, backpressure, tail latency, and how to debug production failures.

Connections

Interviewers may pivot from this topic into search infrastructure, distributed job processing, multi-tenant authorization, LLM serving APIs, or observability design. They may also ask for an evaluation layer, but keep the answer engineering-focused: golden datasets, regression tests, trace inspection, and measurable retrieval/citation failure rates.

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the original RAG paper; useful for understanding the retrieval-plus-generation framing.

• The Illustrated Word2vec — helpful intuition for embeddings and vector similarity without going deep into model architecture.

• HNSW: Efficient and Robust Approximate Nearest Neighbor Search — the core ANN algorithm behind many vector search systems.

Practice questions

Focus area — Harvey handles sensitive customer/legal data; emphasize tenant isolation, authorization, audit logs, and retention.

What's being tested

Interviewers are probing whether you can design a secure multitenant SaaS system where many enterprise customers share infrastructure without sharing data, permissions, or operational blast radius. For Harvey, this matters because legal workflows involve highly sensitive documents, privileged communications, matter-level permissions, and enterprise audit requirements. A strong Software Engineer answer balances tenant isolation, authorization correctness, data protection, operability, and practical tradeoffs like shared versus dedicated infrastructure. The interviewer is not looking for hand-wavy “use encryption” answers; they want to see where isolation is enforced, how failures are contained, and how you prove the system is safe under real production paths.

Core knowledge

• Multitenancy models usually fall into three patterns: shared database/shared schema, shared database/separate schema, and separate database per tenant. Shared schema is cheapest and easiest to operate, but requires flawless tenant scoping; separate databases improve isolation and noisy-neighbor control but increase migrations, connection management, and operational complexity.

• Tenant isolation must be enforced at multiple layers, not just in application code. Typical layers include API authentication, authorization middleware, database predicates like tenant_id = ?, Postgres row-level security policies, object-store path scoping in S3, search-index filters, cache key prefixes, queue routing, and observability access controls.

• Authentication answers “who are you?” while authorization answers “what can you access?” Enterprise SaaS commonly supports SAML or OIDC single sign-on, maps identity-provider groups into application roles, and uses short-lived session tokens or JWTs with claims such as sub, org_id, roles, and token expiry.

• RBAC works well for coarse permissions like admin, member, and viewer; ABAC is better for legal workflows where access may depend on attributes like tenant_id, matter_id, document_classification, jurisdiction, or ethical_wall_group. Many systems combine both: roles grant capabilities, attributes constrain resources.

• Authorization checks should happen close to every resource access. A common pattern is authorize(actor, action, resource) backed by policy code or a service like Open Policy Agent. Avoid checking only at the route level if downstream jobs, batch exports, search, or document-preview endpoints can fetch data independently.

• Database design should make unsafe queries hard to write. In shared tables, include tenant_id in every tenant-owned table, put composite indexes like (tenant_id, matter_id, created_at), and consider UNIQUE (tenant_id, external_id) rather than globally unique business identifiers. Postgres RLS can enforce tenant_id = current_setting('app.tenant_id').

• Object storage isolation needs the same rigor as relational data. Store files under namespaced keys such as tenant/{tenant_id}/matter/{matter_id}/doc/{doc_id}, but do not rely on naming alone; use service-side authorization, scoped signed URLs with short TTLs, bucket policies where possible, and metadata validation before serving bytes.

• Search and vector indexes are common leakage points. If using OpenSearch, Elasticsearch, pgvector, or a vector database, every query must include a tenant and permission filter before returning chunks. For sensitive legal documents, retrieval should filter by tenant_id, matter_id, and user-accessible document IDs, not merely post-filter after top-k retrieval.

• Encryption includes data in transit via TLS, data at rest via storage encryption, and sometimes per-tenant keys through AWS KMS, GCP KMS, or HashiCorp Vault. Per-tenant envelope encryption improves crypto isolation and supports tenant-specific key rotation or deletion, but adds latency, key-management paths, and failure modes.

• Audit logging should capture security-relevant events: login, SSO group sync, permission changes, document upload/download, search, export, admin impersonation, and failed authorization. Use append-only storage semantics, include actor_id, tenant_id, resource_id, action, decision, ip, user_agent, and timestamp, and avoid logging document contents or secrets.

• Noisy-neighbor controls protect availability across tenants. Apply per-tenant rate limits, quota checks, worker-pool isolation, queue partitioning, and query timeouts. For example, a large document import for one tenant should not exhaust all background workers or saturate shared Postgres IOPS for other tenants.

• Defense in depth assumes one layer will eventually fail. Use secure defaults, centralized middleware, typed tenant context, automated tests for cross-tenant access, static checks for unscoped queries where possible, canary tenants, alerting on authorization denials, and production guardrails like “break glass” admin access with mandatory audit records.

Worked example

Design a secure multitenant document management system for enterprise legal teams

A strong candidate starts by clarifying the tenancy model, sensitivity level, and access patterns: “Are tenants law firms or companies? Do users belong to multiple tenants? Are documents scoped to matters? Do we need enterprise SSO, audit logs, and data residency?” Then they declare assumptions: use shared application services, shared Postgres for metadata, object storage like S3 for files, and strict logical isolation by tenant_id and matter_id.

The answer can be organized around four pillars. First, identity and access: OIDC/SAML login, user-to-tenant membership, RBAC for admin/member/viewer, and ABAC constraints for matter-level access. Second, data isolation: every metadata table has tenant_id, database queries are scoped through a tenant context, Postgres RLS is enabled for critical tables, and object keys are namespaced with signed URLs generated only after authorization. Third, operational security: per-tenant rate limits, audit logs for document access and permission changes, encrypted storage, and key management through KMS. Fourth, background processing: document ingestion jobs carry tenant context explicitly, validate permissions at enqueue time and processing time, and write derived artifacts like OCR text or embeddings into tenant-filtered stores.

One tradeoff to flag is shared database versus database-per-tenant. For most tenants, shared Postgres with RLS and strong indexing is simpler and cost-effective; for very large or regulated tenants, the design can support dedicated databases or dedicated storage buckets as an enterprise isolation tier. The close should show maturity: “If I had more time, I’d detail migration strategy, cross-tenant security tests, disaster recovery, and how we verify that search/vector retrieval never leaks chunks across matters.”

A second angle

Design an audit logging and access-control layer for a multitenant SaaS platform

This framing shifts from storage architecture to correctness and traceability of security decisions. The key move is to define a centralized authorization API such as authorize(actor, action, resource) and require every service path—REST endpoints, background jobs, exports, search, and admin tools—to call it before accessing tenant resources. Audit events should be emitted for both successful and denied sensitive actions, but the log pipeline must avoid leaking document contents, prompts, access tokens, or privileged metadata. Compared with the document-management design, the harder tradeoff is consistency: synchronous audit writes improve compliance confidence but add latency and failure coupling, while asynchronous writes improve availability but need durable queues and retry semantics. A strong answer proposes a hybrid: block on authorization, emit audit events to a durable append-only stream, and use monitoring to detect missing or delayed audit records.

Common pitfalls

Pitfall: Treating tenant_id as a UI filter instead of a security boundary.

A tempting answer is “we’ll add WHERE tenant_id = tenant.id to queries.” That is necessary but not sufficient; strong candidates explain how tenant context is propagated, how unscoped queries are prevented, how RLS or policy checks backstop application mistakes, and how search, caches, object storage, and background jobs are also scoped.

Pitfall: Over-indexing on encryption while ignoring authorization.

“Encrypt everything with KMS” sounds secure but does not prevent an authenticated user from reading the wrong matter if authorization is wrong. Encryption protects against storage compromise and supports key lifecycle controls; access-control correctness protects against the most likely application-layer leaks.

Pitfall: Designing only the happy-path API.

Cross-tenant leaks often happen through secondary paths: CSV export, admin impersonation, document preview, search snippets, OCR jobs, embeddings, cached responses, webhook retries, or support tooling. A better answer explicitly walks through at least one asynchronous or derived-data flow and shows where tenant and permission checks are enforced.

Connections

Interviewers may pivot from this topic into distributed authorization, database schema design, secure background job processing, search/vector retrieval isolation, or observability and incident response. They may also ask how the architecture changes for enterprise requirements like data residency, dedicated infrastructure, customer-managed keys, or legal hold.

Further reading

• OWASP Application Security Verification Standard — practical checklist for authentication, access control, session management, and data protection.

• NIST SP 800-53 — broad security-control catalog useful for understanding enterprise SaaS expectations.

• AWS SaaS Tenant Isolation Strategies — concrete patterns for isolation across compute, storage, identity, and networking.

Practice questions

Focus area — Added for Harvey AI: practice measuring answer quality, groundedness, refusal behavior, and guardrail failures in LLM features.

What's being tested

Interviewers are probing your ability to design reliable, observable, and enforceable runtime protections for a production LLM-driven system: measurable evaluation pipelines, automated safety guardrails, deployment/rollout patterns, and operational controls that an engineer can implement. Expect to justify tradeoffs between latency, cost, false-positive filtering, and developer velocity while showing how you instrument and iterate on model quality in production.

Core knowledge

• Evaluation pipeline: separate offline (batched annotations, synthetic tests) and online (production sampling) evaluation flows; store prompt/output pairs, metadata, and hashed user IDs for repro and privacy.

• Factuality / hallucination metrics: define hallucination rate = false factual assertions / total factual assertions; estimate with binomial margin: $n = p(1-p)*(1.96/m)^2$ for 95% confidence.

• Quality & safety SLIs: common SLIs include p99 latency, request error rate, throughput, hallucination rate, and toxic-content-rate; bind these to observable SLOs with alert thresholds and runbooks.

• Annotation tooling: use standard schemas, inter-rater agreement (e.g., Cohen's kappa) to validate label quality; reduce label drift with periodic rater calibration.

• Guardrail patterns: input validation via JSON Schema, allowlist/blocklist checks, rule-based sanitizers, staged classifiers (fast filter → LLM), and output filters (regex, HTML escaping, profanity lists).

• Fallback & routing: implement circuit breakers, graded fallbacks (rule response, template reply, smaller model), and traffic splits/feature flags for canarying new models or prompts.

• Adversarial testing: automated prompt fuzzing, property-based testing (e.g., Hypothesis-style), and red-team scenarios; log adversarial prompts to a replayable corpus.

• Observability & logging: log structured artifacts (prompt, response, model version, embedding id, latency, trace id) to a long-tail store with TTL and sampling; enable deterministic replay for debugging.

• Confidence & verification: use provenance (citations), retrieval-augmented generation, or lightweight verifier classifiers for high-stakes responses; avoid trusting raw token probabilities alone.

• Privacy & compliance: redact PII at ingress, hash identifiers for metrics, and avoid logging raw sensitive text unless explicitly allowed in policy and with access controls.

• Performance tradeoffs: batching and caching embeddings reduce cost but increase tail latency; set p99 SLOs explicitly and pick batch-size vs latency tradeoff per endpoint.

• Deployment hygiene: automate canary rollouts, use idempotency keys for retries, implement backpressure and rate limits, and provide feature flags to disable LLM paths quickly.

Worked example — "Design an evaluation and guardrail pipeline for a customer-facing LLM chat service"

First 30 seconds you clarify: what are the high-risk failures (hallucination, toxicity, PII leakage), acceptable latency SLOs, and sampling budget for human review. Organize your answer into three pillars: (1) offline test-suite (unit tests, adversarial corpus, synthetic assertions), (2) online sampling + automated filters (fast classifier → block/fallback), and (3) ops (logging, canaries, alerts). Concretely, propose sampling 1–2% of production queries for full human review, keep a streaming classifier to catch high-risk outputs with a tuned threshold that prioritizes recall for toxicity, and store structured logs for repro with trace ids and model version tags. Flag the main tradeoff: stricter filters reduce risky outputs but increase false positives and user friction — propose metrics to track both. Close with rollout steps: start with internal-only canary, expand via feature flags to gradually increasing traffic, and iterate on annotation schema and thresholds. If you had more time, add simulated adversarial generation and automated A/B experiments measuring downstream user satisfaction.

A second angle — "How would you monitor hallucination and safety regressions at scale?"

Same core concepts but the framing emphasizes continuous detection: implement a lightweight online verifier that detects factual assertions and routes a sampled subset to fact-checkers; compute daily aggregated SLIs (e.g., hallucination rate per intent) and apply statistical change detection (e.g., control charts or simple proportion tests) to trigger alerts. Instrument histograms of output lengths, token-likelihood anomalies, and classifier score distributions to detect distributional shifts. Use canaries and shadow deployments to compare new model versions without user exposure. This angle forces tradeoffs: you must choose what to sample (uniform vs risk-weighted), how much human labeling budget you allocate, and which automated signals to trust for noisy pre-alarms.

Common pitfalls

Pitfall: Over-relying on proxy metrics like BLEU/ROUGE. These surface-level NLG scores rarely correlate with factuality or safety; pair them with human annotations or task-specific verifiers instead.

Pitfall: Logging everything without privacy controls. Dumping raw prompts/responses to logs violates compliance and creates attack surfaces; redact PII, hash identifiers, and apply strict access controls.

Pitfall: Designing one-off manual filters as the only guardrail. Rule-based filters rot quickly under adversarial input and scale badly; combine classifiers, staged fallbacks, and an iterative red-team corpus to keep defenses current.

Connections

This topic naturally pivots to model serving & autoscaling, feature-flag driven rollouts / canarying, and observability for ML (traceability, lineage, and metric alerting). Interviewers may probe deployment orchestration (autoscale, batching), or how you'd pair evaluation with CI/CD for models.

Practice questions

Relevant adjacent design topic; system design is 3/5 with views but no solved signals, so practice APIs and metadata trade-offs.

What's being tested

This tests whether you can design a production-grade distributed storage system with clean API boundaries, durable object storage, strongly modeled metadata, and well-scoped consistency guarantees. The interviewer is probing for how you separate file bytes from metadata, handle concurrency for operations like rename and move, support large resumable uploads, and reason about failures without hand-waving. Harvey cares because legal and enterprise workflows depend on correctness around documents: losing a file, exposing the wrong version, or corrupting folder state is unacceptable. A strong Software Engineer answer should show practical distributed-systems judgment: which components need strong consistency, which can be eventually consistent, and where to use transactions, idempotency, and background repair.

Core knowledge

• Storage/metadata separation is the starting architecture: store file contents as immutable blobs in `S3`, `GCS`, or an internal object store, and store directory trees, permissions, versions, and upload state in a transactional metadata service like `Postgres`, `MySQL`, `Spanner`, or `DynamoDB`.

• Object keys should not be user-visible paths. Use stable IDs such as `file_id`, `version_id`, and `blob_id`; /client/matter/doc.pdf is metadata. This prevents rename from requiring object rewrites and lets the same blob be referenced by multiple versions or deduplicated.

• Directory modeling usually uses either an adjacency list table, with `parent_id` and `name`, or a materialized path / closure table for faster subtree queries. Adjacency lists are simple and transactional; materialized paths make subtree listing easier but complicate renames and moves.

• Uniqueness constraints are essential for correctness. A typical table has UNIQUE(parent_id, name) to prevent two entries with the same name in one folder. For case-insensitive filesystems, normalize names into `normalized_name` and enforce UNIQUE(parent_id, normalized_name).

• Atomic rename/move should be a metadata transaction, not a blob operation. Update `parent_id`, `name`, `updated_at`, and possibly path cache inside one database transaction. Validate permissions, destination existence, name uniqueness, and cycle prevention before commit.

• Per-directory limits need concurrency-safe enforcement. If a folder has max N children, do not read COUNT(*) and then insert without protection. Use a locked counter row, SELECT ... FOR UPDATE, serializable isolation, or a conditional update like UPDATE folders SET child_count = child_count + 1 WHERE id = ? AND child_count < N.

• Transaction boundaries should keep external object storage out of database transactions. Common flow: create upload session in metadata, upload bytes to object storage, verify checksum, then commit a file version referencing the blob. Use a garbage collector for orphaned uploaded blobs.

• Idempotency keys protect create/upload/commit APIs from client retries. Store (user_id, operation, idempotency_key) -> response with a TTL or durable record. Stripe’s idempotency-key pattern is useful: the same key and parameters return the same result; parameter mismatch returns an error.

• Large-file uploads should support multipart upload and resumability. Split files into chunks, track uploaded parts with part numbers, byte ranges, ETags, and checksums. Finalize only after all required parts are present and the full-file checksum matches, e.g. SHA-256(file) == expected_hash.

• Versioning is usually append-only. files represents the logical document, file_versions represents immutable content snapshots, and current_version_id points to the latest committed version. This supports rollback, audit history, collaborative edits, and safe reads during concurrent writes.

• Consistency model should be explicit. Metadata operations like create, rename, delete, and permission changes generally require strong consistency. Blob reads can be served from object storage or CDN with cache validation, but clients should resolve file identity and version through strongly consistent metadata.

• Sync clients need change tracking, not recursive polling. Maintain a monotonically increasing change_seq or per-namespace log of events: create, update, rename, delete, permission change. Clients call GET /changes?cursor=...; if the cursor is too old, force a full resync.

Worked example

For Design a production file storage service, start by clarifying scope: “Are we designing Dropbox-like user storage, enterprise document storage, or a backend service? Do we need folders, rename, permissions, file versioning, and per-directory limits? What scale should I assume for file count, object size, and QPS?” Then declare assumptions: file bytes are large and immutable, metadata must be strongly consistent, and object storage is durable but not transactional with the metadata database.

A strong answer can be organized around four pillars: API design, metadata schema, blob storage/upload flow, and concurrency/failure handling. For APIs, define CreateFolder, StartUpload, UploadPart, CommitUpload, Rename, Move, ListFolder, GetDownloadUrl, and Delete. For schema, separate nodes or files/folders, file_versions, blobs, upload_sessions, and idempotency_keys, with constraints like UNIQUE(parent_id, normalized_name).

The key tradeoff to flag is metadata consistency versus storage scalability: keep rename and directory limits in a transactional database, while storing bytes externally in `S3`-style storage and reconciling through background cleanup. When discussing per-directory limits, call out the race condition explicitly: two concurrent creates can both see 999 children and insert, so the limit must be enforced with a locked counter or conditional write. Close by saying that, with more time, you would add permission inheritance, audit logging, cross-region replication, and disaster recovery testing.

A second angle

For Design a Cloud File Storage Service, the same architecture applies, but the emphasis shifts toward client synchronization, collaboration, and user-visible behavior. You still separate immutable blobs from strongly consistent metadata, but now you should spend more time on GET /changes, conflict handling, offline edits, and version history. A rename might be represented as a metadata event rather than a delete-plus-create, so sync clients preserve identity and avoid re-downloading unchanged bytes. Resumable upload becomes more prominent because cloud clients may upload multi-GB files over unreliable networks. The design should state whether last-writer-wins is acceptable or whether conflicting versions are preserved as separate file versions.

Common pitfalls

Pitfall: Treating paths as the source of truth.

A tempting answer is to store files directly under keys like user_id/folder/doc.pdf and “rename” by copying objects. That fails for large files, concurrent renames, and version history. A better answer uses stable IDs for logical files and treats paths as mutable metadata.

Pitfall: Saying “use S3 and Postgres” without defining invariants.

The interviewer is not testing whether you know popular services; they are testing whether you can preserve correctness under retries, partial failures, and concurrent operations. State invariants such as “no duplicate child names under the same parent,” “a committed version always references an existing blob,” and “a folder’s child count never exceeds its configured limit.”

Pitfall: Over-indexing on scalability while ignoring atomicity.

Some candidates jump immediately to sharding, CDNs, and queues, then miss the hard correctness problem in rename, move, and create. Scalability matters, but for this system the sharp edge is transactional metadata. First design one correct shard or namespace, then explain how to partition by tenant_id, drive_id, or root folder once the invariants are clear.

Connections

Interviewers can pivot from this into distributed transactions, object storage internals, sync protocols, access control, or multi-region replication. Be ready to compare strong consistency versus eventual consistency, discuss cache invalidation for download URLs, and explain how audit logs or event streams support search indexing and client sync without becoming the source of truth.

Further reading

• The Google File System — classic paper on distributed storage tradeoffs, chunking, replication, and metadata coordination.

• Amazon DynamoDB paper — useful background on partitioning, availability, and eventual consistency, even if your metadata layer uses stronger transactions.

• Stripe API Idempotency Keys — practical reference for designing retry-safe create and commit APIs.

Practice questions