Discuss complex project and multi-approach example

## What the interviewer is listening for The two prompts probe different things, and candidates often collapse them into one story. - **Prompt 1** tests scope and ownership: can you carry something genuinely hard from ambiguity to a measured result, and can you separate your contribution from the team's? - **Prompt 2** tests decision-making: do you generate real alternatives, evaluate them against criteria, and change your mind on evidence — or do you commit to the first idea and defend it? Use two different projects if you can. If you must reuse one, make the second pass clearly about the decision process, not the project narrative again. ## Prompt 1 — the most complex project you led Use STAR+L (Situation/Task, Actions, Results, Lessons) and cover these in order: 1. **Context and stakes.** One or two sentences: the problem, the scale (QPS, data size, users, money), the constraints (SLOs, security, compliance, deadlines). 2. **What made it complex.** Be specific about the *source* of complexity — hard latency budget plus multi-source joins, a zero-downtime migration, cross-team dependencies, incomplete labels, regulatory review. "It was a big project" is not complexity. 3. **How you decomposed it.** Name the workstreams and their order. This is where leadership shows. 4. **Your specific contributions.** What *you* designed, wrote, decided, or unblocked — and, just as usefully, what you did not own. Interviewers trust candidates who draw that line honestly. 5. **Key risks → mitigations.** Latency, correctness, data quality, availability, cost, privacy. 6. **Measurable outcomes.** Before/after on the metric you named in step 1, plus reliability and operational load. 7. **Lessons.** Two or three reusable insights, including what you would do differently. ### Sample answer (illustrative numbers — substitute your own) **Situation/Task.** I led a six-month effort at a crypto exchange to build real-time risk scoring for withdrawals. Goal: cut fraud and account-takeover losses from about 12 bps to under 5 bps while keeping more than 99% of legitimate withdrawals under two minutes. Peak traffic ~1.5k RPS, p95 latency budget 80 ms, availability target 99.99%. The asymmetry that drove every design choice: an on-chain withdrawal is irreversible, so a miss cannot be clawed back the way a card chargeback can. **What made it complex.** - A tight latency budget on a decision that needed joins across device, IP, funding history, and account age — with sparse, delayed fraud labels. - A zero-downtime migration off a batch rules engine onto an online decisioning path. - Large blast radius: compliance, payments, support, and SRE all affected, with strict auditability requirements on every decision. **Decomposition.** 1. Requirements and success metrics, agreed with Product, Risk, and Compliance. 2. Architecture and service interfaces. 3. Feature pipeline and online feature store. 4. Online scoring service and integration into the withdrawal path. 5. Observability, SLOs, and guardrails. 6. Rollout: shadow → canary → ramp → rollback path. 7. Change management: runbooks, on-call training, appeal workflow for support. **My specific contributions.** I wrote the design doc and drove it through review; built the stateless gRPC scoring service and the feature-hydration layer (Redis with TTLs for hot features, batch backfill for offline updates); defined the feature contract and versioning scheme with the ML team; owned the rollout plan and was primary on-call for the first quarter. I did *not* build the model — ML owned training and calibration — but I owned the decision layer that combined the score with business constraints (KYC tier, amount, velocity), and I owned the fallback policy. **Key risks and mitigations.** - *Latency blowups from downstream lookups* → per-dependency timeouts inside a 60 ms request budget, bulkheads, circuit breakers, and cached features. - *Scorer or feature store unavailable* → fall back to the deterministic rules engine; **above a value threshold, hold the withdrawal for manual review rather than auto-approving.** Defaulting to allow is the tempting shortcut and the wrong one here — for an irreversible transfer, the safe failure mode is delay, not release. Below the threshold, auto-approve to protect the experience. - *False positives harming good users* → conservative launch thresholds, segment-specific thresholds, an appeal path staffed by support, weekly calibration review. - *Data quality drift* → feature freshness SLAs, schema validation, null-safe defaults, population-shift alerts. - *Compliance and auditability* → immutable decision logs capturing the feature snapshot and model version behind every decision. **Measured outcomes (first 60 days).** Losses 12 bps → 4.9 bps (~59% reduction). 99.6% of legitimate withdrawals completed in under two minutes; manual reviews down 40%. Availability 99.992%, p95 62 ms, p99 95 ms. Pages from this flow dropped from 7/month to 2, and two legacy cron jobs were retired. **Lessons.** Ship a simple, observable baseline before the sophisticated version — the calibrated logistic regression bought us the latency headroom to iterate. Most incidents were data problems, not model problems, so invest in freshness and validation early. Agree risk tolerance with Product and Compliance *before* tuning thresholds, or you will retune under pressure. Kill switches and fallbacks are features, not afterthoughts. ## Prompt 2 — the same problem, several methods Structure the answer as a decision, not a chronology: 1. **State the problem as a metric with a guardrail.** "Cut p99 on the orders API from 900 ms to under 500 ms without pushing error rate above 0.5% or staleness past 2 s." 2. **List two to four methods that attack different root causes.** Algorithmic/serialization, caching, data-layer tuning, concurrency, protocol. Methods that all attack the same root cause are one method wearing three hats. 3. **Name your selection criteria.** Impact, risk, blast radius, time-to-value, correctness, ongoing operational load. 4. **Explain how you measured.** Profiling, canaries, A/B, shadow comparison, synthetic peak load. 5. **State the decision — including what you rejected and why.** The rejection is often the most informative part. Method categories and their usual trade-offs: | Method | Upside | Cost / risk | |---|---|---| | Algorithm, data structure, serialization | Durable, no new moving parts | May need refactors | | Caching | Fast to land, big wins on hot keys | Staleness, invalidation, stampedes | | Data-layer tuning (indexes, replicas, denormalization) | Attacks the real bottleneck | Consistency and replica-lag trade-offs | | Concurrency / async | Throughput and tail latency | Needs idempotency and backpressure | | Protocol / compression | Cuts wire cost | Cross-service coordination | ### Sample answer (illustrative numbers — substitute your own) **Problem.** The order-history endpoint ran a p99 of about 900 ms at peak. Target: under 500 ms p99, error rate under 0.5%, read staleness under 2 s. **Methods, and why each.** 1. *Profiling and algorithmic fixes.* Flame graphs showed ~35% of time in JSON serialization and ~25% in an in-process sort. Switched the internal hop to protobuf and pushed ordering into the database behind a composite index, removing the in-app sort. Lowest risk — code-only, no new infrastructure. 2. *Read-through cache.* A 2 s TTL cache on order summaries keyed by user and status, with single-flight request coalescing so a cold key cannot stampede the database. Chosen for a quick win on hot keys, fenced by a circuit breaker and hit-rate monitoring. 3. *Indexes and read replicas.* Composite indexes plus routing read-heavy traffic to replicas, with replica lag alarmed at 1.5 s to keep the freshness guarantee honest. **How I evaluated.** Each change canaried at 5% then 25%, with synthetic peak load to see behavior above organic traffic. Guardrails: error rate under 0.5%, replica lag p95 under 1.5 s, and a sampled correctness check comparing cached responses against the source of truth. **Outcome.** Serialization and sort fixes alone: p99 ~650 ms, CPU down ~18%. Adding the cache: p99 ~520 ms at a ~72% hit rate, no correctness regressions. Adding indexes and replicas: p99 ~430 ms, error rate 0.2%, freshness p95 1.2 s. All three shipped behind flags with dashboards for cache health and replica lag. **What I rejected and why.** An async write path would have helped throughput, but the order and balance endpoints cannot tolerate a window where a filled order is not yet visible — the consistency risk outweighed the latency gain. I also deferred a full read-model rewrite: similar projected win, far larger blast radius, and the three cheaper changes already cleared the SLO. **Lesson.** The three effects were not additive — the cache absorbed part of the win the indexes would otherwise have delivered, so I measured each in isolation before stacking them. If I had shipped them together I would have "proved" whichever one landed last. ## Reusable template - Context: [domain], [scale], [constraints and SLOs]. - Complexity: [the specific thing that made it hard]. - Decomposition: [workstreams, in order]. - My contribution: [what I designed / built / decided / unblocked], [what I did not own]. - Risks → mitigations: [top three]. - Results: [baseline → target → actual], [reliability], [operational load]. - Multi-method: problem [metric + guardrail] → methods [A, B, C] → criteria [impact, risk, time] → evaluation [canary, profiling, shadow] → decision [chosen, rejected, why] → lesson. ## Pitfalls to avoid - **Vague impact.** "Improved performance" tells the interviewer nothing. Give the before, the after, and how you measured. - **Hiding behind "we."** If you cannot say what you personally decided or built, the story does not count as yours. - **Skipping rollout and operability.** No canary, no kill switch, no on-call plan reads as someone who has never shipped something risky. - **One-dimensional fixes.** Prompt 2 exists specifically to see alternatives and trade-offs; presenting a single approach fails it outright. - **Fail-open defaults on a risk or safety path.** Say what happens when your dependency is down, and make sure the default is defensible. - **Numbers you cannot defend.** If you do not have exact figures, give a range and say how you measured or would measure. Fabricated precision collapses under one follow-up. ## Self-check before you answer - Baseline and target metrics stated up front. - Complexity attributed to a specific cause, not to size. - Your ownership boundary drawn explicitly. - At least two genuine alternatives with the criteria that separated them. - Guardrails named: canary plan, rollback condition, correctness checks. - One honest lesson, including something you would do differently.

Rubric for both prompts. Prompt 1 is scored on scope and ownership: a specific source of complexity, a decomposition that shows leadership, an explicit boundary between what the candidate owned and what the team owned, named risks with mitigations, and quantified before/after outcomes. Prompt 2 is scored on decision quality: a problem framed as a metric plus a guardrail, two or more alternatives that attack different root causes, stated selection criteria, a real evaluation method (profiling, canary, shadow, A/B), and a defended rejection. Strong answers carry numbers and one honest lesson; weak answers are chronological, use "we" throughout, and present a single approach.