Walk through a project deep dive

### What this interview actually tests This is a hiring-manager **project deep dive**, not a coding or whiteboard round. The format is usually ~10 minutes of the HM describing their team, then **30–40 minutes is yours** to drive. The signal the interviewer wants: - **Ownership** — can you articulate what *you* did versus what the team did? - **Technical depth** — do you understand *why* the system is built the way it is, not just *what* it does? - **Judgment** — did you weigh real alternatives and accept trade-offs deliberately? - **Impact** — did the work move a metric that mattered to users or the business? - **Communication & leadership** — can you teach a complex system to a smart stranger, and have you grown others (mentorship)? A practical note from candidates who have taken this exact format: **prepare slides.** A handful of clean diagrams with little text, where you talk over the visuals, lands far better than a wall of words or a purely verbal walkthrough. You are being evaluated partly on whether you can *present*. Drive the conversation, control the pacing, and let the visuals carry the structure so the interviewer can follow without effort. Most of the round is you talking, so the bar is "could this person credibly lead a project review with my team?" --- ### How to pick the right project Choose a project where **you were clearly central** (not a bystander on a big team), that has **real stakes** (reliability, performance, cost, security/privacy, or a major user/business win), and where you can speak to **end-to-end ownership** — problem framing through launch and measurement. Avoid projects you only touched peripherally; the deep questions will expose it. One project told well beats three told shallowly. If you lack precise before/after numbers, that is fine — give honest ranges and describe **how you would measure it now**. Interviewers care more about whether you *think* in metrics than whether you memorized a specific figure. --- ### Time budget for 30–40 minutes Rehearse to roughly this split so you don't spend 25 minutes on context and run out of time before impact. The targets below are sized so even the high end fits a 40-minute round with a couple of minutes to spare: | Section | Target | |---|---| | Problem context & goals | 3–4 min | | Your role & responsibilities | 1–2 min | | Architecture & technical approach | 7–8 min (the centerpiece) | | Key decisions & trade-offs | 4–5 min | | Timeline & constraints | 2–3 min | | Metrics & impact | 3–4 min | | Biggest challenge deep dive | 5–7 min | | Lessons learned | 2–3 min | | Mentorship | 2–3 min | The HM will almost certainly interject on **mentorship** and **biggest challenge**, so have those rehearsed as standalone mini-stories you can pull forward on demand, not buried inside the main narrative. If they pull those questions out separately, fold the corresponding rows back into their interjection rather than spending the time twice. --- ### A reusable scaffold for each requested section Below is a section-by-section checklist of what a strong answer covers. Fill it with **your own** project. The worked example that follows shows the *level of specificity* to aim for — treat its numbers as illustrative placeholders, not lines to recite. **1) Problem context and goals** - What was broken or missing, and who felt the pain (which users/customers/internal teams)? - *Why now* — what forcing function made this worth doing (incidents, growth, a compliance deadline, a cost ceiling)? - The success criteria stated as **measurable targets**, not vibes ("cut p95 from 800ms to <200ms," not "make it faster"). **2) Your role and responsibilities** - Your title/scope, team size, and key collaborators (other teams, security, SRE, PM, design). - A crisp sentence on **what you personally delivered** versus what others did. Use "I" for your work and "we" for the team — interviewers listen for this split. **3) Technical approach and architecture** - The high-level design: components, data flows, and **interfaces/contracts** between them. - This is where the diagram earns its keep. Walk the request path end to end. Be ready to zoom into any box if asked. **4) Key decisions and trade-offs** - For 2–3 decisions: the **alternatives** you considered, why you chose what you chose, and the **risk you knowingly accepted**. - Strong candidates name the *cost* of their choice, not just its benefit ("a central service adds a network hop; we accepted that and mitigated it with caching"). **5) Timeline and constraints** - Milestones and how you sequenced them (e.g., POC → pilot → broad rollout). - Real constraints: backward compatibility, compliance/security review, performance/SLA targets, budget, dependencies on other teams. **6) Metrics of success** - **Baseline vs. outcome**, side by side. Pick from: latency, throughput, availability, error rate, cost, developer velocity, adoption. - Tie at least one technical metric to **user or business value** ("99.99% availability meant zero auth-related Sev-2s that quarter, which unblocked the audit"). **7) Lessons learned** - One thing you'd repeat, one you'd avoid, one you'd do differently. Make it specific and a little self-critical — generic "communicate more" answers are forgettable. --- ### Worked example (illustrative — use it as a template, not a script) > The example below is a centralized authorization (AuthZ) service. It is here to show **structure and depth**, not for you to memorize. Swap in your own project and your own honest numbers. **Problem context & goals.** ~15 microservices each implemented authorization on their own — duplicated logic, inconsistent enforcement, and no central audit trail to answer "who could access what, and when?" Auth bugs were causing recurring Sev-2 incidents, and an upcoming compliance review needed evidence we couldn't easily produce. *Why now:* the compliance deadline plus a third auth incident in two quarters. **Targets:** one central check API at **p95 ≤ 5 ms** and **≥ 99.99% availability**, near-real-time revocation (policy change to enforced effect in **≤ 2 s**), and complete, queryable audit logs. **Role.** Tech lead and primary IC on a team of three engineers plus a part-time SRE. *I* designed the architecture and the cache-consistency mechanism and built the policy-evaluation core; *we* split the migration tooling and rollout, and I ran the design reviews and on-call readiness. **Architecture.** A stateless central service behind a load balancer, multi-AZ. - **Interfaces:** `Check(user, action, resource) → allow/deny + reason`, `ListPermissions(user)` for UIs, and a `DryRun` mode that returns the decision *and* — where the engine's decision-logging/trace support can surface it — *which policy or rule matched* (so services could migrate safely before enforcing). Note that "which policy matched" isn't free: for a hybrid RBAC+ABAC bundle, a deny in particular requires the evaluator to emit per-rule coverage/trace data (e.g., explicit decision logging in an OPA/Rego-style engine), so call out that dependency rather than assuming it comes for nothing. - **Policy model:** hybrid RBAC + ABAC — roles for the common cases, attributes (tenant, department, data sensitivity) for finer control. - **Data path:** Postgres is the source of truth for role bindings and attributes; changes flow via change-data-capture into a message bus. The read path serves from a Redis cluster (materialized effective permissions) with a small per-instance in-memory cache for the hottest lookups. Permission updates publish invalidation messages that update Redis and notify service instances. - **Safety:** fail-closed by default, with an explicit allowlist for health checks and an admin break-glass path. **Key decisions & trade-offs.** - *Central service vs. embedded library/sidecar:* chose central for consistent policy and unified audit; **accepted** the extra network hop and mitigated it with two cache layers and multi-AZ placement. - *RBAC vs. ABAC vs. fully custom DSL:* chose hybrid to avoid the long-term cost and auditability problems of a bespoke DSL while still covering fine-grained cases. - *Caching vs. revocation speed:* caches cut latency but risk staleness; we accepted short TTLs on high-risk resources plus an explicit invalidation channel rather than relying on TTLs alone. **Timeline & constraints.** Roughly two quarters: Q1 was discovery, the policy-model POC, API design, security review, and two pilot services; Q2 was hardening, load testing, runbooks, and rollout to the rest, plus a migration tool to map legacy roles. Constraints: legacy services needed backward-compatible semantics, a hard compliance deadline, and a budget that ruled out exotic infra (we reused managed Redis/Postgres). **Metrics & impact.** Check latency landed around **p50 ~2 ms / p95 ~4 ms**; availability **99.99%+** in the first full quarter with **zero auth-related Sev-2s**; revocation propagation held under the 2 s target. Audit-evidence gathering went from days to minutes, which is what actually got us through the compliance review. (Use your own real numbers; ranges are fine if you're honest about them.) **Lessons.** Design revocation explicitly — TTLs are not a consistency strategy. A `DryRun`/explainability surface de-risked every migration and paid for itself in incident response — but budget for it: surfacing *which* rule matched (especially on a deny) means turning on per-decision trace/coverage logging, not a free byproduct of evaluation. And we should have scheduled stakeholder demos earlier to lock down the API contract before building. --- ### Part B — Biggest challenge (have this ready as a standalone story) The HM will likely pull this out separately. Structure it as **what → why hard → diagnosis → fix → alternatives → impact**: - **What:** Rare but critical *cache staleness* — after a role change, a permission revocation occasionally didn't take effect fast enough, especially during cross-AZ message-bus failover. - **Why it was hard:** Three layers of caching plus eventual consistency on the change-capture stream meant several interacting failure modes, and the race was timing-sensitive and hard to reproduce on demand. - **Diagnosis:** Added distributed tracing with a decision ID, cache hit/miss tags, and a per-principal "permission epoch" tag so I could correlate which epoch each decision was computed at and see when a cache served an epoch older than the source. Then built a chaos test that injected message delay, duplication, and AZ failover — which reproduced the stale decision reliably. - **Fix:** Introduced a monotonic **revocation epoch** per principal at the **source of truth**, incremented on any permission change. The key is to compare cached decisions against that *current source epoch*, not against the token: a revocation does not re-mint outstanding tokens, so the token's epoch stays old and can't be the trigger — the live signal has to come from the source epoch advancing. Concretely: the invalidation channel propagates each new source epoch forward, so every Redis entry and in-memory cache entry records the epoch it was materialized at, and each instance keeps a cheap per-principal "latest known source epoch" updated from that channel (falling back to a source read on a miss). On `Check`, if the principal's current source epoch is **greater than** the cache entry's epoch (`source_epoch > cache.epoch`), the entry is stale: bypass the caches, read fresh from source, and backfill at the new epoch. The token's own epoch is used only as a *floor* — never serve a decision computed at an epoch older than the token was minted at — which protects a fresh token that lands on a not-yet-updated instance. Added a short max-staleness guard that fails closed if an instance hasn't seen an epoch update within N seconds, with the admin break-glass route as the escape hatch. - **Alternatives weighed:** disabling the in-memory cache (too costly at peak QPS), ultra-short global TTLs (spiked the cache tier and tail latency), and a per-service sidecar (great locality but lost the centralized audit we'd built the whole thing for). The epoch approach gave correctness without sacrificing latency or audit. - **Impact:** Revocation-correctness issues dropped to zero in the following quarters, p95 revocation held under target even during failovers, and we didn't have to overprovision the cache tier to get there. The point of this section is to show you can **diagnose under uncertainty** (instrument → reproduce → fix) and that you reach for principled mechanisms, not band-aids. --- ### Part C — Mentorship (also have this ready standalone) Frame it with **goals → guidance → measurement → outcome → what you'd change**. Keep it concrete and humble; don't claim you single-handedly transformed someone. - **Goals you set:** e.g., onboard an intern to ship a real, scoped feature (say, the `DryRun` API and a simple "explain this decision" UI) — a minimal version by week 4, production-ready by week 8 — with an explicit quality bar (test coverage, CI integration, a latency budget). - **How you guided:** a written onboarding doc (architecture, sample traces, test data) so they weren't blocked on you; an early pairing/design session to shape the API together; weekly 1:1s plus quick async check-ins; code reviews that emphasized testability and error handling, with a couple of your own past PRs as reference examples. The goal is to **unblock without doing it for them**. - **How you measured progress:** clear acceptance criteria on each task, tracked PR cycle time and test pass rates, and a mid-point demo to the consuming teams to validate scope and gather feedback. - **Outcome:** they shipped the feature, it got adopted by the downstream services, and (if true) they converted to full-time and started owning runbooks. Use a real, modest outcome over an inflated one. - **What you'd do differently:** e.g., scope the first deliverable even tighter to build momentum, and schedule the stakeholder demo earlier so requirements were locked before they wrote code. What this demonstrates: you can **set direction, delegate trust, give actionable feedback, and grow people** — which is exactly the leadership signal a hiring manager is screening for. --- ### Delivery checklist - **Lead with the diagram.** Open the architecture section by orienting the interviewer on the picture, then narrate the request path. Don't read text off a slide. - **Say "I" for your work, "we" for the team's** — consistently. - **Quantify before/after**, and translate at least one technical metric into user/business value. - **Name your trade-offs**, including the cost you accepted, not just the upside. - **Time-box yourself.** Watch the clock; don't let problem-context eat the impact section. - **Invite the deep dive.** Pause at natural points ("happy to go deeper on the cache layer or the policy model — what's most useful?") to show you can go arbitrarily deep and to keep the HM engaged. - **Keep proprietary details anonymous** — describe *capabilities*, not internal names or secrets. ### Common pitfalls to avoid - Drowning in setup and never reaching the technical core or the impact. - A monologue with no visuals in a round that rewards visual, well-paced presentation. - Vague impact ("it was a big success") with no baseline or number. - Claiming team work as solo work — experienced interviewers probe this and catch it. - Presenting only the happy path: no alternatives, no trade-offs, no honest failure mode. Showing one thing that went wrong and how you handled it makes the whole story more credible.