# How to Answer: The Project Deep Dive This round is not testing whether your project was impressive — it's testing **your judgment, ownership, and communication**. A mediocre project narrated with sharp decision-making and honest reflection beats a flashy project narrated as a feature tour. Below is a framework for *how* to answer, followed by a complete worked example you can adapt, and finally how to handle the follow-ups. ## Part 1 — Pick the right project Choose a project where **all three** are true: - You owned a meaningful, identifiable slice (ideally you were the tech lead or primary IC on the hardest part). - You can go several layers deep when probed — down to a specific data model, a hot code path, or a postmortem. - It has **measurable outcomes** you can attribute, at least partly, to your own decisions. Avoid: pure team efforts where you were peripheral, projects you can only describe at the slide-bullet level, and "it just worked" projects with no interesting trade-offs or failures. ## Part 2 — Structure the narrative (outcome-first) Lead with the punchline, then earn it. A reliable spine is **Situation → Task → Action → Result**, weighted heavily toward *Action* (your decisions) and *Result* (quantified impact). Suggested 6–10 slide allocation for a 6–8 minute talk: | Slide(s) | Content | |---|---| | 1 | **Title + TL;DR** — problem in one line, your role, 2–3 quantified outcomes | | 2 | Problem and goals (with targets/SLOs) | | 3–4 | Architecture and key components | | 5 | Data model and APIs | | 6 | Design decisions and trade-offs | | 7 | Performance/scalability (with simple math) | | 8 | Testing and rollout | | 9 | Metrics and outcomes | | 10 | Incidents and lessons learned | ## Part 3 — Make ownership and judgment unmistakable This is where most candidates lose points. - Use **"I decided / I built / I owned"** for your work and **"we / the team"** for shared work — and be explicit when the distinction matters. Inflating "we" into "I" reads as dishonest under probing. - Frame every major choice as a **trade-off**: name the alternative, its downside, and why the constraints made your choice right. "We used Kafka because it's good" is weak; "we chose Kafka over SQS to get per-account ordering, accepting heavier ops" is strong. - Quantify impact with **before → after** numbers, and tie each to a decision. Honestly scope what was yours versus the team's versus tailwinds. - Show **reflection**: a real "what I'd do differently" signals seniority. A flawless highlight reel signals the opposite. - Expect interruptions. Adjust depth on request, and when you hit the edge of your knowledge, say **"I don't know — here's how I'd find out."** --- # Worked Example: Real-Time Webhook Delivery Platform v2 (multi-tenant) A fintech platform delivering real-time account and transaction updates to thousands of client endpoints with strict reliability and latency goals. Replace the specifics below with your own. > **TL;DR (Slide 1):** I rebuilt our webhook delivery system for reliability, latency, and cost at multi-tenant scale. As tech lead and primary IC, I owned the design, the delivery service, and the retry scheduler. Result: on-time delivery 97.8% → 99.97%, p99 latency 1.2 s → 240 ms, duplicates −98%, cost/1M deliveries −35%. ## 1) Problem and goals - **Situation:** Webhook delivery was unreliable during traffic spikes and partner outages — 97.8% delivered within 60 s, p99 ≈ 1.2 s, duplicate deliveries during retries, and "noisy neighbor" tenants degrading others. Cost/1M deliveries was high due to inefficient retries and hot shards. - **Goals (6-month targets):** ≥ 99.95% delivered within 60 s; duplicates < 5 per 1M; p99 < 300 ms for endpoints responding within 200 ms; sustain 25k events/s, burst 50k; −30% cost/1M; data residency (US/EU), signed deliveries, tenant isolation. - **My role:** Tech lead + primary IC. I wrote the RFC, led the design, implemented the delivery service and retry scheduler, drove the rollout, and owned on-call for the new system. ## 2) Stakeholders and constraints - **Stakeholders:** Partner/Customer Engineering (integration success), SRE (SLOs, on-call), Security (signing, egress control), Product (feature parity), Finance (cost). - **Constraints:** backward-compatible payloads and zero-downtime migration; **at-least-once** semantics (consumers must be idempotent); per-region data residency and per-tenant isolation; unknown partner rate limits (so fair-sharing was required); 2 quarters, team of 4 engineers + 1 SRE. ## 3) System architecture and key components - **Producers:** event pipeline emits normalized domain events (`account_linked`, `transaction_posted`, `balance_updated`). - **Event bus:** Kafka, partitioned by `account_id` to preserve per-account ordering; cross-region replication for DR. - **Delivery service (Go):** consumes events, applies tenant/topic policy, computes an idempotency key, signs the payload (HMAC-SHA256), and POSTs to the tenant URL. Per-tenant token-bucket rate limiter + circuit breakers; connection reuse (keep-alive, HTTP/2 where supported). - **Retry orchestrator:** exponential backoff with jitter via staged delay topics (10 s, 1 m, 10 m, 1 h) to avoid tight loops; dead-letter queue (DLQ) to object storage with alerting after N attempts. - **Idempotency/dedupe:** key-value store with PK = `hash(tenant_id + event_id)`, 7-day TTL; conditional write detects duplicates. - **Config/tenancy:** endpoint registry (`url`, `secret`, `topics`, rate limit, region); per-tenant concurrency budgets. - **Observability/control:** metrics dashboards with SLIs (success-within-60 s, p50/p95/p99 latency, retries, queue depth, DLQ rate); feature flags, traffic shadowing, kill switches. ## 4) Data model and APIs Core schemas: - `Event { event_id, tenant_id, type, created_at, data_hash, payload, size_bytes }` - `Delivery { delivery_id, event_id, tenant_id, endpoint_id, attempt, status, http_code, latency_ms, next_attempt_at }` - `Endpoint { endpoint_id, tenant_id, url, secret, topics[], rate_limit_rps, region, version, created_at }` Outbound delivery (to tenant): - Body: `{ id, type, created_at, data }` - Headers: `X-Webhook-Id`, `X-Request-Id`, `Idempotency-Key`, `X-Signature` (HMAC over `timestamp + body`), `X-Timestamp`. Management APIs: - `POST /v1/webhooks/endpoints`, `PATCH /v1/webhooks/endpoints/:id` - `GET /v1/webhooks/deliveries?since=...&status=...` - `POST /v1/webhooks/_test { endpoint_id }` ## 5) Major design decisions and trade-offs - **Delivery semantics — at-least-once vs exactly-once:** chose **at-least-once + strong idempotency** (keyed by `tenant_id + event_id`). Exactly-once across heterogeneous HTTP targets is brittle and costly; pushing dedupe to a single owned store is cheaper and more reliable. - **Event bus — Kafka vs SNS/SQS:** chose **Kafka** for per-account ordering and high burst throughput, accepting heavier operational overhead. SNS/SQS is simpler to run but gives weaker ordering and needs more plumbing for staged retries. - **Retry strategy — time-wheel vs staged delay topics:** chose **staged delay topics** for operational simplicity and linear cost, with bounded backoff + jitter to avoid synchronized retry storms. (In hindsight a unified time-wheel scheduler would reduce topic sprawl — see lessons.) - **Security — HMAC vs mTLS:** chose **HMAC-SHA256 by default** for easy client integration, with optional **mTLS** for high-security tenants. - **Isolation — global worker pool vs per-tenant budgets:** chose **per-tenant rate limits + circuit breakers** to stop noisy-neighbor impact, with global guardrails protecting shared infrastructure. ## 6) Performance and scalability considerations - Target 25k events/s sustained, 50k burst; avg payload ≈ 2 KB → peak outbound ≈ 100 MB/s. - **Concurrency sizing (Little's Law, $L = \lambda W$):** with avg endpoint RTT $W \approx 0.15$ s and $\lambda \approx 25{,}000$/s, concurrency $L \approx 25{,}000 \times 0.15 \approx 3{,}750$ in-flight requests. Add ~50% headroom for bursts/GC/network variance → ≈ 5{,}600 workers; autoscale on outstanding requests and queue depth. - **Partitioning:** enough Kafka partitions to avoid hot shards; key by `(tenant_id, account_id)` so heavy tenants spread across partitions. - **Latency controls:** keep-alive + connection pools, per-tenant dial timeouts, TLS session reuse, fast-path signing over pre-hashed payloads, and low GC pressure (buffer pooling in Go). - **Backpressure/fairness:** token bucket (burst ≈ 5× steady RPS) with slow-start for recovered endpoints. - **Cost controls:** retry caps (e.g. 6 attempts over 24 h), DLQ archiving, Kafka compression, right-sized instances, and batched status updates to avoid hammering hot stores. ## 7) Testing and rollout plan - **Testing:** unit + property-based tests for signing, idempotency-key generation, and retry math; **golden-payload tests** ensuring canonical JSON serialization before signing; contract tests against the OpenAPI spec; integration tests in ephemeral environments with **fault injection** (timeouts, 429, 5xx, TLS errors); load tests ramping to 60k events/s; chaos tests (broker failover, network partitions). - **Rollout:** **shadow mode** — mirror ~10% of events to v2 and diff delivery decisions *without* calling tenant endpoints; then **canary** to 5 pilot tenants → 10% → 50% → 100% over ~2 weeks, with automatic rollback if an SLO is breached for 10 minutes. Kill switches at per-tenant, per-topic, and global scope; runbooks documented before cutover. ## 8) Metrics and outcomes - **Reliability:** success-within-60 s 97.8% → **99.97%**. - **Latency:** p99 1.2 s → **240 ms**; p50 180 ms → **95 ms**. - **Duplicates:** ~150 per 1M → **3 per 1M** (−98%). - **Scale:** sustained 30k events/s in production; absorbed 2× bursts during bank outages without customer impact. - **Cost:** −35% per 1M deliveries via smarter retries, batching, and right-sizing. - **Ops:** on-call pages ~12/month → **2/month**; MTTR 38 min → **14 min**. (Be ready to scope honestly: the reliability and duplicate wins were directly mine via the idempotency + retry design; the cost win was shared with SRE's right-sizing work.) ## 9) Notable failures/incidents and mitigations - **Hot-partition incident:** a tenant with large batches drove a single partition to 95% CPU and queue buildup. **Root cause:** keying only by `tenant_id` created skew. **Fix:** composite key `(tenant_id, account_id)` + a repartitioning tool; autoscale on partition lag. - **Signature mismatch with a major tenant:** 401s after a JSON field-reordering change. **Root cause:** non-canonical JSON serialization in one code path. **Fix:** canonicalized serialization, added golden tests, and **versioned signing** (v1/v2) so tenants could migrate without breakage. - **Retry storm during a partner outage:** a thundering herd of retries amplified the partner's downtime. **Fix:** exponential backoff with jitter, retry caps, a per-domain circuit breaker, and dynamic backoff driven by a partner health score. ## 10) Lessons learned and what I'd do differently - Define **SLOs and error budgets up front** and let them gate the rollout — they prevented a premature 100% cutover. - Bake **tenant isolation in early** (rate limits, circuit breakers); retrofitting it after the hot-partition incident was costly. - Treat **idempotency, canonical serialization, and versioning as first-class** — the signature incident was avoidable. - **What I'd change:** adopt a unified time-wheel scheduler to kill delay-topic sprawl; make mTLS the default for high-risk tenants; and ship per-tenant sandbox/self-serve validation earlier to cut integration support load. --- # Addressing the Follow-up Questions - **"Hardest decision and the counterfactual?"** At-least-once vs exactly-once. Had I chased exactly-once, I'd have built distributed coordination across thousands of uncontrolled HTTP targets — fragile, expensive, and still defeated by partner-side retries. At-least-once + an owned idempotency store gave the same *effective* guarantee for consumers at a fraction of the complexity. The cost: consumers must be idempotent, which I de-risked with clear docs and an `Idempotency-Key` header. - **"Where did you disagree, and how did it resolve?"** SRE pushed for SNS/SQS to minimize operational load. I agreed on the ops cost but argued ordering was a hard product requirement; we resolved it by my committing to own Kafka runbooks and on-call for the first quarter, with a review gate to revisit if ops burden proved unsustainable. It didn't, and the gate built trust. - **"Which gains were truly yours?"** The reliability and duplicate-rate wins trace directly to the idempotency + retry design I owned. The cost reduction was partly mine (retry policy) and partly SRE's (right-sizing) — I'd present it as a shared win. - **"What breaks first at 10× load?"** The synchronous idempotency conditional-write becomes the bottleneck before raw delivery throughput does. I'd shard the dedupe store by `tenant_id`, move to a local bloom-filter pre-check to skip the remote round-trip for obvious non-duplicates, and pre-scale partitions ahead of the ramp rather than reacting to lag. --- ## One-paragraph version (if you only have 90 seconds) "I rebuilt our multi-tenant webhook delivery platform. The old system delivered 97.8% of events on time with p99 latency of 1.2 s and frequent duplicates, and one noisy tenant could degrade everyone. As tech lead and primary IC I owned the design, the Go delivery service, and the retry scheduler. The core decision was at-least-once delivery backed by a strong idempotency store rather than chasing exactly-once across uncontrolled HTTP endpoints, plus per-tenant rate limits and circuit breakers for isolation. I rolled it out via shadow mode then a staged canary gated on SLOs. Result: on-time delivery hit 99.97%, p99 dropped to 240 ms, duplicates fell 98%, and cost per million deliveries dropped 35%. The biggest lesson was building tenant isolation in from day one — we learned that the hard way through a hot-partition incident."