Walk through a project in detail

# How to Structure Your Answer (Quick Framework) - Situation and Goal: One sentence context and why it mattered. - Scope and Constraints: SLOs, scale, dependencies. - Plan and Milestones: How you phased the work. - Your Role as DRI: Decisions you owned; examples of doc/PR artifacts. - Decision Log: Alternatives → criteria → chosen approach. - Risk Register and Mitigations: What you tested and why. - Results: Before/after metrics and business impact. - Learnings: What changed in your mental model/process; what you’d do next. I’ll demonstrate with a concrete example suitable for a consumer two‑sided marketplace. # Example Project: Real‑Time Search Suggestions Re-architecture (Latency, Relevance, Cost) ## 1) Situation and Goal - Context: Our typeahead suggestions backed by a monolith had p95 latency ~450 ms, p99 ~900 ms during peak; relevance degraded for new inventory; cache hit rate ~65%. This impacted search session conversion. - Goal: Reduce p95 to <200 ms and p99 <400 ms globally, improve relevance CTR by ≥5%, and cut infra cost by ≥15%, without harming error rate or freshness. ## 2) Timeline and Milestones - Jan 9 – Feb 3: Discovery and design (RFC v1, load testing of baseline, field studies with DS and UX). - Feb 6 – Mar 10: Prototype service (gRPC, Redis cluster, feature gating); shadow traffic in one region. - Mar 13 – Apr 7: Data pipeline and feature store integration; consistency model, backfill jobs. - Apr 10 – May 5: Hardening (cache stampede controls, circuit breakers, retries, chaos tests). - May 8 – May 26: Canary rollout (5% → 25% → 50% of traffic); experiment and guardrails. - May 29 – Jun 16: Global rollout; follow-up optimization and cost tuning. ## 3) Scope and Success Criteria - In scope: Suggestion API rewrite, per‑region Redis cache, Kafka ingestion for inventory updates, relevance model feature hooks, observability and SLOs. - Out of scope: Full ranking model retrain, i18n expansion, UI changes beyond minor logging hooks. - SLOs: Availability 99.95%, p95 <200 ms, freshness SLA ≤1 min for new inventory in suggestions. - Success criteria: +5% CTR on suggestions, +0.5 pp search‑to‑booking conversion, −15% infra cost. ## 4) Team and Roles - 1 Staff SWE (me, DRI): Architecture, RFCs, cache design, rollout, performance. - 1 Backend SWE: Data pipeline (Kafka consumers), feature store integration. - 1 SRE: Capacity planning, HPA policies, canary/rollback automation. - 1 DS: Experiment design, metrics, attribution, power analysis. - 1 PM: Scope, milestones, stakeholder comms. - 1 UX: Logging taxonomy for suggestion categories and click tracking. ## 5) My Responsibilities - Authored design doc and decision log; presented trade-offs. - Implemented service API (gRPC), request coalescing, cache invalidation, fallback flow. - Built dashboards and SLOs (p50/p95/p99, error rate, hit rate, freshness lag). - Led load testing, chaos drills, and global rollout with feature flags and canary. - Owned 70+ PRs; ran postmortems for two early incidents. ## 6) Architecture (Before → After) - Before: Monolith endpoint → DB queries + in-process LRU cache → single region; sync fan-out calls; no backpressure; high cold misses at peak. - After: Edge → Suggestion Service (gRPC) → - Tier‑1 per‑region Redis Cluster (sharded, 3 replicas) with request coalescing and TTL jitter. - Kafka topic of inventory/behavioral updates; consumers update cache and feature store. - Fallbacks: If miss or cache degraded, call read‑only replica; degrade to static popular queries list. - Observability: RED and USE metrics, tracing, SLO burn alerts. ## 7) Key Technical Decisions and Trade‑offs 1) Protocol: gRPC vs REST - Criteria: Latency, type safety, observability, internal network. - Choice: gRPC (HTTP/2 multiplexing, protobuf). Alternative REST would ease browser clients but not needed internally. 2) Cache: Redis vs Memcached - Need for sorted sets and Lua for request coalescing and stampede control. - Chose Redis Cluster. Memcached had lower overhead but weaker data structures. 3) Invalidation Strategy: TTL-only vs Pub/Sub vs Write‑through - Chose event‑driven updates via Kafka + TTL with jitter. Pure TTL was too stale; write‑through increased coupling. 4) Consistency: Eventual vs strong - Eventual with freshness SLA ≤60 s. Strong consistency would require synchronous writes on every inventory change, hurting write path and availability. 5) Deployment: Blue‑Green vs Rolling - Blue‑Green in initial rollout for easy rollback; Rolling later to reduce capacity cost. 6) Backpressure: Token Bucket vs Leaky Bucket - Implemented token bucket per caller to cap overload while allowing bursts. Leaky bucket smoothed too aggressively and added latency. ## 8) Risks and Mitigations - Hot key and cache stampede - Mitigations: Single‑flight request coalescing; probabilistic early refresh; TTL jitter; hot-key sharding. - Freshness lag after spikes in updates - Mitigations: Consumer autoscaling on Kafka lag; DLQ with replay; idempotent updates via content hash. - Regional failover - Mitigations: Multi‑region Redis read replicas; circuit breaker to degrade to static list; retry with exponential backoff and jitter; 95th percentile latency SLO burn alerts. - Cost overruns from Redis cluster size - Mitigations: Key size audit; compression for large payloads; adaptive TTL by popularity; eviction policy tuning (allkeys‑LFU). - Privacy and PII in logs - Mitigations: Field‑level anonymization; sampling; data retention policy; access controls; audit logs. ## 9) Validation and Guardrails - Load Testing: Generated realistic QPS and key distribution (Zipfian). Calculated capacity for p99 under N+1 node failure. - Shadow Traffic: Mirrored 10% of prod queries; confirmed parity of suggestion sets and latency distributions. - Canary: 1% → 5% → 25% per region with automatic rollback on: - p95 > baseline + 15%, error rate >0.5%, hit rate <70%, freshness lag >60 s. - Experiment: 50/50 A/B for two weeks; primary metric CTR on suggestions; guardrail conversion and bounce rate. Small numeric example: Expected latency with cache E[L] = p_hit × L_cache + (1 − p_hit) × L_origin - Before: p_hit = 0.65, L_cache = 15 ms, L_origin = 500 ms → E[L] ≈ 0.65×15 + 0.35×500 ≈ 9.75 + 175 = 184.75 ms (but high tail due to thundering herd). - After: p_hit = 0.88, L_cache = 8 ms, L_origin = 220 ms → E[L] ≈ 0.88×8 + 0.12×220 ≈ 7.04 + 26.4 = 33.44 ms; tail controlled by coalescing and backpressure. ## 10) Measurable Outcomes - Latency: p95 from 450 ms → 180 ms (−60%); p99 from 900 ms → 340 ms (−62%). - Reliability: Error rate from 0.9% → 0.2%; SLO 99.97% over last 90 days. - Relevance: Suggestion CTR +7.1%; dwell time +4.3%. - Business: Search‑to‑booking conversion +0.62 pp; modeled revenue +$8.3M annualized. - Cost: Redis spend up +$12k/mo, but origin DB and compute down −$41k/mo → net −22% infra cost. ## 11) Post‑Launch Learnings - Request coalescing eliminated stampedes but introduced head‑of‑line blocking for a few long‑tail keys; fixed by setting a coalescing timeout and partial results fallback. - Adaptive TTL by popularity cut cost further without hurting freshness. - Early alerting on Kafka consumer lag prevented regional freshness regressions. - Documentation and runbooks reduced on-call pages by ~30%. ## 12) Deep‑Dive Artifacts (Representative) - Design doc sections - Problem statement, SLOs, traffic patterns, proposed architecture diagrams, decision log, failure modes (FMEA), rollout plan, and observability. - Key PRs - PR#1421: gRPC API and protobuf definition; PR#1478: single‑flight coalescing; PR#1510: Kafka consumer with idempotency keys; PR#1542: circuit breakers and token bucket; PR#1567: canary + automated rollback. - Dashboards and Alerts - RED/USE dashboards per region, SLO burn rate alerts (2%/1h, 5%/6h), Kafka lag alerts, cache hit rate monitor. ## 13) What I’d Do Next - Move popular keys to a small in‑memory cache at edge to shave another ~5–8 ms. - Explore approximate nearest neighbor (ANN) for semantic suggestions in long‑tail queries with a separate recall tier. - Multi‑armed bandit for suggestion ordering to accelerate learning without large experiment tax. # Pitfalls to Anticipate in Interview Q&A - Trade‑offs: Be explicit about why you accepted eventual consistency and how you bounded impact. - Edge cases: Hot keys, regional failover, partial outages, replayed events, and GDPR deletion requests. - Validation: Show you measured both system metrics and business outcomes, and guarded against Simpson’s paradox by segmenting traffic (e.g., new vs. returning users, region). - Ownership: Call out a tough decision you made (e.g., cutting scope to hit SLO) and how you aligned stakeholders. Use this structure with your own project details; keep numbers, dates, and artifacts concrete so the interviewer can probe.