Describe a flagship project’s architecture and tradeoffs

How to structure your answer (STAR++): - Situation: 1–2 sentences on business context and problem. - Task: Your objectives, constraints, success criteria (SLOs/KPIs). - Actions: What you personally designed/built/led (architecture, code, process). - Results: Quantified outcomes, validation method (A/B, canary, backtests). - Evidence: Dashboards, logs, customer feedback, incident trends. - Reflection: Tradeoffs, what you’d change, lessons for future projects. Fill-in template (copy and adapt): - Business problem: [Who was affected], [what metric or risk], [why now]. - My role: [Title/scope], [team size], [partners], [what I owned end-to-end]. - Architecture: Clients → [Gateway] → [Service A] → [Cache/DB]; [Event bus] for [events]; [Observability]. - Data flows: - Read: [cache tiers, fallback], [timeouts/retries], [coalescing]. - Write: [transactions], [events/CDC], [idempotency]. - Implementation details: [key schema], [TTL/consistency], [API contracts], [testing], [deploy strategy]. - Challenges and diagnosis: [symptom] → [hypothesis] → [data/trace/logs] → [root cause]. - Resolutions and tradeoffs: [solution], [alternatives], [why chosen under constraints]. - Metrics moved: Baseline → Outcome; guardrails (e.g., cost, error rate, support tickets). - What I’d do differently: [process/tech/validation improvements]. Sample answer (illustrative): Real-time Availability Service to Reduce Overbooking and Latency - Business problem: Our booking flow had high p95 latency (~420 ms) and occasional overbookings (~0.30% of booking attempts) during traffic spikes. This hurt conversion (~−0.7% relative) and generated costly support tickets. - My role: Senior IC leading a 5-engineer effort. I owned the Availability Service redesign: API contracts, cache strategy, event-driven invalidation, and rollout. Partners: Reservations, Search, SRE, Data Science. Architecture (high level): - Clients (web/mobile) → API Gateway → Availability Service (stateless, autoscaled) - L1 near-cache (in-process, 60s TTL, request coalescing) and L2 distributed cache (Redis cluster, 8 shards, 10m TTL with jitter) - Source of truth: Reservations DB (PostgreSQL) - Event stream: Kafka topics for ReservationCreated/Updated/Cancelled (via CDC) - Invalidation worker: consumes events, updates/invalidate cache keys - Observability: Tracing (OpenTelemetry), metrics (Prometheus/Grafana), logs (ELK), SLOs (latency, correctness/freshness) Key data flows: - Read path: Check L1 → miss → L2 (Redis) → miss → compute from DB/materialized view → write-back to caches → return. - Write path: Reservation commits in Postgres → CDC (Debezium) → Kafka → invalidation worker updates affected keys (idempotent, partitioned by listing_id) → metrics emit freshness lag. Implementation details: - Data model: For each listing_id and month, store a compact bitmap of booked days: key avail:v3:{listing_id}:{yyyy-mm}. - Querying: For date range, load monthly bitmaps and bitwise-AND to test availability; supports O(1) check per day. - Consistency/freshness: - L1 TTL 60s; L2 TTL 10m with ±20% jitter to prevent herds. - Stale-while-revalidate for hot keys; background refresh for top N listings. - Event-driven invalidation on reservation changes; overlap-aware update for affected days. - Contention and correctness: - Unique constraint on (listing_id, day) in reservations table prevents double-book writes. - Idempotency key on booking API (24h retention) to avoid duplicate submissions. - Request coalescing (singleflight) to collapse identical cache-miss requests per host. - Resilience: Circuit breakers to DB, exponential backoff on cache misses, canary deploys with 5% traffic and auto-rollback. - Testing: Property-based tests for overlap logic; chaos tests on cache outages; load tests to p99 under event spikes; replay of 7 days of prod traffic in staging. Main challenges and how I diagnosed/resolved them: 1) Overbooking incidents during spikes - Diagnosis: Traces showed invalidation lag up to 2–3s when Kafka consumer lag spiked; bookings created in that window read stale cache. - Resolution: Switched from app-emitted events to CDC (Debezium) to reduce event loss; increased consumer parallelism, added partitioning by listing_id to preserve ordering, and implemented watermark alerting when freshness_lag > 500 ms. Also shortened L1 TTL to 30s for hot keys and performed targeted update instead of full invalidation. - Tradeoff: Higher Kafka and cache CPU costs (+8%) for stronger freshness guarantees. 2) Hot keys and thundering herd on popular listings - Diagnosis: Redis shard CPU >85%, spikes aligned with TTL expirations; cache-miss storms visible in logs. - Resolution: Added TTL jitter, request coalescing, and pre-warming for top 10k listings; expanded Redis to 8 shards with 1024 virtual nodes; enabled client-side batching/pipelining. - Tradeoff: More memory footprint (~+10%) and operational complexity vs. materially lower p95 latency. 3) Booking correctness across time zones and DST - Diagnosis: Support tickets clustered around DST transitions; logs showed off-by-one-day checks for cross-midnight stays. - Resolution: Normalized to UTC in persistence and keying; client-side conversion only for display; added property-based tests around DST and leap days. - Tradeoff: Minor migration complexity; large correctness gains. Metrics and validation: - Primary: Availability API p95 latency 420 ms → 95 ms; p99 1100 ms → 220 ms. - Correctness: Overbooking incidents 0.30% → 0.02% of attempts; 60 consecutive days without a confirmed double-book. - Business: Checkout-to-booking conversion +0.9% relative (A/B 50:50, 2 weeks, p<0.05). - System: L2 cache hit rate 62% → 92%; DB read QPS −55% on availability tables. - Guardrails: Support tickets −18%, infra cost +12% (more Redis shards and consumers), error budget burn reduced below SLO (p95<120 ms, freshness lag <500 ms). - Validation: Canary by region (5% → 25% → 50% → 100% over 4 days), feature flag kill-switch, holdout monitoring for cancellations and customer contacts. Tradeoffs considered: - TTL-only caching vs. event-driven invalidation: chose hybrid (event-driven + bounded TTL) for better freshness without DB overload. - Strong consistency (DB-only checks) vs. eventual consistency with caches: chose eventual with strict write constraints (unique index, idempotency) to prevent double-book writes. - Precomputed monthly bitmaps vs. on-the-fly range scans: chose bitmaps for lower tail latency at the cost of moderate memory. - CDC vs. app-level events: chose CDC for reliability and ordering; added schema versioning to avoid consumer breakage. What I’d do differently: - Define SLOs and error budgets up front in the PRD to align decisions early. - Use CDC from day one instead of app-level events. - Run a formal failure-mode analysis (game days) before full rollout. - Expand property-based tests to include concurrency scenarios and rapid-flip events. Common pitfalls to avoid in your interview answer: - Vague ownership: explicitly state what you designed/built/decided. - No numbers: quantify baselines, deltas, and confidence. - Skipping validation: mention canaries, A/B, and guardrails. - Over-indexing on success: include a real challenge and how you debugged it. Quick prep checklist: - One-page architecture sketch (in your notes) with components and data paths. - 3–5 quantified metrics (baseline → outcome) and 2 guardrails. - 2 hard challenges with diagnosis steps and the data you used. - 1–2 tradeoffs you considered and why your choice fit constraints. - A thoughtful "what I’d change" to show learning and maturity.