Walk through a recent technical project

# How to structure your answer (1–2 minutes) - Lead with impact: problem, goals, constraints, and why it mattered to the business. - Walk the interviewer through your architecture and key decisions, highlighting trade-offs. - Quantify scaling and reliability with concrete numbers and budgets. - Explain validation, rollout, and how you measured success (including experiment design). - Close with the hardest challenge, your debugging approach, and pragmatic V2 improvements. --- # Sample deep-dive answer: Real-time Feed Ranking Service ## 1) Problem and goals - Problem: Our feed was ranked by simple heuristics, causing low engagement and inconsistent latency due to a monolithic service. We needed a real-time ranking microservice with sustained improvements to CTR and dwell time. - Goals - +5% relative CTR and +3% dwell time within 60 days. - p99 latency ≤ 100 ms for ranking requests; availability ≥ 99.99%. - Peak 6k RPS; handle traffic spikes 2× with graceful degradation. - Cost: ≤ $0.25 per 1k ranking requests. - Constraints - Backward-compatible API with the existing feed aggregator. - Privacy: PII minimization and consent-aware features; regional data residency. - Team: 3 engineers, 1 data scientist; 12-week delivery window. ## 2) Architecture overview Components and flow (request path): - Feed API (existing): Calls Ranker with candidate item IDs and user_id. - Candidate Service: Provides up to 200 candidate items per user (rule-based and recall models). - Feature Service - Online features (Redis): user and item features precomputed via stream jobs; TTL 24 h with jitter. - On-demand join: fills missing features from a columnar store (read-optimized) with a 10 ms budget. - Ranker Service - Batch-scores candidates using a vectorized XGBoost model (ONNX runtime). - Applies light business constraints (e.g., diversity caps, safety filters). - Logging/Telemetry: Impressions and clicks emitted to Kafka, then to data lake for training. Key design decisions and trade-offs - Model serving via ONNX runtime vs. custom: ONNX gave 2–3× speedup, portable, and standardized I/O; avoided framework lock-in. Trade-off: more careful feature typing and conversion. - Online feature store in Redis vs. only on-demand: Redis gave predictable low latency for hot features; trade-off is cache coherency and cost. Mitigated with expiration and streaming upserts. - REST vs. gRPC: Kept REST for compatibility in v1. Trade-off: some serialization overhead vs. faster adoption; plan gRPC in v2. - Consistency: Eventual consistency for features was acceptable given real-time constraints; mitigated with feature version tags and model calibration. ## 3) Data model Core entities (simplified): - users(user_id, locale, consent_flags, created_at) - items(item_id, author_id, created_at, language, topic_tags[]) - online_user_features(user_id, ctr_7d, dwell_avg_7d, last_active_ts, interests_embedding) - online_item_features(item_id, ctr_global_30d, quality_score, freshness, content_embedding) - impressions(user_id, item_id, ts, position, model_version, request_id) - clicks(user_id, item_id, ts, request_id) Indexes and partitioning - impressions/clicks: partitioned by day, clustered by user_id for training joins. - Redis keys: user:features:{user_id}, item:features:{item_id}; sharded by consistent hashing. ## 4) Interfaces and contracts Rank endpoint (REST): - POST /v1/rank - Request: { user_id: string, candidate_item_ids: string[], k: int, request_id: string } - Response: { ranked_items: [{ item_id: string, score: float }], model_version: string, latency_ms: int, request_id: string } Contracts - Idempotent by request_id. Schema versioned via model_version and feature_version. - Timeouts: 80 ms server-side; client retries with exponential backoff; circuit breaker on sustained 5xx. ## 5) Scaling and reliability Traffic and capacity - Peak RPS: 6k; avg RPS: 2k. - Candidates per request: 200 → 1.2M item-scores/sec at peak. - Vectorized XGBoost scoring ≈ 0.01 ms/item → ≈ 12 CPU-sec/sec (~12 vCPU for scoring). With 3× headroom: 36 vCPU across 6 pods. Latency budget (p99 ≤ 100 ms) - Network + gateway: 10–15 ms - Redis feature fetch (batched/pipelined): 15–25 ms - On-demand backfill (rare): ≤ 10 ms - Model scoring (batched): 10–20 ms - Business rules + marshalling: 10–15 ms - Buffer: ≥ 10 ms Techniques - Batching and vectorization for scoring. - Redis pipelining and MGET; hot-key sharding. - Backpressure: bounded queues; drop to a safe fallback (heuristic ranking) when Redis or model is unavailable. - Circuit breakers and jittered retries to prevent cascading failures. - Autoscaling: HPA on CPU and p95 latency; pre-warm pods during known spikes. ## 6) Testing and validation - Unit tests: feature transforms, model I/O schema, ranking rules. - Contract tests: JSON schema and backward compatibility for /v1/rank. - Property-based tests: idempotency and ordering invariants. - Load testing (k6): 2× peak for 30 min; verified p99 and tail behavior under jitter. - Chaos testing: Redis node failover; ensured fallback ranking activated within 1 s and SLOs degraded gracefully. ML/data validation - Offline: AUC/NDCG@K on a time-sliced holdout; calibration via isotonic regression. - Data quality: great_expectations checks for ranges/nulls; feature drift monitors (PSI/KL) with alerts. - Shadow mode: 100% traffic scored by the new ranker, but only logged; compared top-K overlap and score calibration vs. baseline for a week before canary. ## 7) Rollout plan - Phase 0: Shadow traffic and log-only. - Phase 1: 1% canary with kill switch; monitor guardrails (p99, 5xx, CTR, DNU impact). - Phase 2: Ramp 1% → 10% → 50% → 100% over 7 days; automatic rollback if any guardrail breaches thresholds. - Migration: Dual-write impressions/clicks with model_version; dashboards separate cohorts by version to avoid metric mixing. ## 8) Measuring success Primary KPIs - CTR +5% relative; dwell time +3% relative. - Reliability: p99 ≤ 100 ms; availability ≥ 99.99%. - Cost: ≤ $0.25 per 1k requests. Experiment design (two-proportion test for CTR) - Baseline CTR p = 0.05; target +5% relative → 0.0525 (δ = 0.0025 absolute). - Sample size per arm (α = 0.05, power = 0.8): n ≈ 2 · (Z_{1-α/2} + Z_{1-β})^2 · p(1−p) / δ^2 With Z_{1-α/2}=1.96 and Z_{1-β}=0.84: n ≈ 2 · (2.8)^2 · 0.05·0.95 / 0.0025^2 ≈ 119,000 users per arm. - Guardrails: p99 latency, 5xx rate, session starts, and content policy thresholds. Outcome - Achieved +6.2% CTR and +3.8% dwell time at 100% rollout; p99 78–85 ms; availability 99.995%. Cost ~$0.19 per 1k requests. ## 9) Toughest technical challenge and debugging Symptom - Tail latency spikes (p99 → 160–220 ms) during Redis node failover and traffic bursts. Investigation - Added distributed tracing (request_id) to break down time by stage; found feature fetch dominated spikes. - Metrics showed hot-key amplification: a small cohort with many shared features drove >10% of Redis QPS to one shard. - Redis misses caused cascading on-demand backfills; retries without jitter amplified load (retry storm). Fixes - Hot-key sharding and MGET pipelining; introduced client-side microbatching (combine requests within 2 ms). - Single-flight deduplication: coalesce concurrent requests for the same key. - Randomized TTLs (±20%) to avoid synchronized expirations (cache stampede). - Jittered exponential backoff; capped retries; protected with a circuit breaker. - Pre-warm frequently requested features after failover. Result - p99 dropped from 160–220 ms to 75–90 ms during failover tests; steady-state p99 ~80 ms. ## 10) V2 improvements - Migrate Ranker to gRPC with protobuf for lower overhead and stronger contracts. - Unified feature store (e.g., consistent offline/online definitions) to reduce skew. - ANN-based candidate pre-filtering for diversity and relevance with bounded latency. - Multi-armed bandit for exploration to reduce regret while learning faster. - Per-tenant SLOs and cost attribution; resource isolation via dedicated pools. ## Pitfalls and guardrails (generalizable) - Idempotency and dedup: ensure request_id propagates through logging and ranking. - Backward compatibility: additive schema changes; feature_version pinning. - Privacy: data minimization, consent checks at feature compute time, timely deletion flows. - Observability-first: per-stage latency histograms, RED metrics (rate, errors, duration), and SLIs tied to SLOs. --- This structure and example demonstrate depth in problem framing, architecture, data modeling, interfaces, scaling, testing/validation, rollout, success measurement, and learning from production challenges—while making trade-offs explicit and quantifying outcomes.