Present a project deep dive

# How to answer (structure + model example) Use a tight, outcome-first narrative: 1–2 slides on context, 2–3 on architecture/design, 1–2 on performance and testing, 1 on results, 1 on incidents/lessons. Lead with a TL;DR. ## Slide-by-slide structure (6–10 slides) - Slide 1: Title + TL;DR (problem, your role, 2–3 quant outcomes) - Slide 2: Problem and goals (targets/SLOs) - Slide 3–4: Architecture and key components - Slide 5: Data model and APIs - Slide 6: Design decisions and trade-offs - Slide 7: Performance/scalability (with simple math) - Slide 8: Testing and rollout - Slide 9: Metrics and outcomes - Slide 10: Incidents and lessons learned Below is a complete model answer you can adapt. # Model project: Real-time Webhook Delivery Platform v2 (multi-tenant) A fintech platform delivering real-time account and transaction updates to thousands of client endpoints with strict reliability and latency goals. ## 1) Problem and goals - Situation: Existing webhook delivery was unreliable during traffic spikes and partner outages. Pain points: - 97.8% success within 60s; p99 latency ≈ 1.2s; duplicate deliveries during retries. - Noisy neighbors: a few tenants spiked traffic and degraded others. - Cost per 1M deliveries was high due to inefficient retries and hot shards. - Goals (6-month target): - Reliability: ≥ 99.95% delivered within 60s; duplicates < 5 per 1M events. - Latency: p99 < 300 ms for partner endpoints responding within 200 ms. - Scale: sustain 25k events/s, burst 50k events/s. - Cost: -30% cost/1M deliveries. - Compliance: data residency (US/EU), signed deliveries, tenant isolation. - My role: Tech lead + primary IC. Wrote the RFC, led design, implemented delivery service and retry scheduler, led rollout, owned on-call. ## 2) Stakeholders and constraints - Stakeholders: Partner/Customer Engineering (integration success), SRE (SLOs, on-call), Security (signing, egress control), Product (feature parity), Finance (costs). - Constraints: - Backward compatible payloads; zero-downtime migration. - At-least-once semantics required; consumer endpoints must be idempotent. - Region/data residency constraints; per-tenant isolation. - Unknown partner rate limits; must implement fair sharing. - 2 quarters; team of 4 engineers + 1 SRE. ## 3) System architecture and key components - Producers: Event pipeline emits normalized domain events (account_linked, transaction_posted, balance_updated). - Event bus: Kafka (MSK) with partitions keyed by account_id to preserve order per account; cross-region replication for DR. - Delivery service (Go): - Consumes events, applies policy (tenant/topic filters), computes idempotency key, signs payload (HMAC-SHA256), and sends POST to tenant URL. - Per-tenant token bucket rate limiter + circuit breakers. - Connection reuse (HTTP/1.1 keep-alive, HTTP/2 where supported). - Retry orchestrator: - Exponential backoff with jitter; retry topics (10s, 1m, 10m, 1h) to avoid tight loops. - Dead-letter queue (DLQ) to S3 + alerting after N attempts. - Idempotency and dedupe: - DynamoDB table with PK = hash(tenant_id + event_id), TTL = 7 days; conditional write to detect duplicates. - Config and tenancy: - Endpoint registry (url, secret, topics, rate limit, region). Tenant-level concurrency budgets. - Observability and control: - Prometheus/Grafana; SLIs: delivery success within 60s, p50/p95/p99 latency, retries, queue depth, DLQ rate. - Feature flags, traffic shadowing, kill switches. ## 4) Data model and APIs - Core schemas: - Event: { event_id, tenant_id, type, created_at, data_hash, payload, size_bytes } - Delivery: { delivery_id, event_id, tenant_id, endpoint_id, attempt, status, http_code, latency_ms, next_attempt_at } - Endpoint: { endpoint_id, tenant_id, url, secret, topics[], rate_limit_rps, region, version, created_at } - Outbound delivery (to tenant): - HTTP POST body: { id, type, created_at, data } - Headers: X-Webhook-Id, X-Request-Id, Idempotency-Key, X-Signature (HMAC over timestamp + body), X-Timestamp. - Management APIs (examples): - POST /v1/webhooks/endpoints { url, secret, topics, rate_limit_rps, region } - PATCH /v1/webhooks/endpoints/:id { ... } - GET /v1/webhooks/deliveries?since=...&status=... - POST /v1/webhooks/_test { endpoint_id } ## 5) Major design decisions and trade-offs - Delivery semantics: At-least-once vs exactly-once - Chosen: At-least-once with strong idempotency guarantees (keyed by tenant_id + event_id). Exactly-once across heterogeneous HTTP targets is brittle and costly. - Event bus: Kafka vs SNS/SQS - Chosen: Kafka for ordered consumption per account and high burst throughput. SNS/SQS simpler ops but weaker ordering and more plumbing for retries. - Retry strategy: Time-wheel vs multiple delay topics - Chosen: Multiple delay topics for operational simplicity and linear cost; bounded backoff with jitter to avoid synchronized retries. - Security: HMAC signatures vs mTLS - Chosen: HMAC-SHA256 by default (simple client integration), optional mTLS for high-security tenants. - Multi-tenant isolation: Global worker pool vs per-tenant budgets - Chosen: Per-tenant rate limits and circuit breakers to prevent noisy-neighbor impact; global guardrails to protect infrastructure. ## 6) Performance and scalability considerations - Throughput target: 25k events/s sustained, 50k burst. Avg payload 2 KB. Peak outbound ~100 MB/s. - Concurrency sizing (Little's Law: concurrency ≈ throughput × latency): - If avg tenant endpoint RTT ≈ 150 ms, concurrency ≈ 25,000 × 0.15 ≈ 3,750 workers. - Add 50% headroom for bursts/GC/network variance → ~5,600 workers. Autoscale on outstanding requests and queue depth. - Partitioning: - 512 Kafka partitions to minimize hot shards and allow parallelism. Key by (tenant_id, account_id) to spread heavy tenants. - Latency controls: - HTTP keep-alive, connection pools, per-tenant dial timeouts, TLS session reuse. - p99 < 300 ms achieved via fast-path signing (pre-hashed payloads) and low GC pressure in Go (object pooling for buffers). - Backpressure and fairness: - Token bucket (burst = 5× steady RPS) and leaky bucket smoothing. Slow-start for newly recovered endpoints. - Cost controls: - Retry caps (e.g., 6 attempts over 24h), DLQ archiving, compression on Kafka, right-sized instances. Avoided double writes to hot stores by batching updates. ## 7) Testing and rollout plan - Testing: - Unit and property-based tests for signing, idempotency key generation, and retry math. - Contract tests against OpenAPI; golden payload tests to ensure canonical JSON serialization before signing. - Integration tests with ephemeral environments; mocked partner endpoints with fault injection (timeouts, 429, 5xx, TLS errors). - Load tests (k6/Locust): stepped ramps to 60k events/s; chaos tests (broker failover, network partitions). - Rollout: - Dark launch: mirror 10% of events to v2, compare delivery logs without calling tenant endpoints (shadow mode). - Canary: enable for 5 pilot tenants, then 10%, 50%, 100% over 2 weeks; automatic rollback if SLO breached for 10 minutes. - Kill switches: per-tenant disable, per-topic disable, global pause. Runbooks documented. ## 8) Metrics and outcomes - Reliability: success within 60s improved from 97.8% → 99.97% (monthly). - Latency: p99 reduced from 1.2s → 240 ms; p50 from 180 ms → 95 ms. - Duplicates: from ~150 per 1M → 3 per 1M deliveries. - Scale: Sustained 30k events/s in production; handled 2× burst during bank outages without customer impact. - Cost: -35% cost per 1M deliveries via better retry policy, batching, and right-sizing. - Ops: On-call pages fell from ~12/month → 2/month; MTTR improved from 38 min → 14 min. ## 9) Notable failures/incidents and mitigations - Hot partition incident: - Symptom: One tenant with large batches caused a single partition to hit 95% CPU and queue buildup. - Root cause: Keying only by tenant_id created skew. - Fix: Composite key (tenant_id, account_id) and repartitioning tool; autoscaling by partition lag. - Signature mismatch with a major tenant: - Symptom: 401s due to HMAC mismatch after JSON field reordering. - Root cause: Non-canonical JSON serialization in one code path. - Fix: Canonicalize serialization; added golden tests; versioned signing (v1, v2) to allow tenant migration. - Retry storm during partner outage: - Symptom: Thundering herd of retries amplifying partner downtime. - Fix: Exponential backoff with jitter, retry caps, circuit breaker per domain, dynamic backoff based on partner health score. ## 10) Lessons learned and what I’d do differently - Define SLOs and error budgets up front; let them drive design and rollout gates. - Bake in tenant isolation early (rate limits, circuit breakers) to avoid noisy-neighbor surprises. - Treat idempotency, canonical serialization, and versioning as first-class to avoid painful migrations. - Shadow traffic and compare at-least-once semantics before cutover; invest in diff tooling. - What I’d change: adopt a unified scheduler (time-wheel) to reduce delay-topic sprawl; push mTLS defaults for high-risk tenants; add per-tenant sandbox tooling earlier for self-serve validation. --- ## Quick reference summary (to copy into slides) - TL;DR: Rebuilt webhook delivery for reliability, latency, and cost at multi-tenant scale; achieved 99.97% on-time delivery, p99 240 ms, -35% cost. - Architecture: Kafka → Go delivery workers → per-tenant rate limiting → retry topics → DLQ; HMAC signatures; DynamoDB idempotency. - Key choices: At-least-once + idempotency; Kafka for ordering; exponential backoff with jitter; per-tenant isolation. - Performance math: concurrency ≈ throughput × latency; 25k/s × 0.15s ≈ 3.75k workers (+50% headroom). - Rollout: dark launch → canary → staged; SLO-gated with automatic rollback. - Results: Reliability +2.1 pp, p99 -960 ms, duplicates -98%, cost -35%, pages -83%. Use this structure with your own project specifics, replacing the example metrics and components with your actual numbers, diagrams, and decisions.