Summarize background, challenge project, and failure

# 1) Career Narrative I am a backend-focused software engineer with ~8 years of experience building distributed systems, data platforms, and developer-facing services. I’ve led projects that required high availability, strong/causal consistency semantics, and predictable performance under multi-tenant load. - Roles: Backend Engineer → Senior Engineer → Tech Lead. - Domains: Distributed storage/indexing, stream processing, cloud infrastructure, and reliability engineering. - Notable achievements: - Designed and shipped an online index build feature for a large, multi-tenant document store, enabling zero-downtime index creation with 99.99% availability and <5% p99 latency regression during builds. - Led a cross-region replication improvement that reduced RPO from minutes to <5 seconds and increased failover reliability via improved fencing and lag-aware elections. - Drove an observability revamp (RED + USE metrics, SLOs, canaries) that cut time-to-detect by 60% and time-to-mitigate by 45%. # 2) Deep-Dive: Most Challenging Project — Online Index Build for a Multi-Tenant Document Store ## a) Problem & Scope - Problem: Customers needed to add secondary indexes to large collections (TB-scale) without taking write downtime or risking inconsistent reads. Prior process required maintenance windows and caused unpredictable tail latencies. - My role: Tech Lead and primary designer/implementer. Owned end-to-end design, rollout plan, and coordination with SRE and query engine teams. - Constraints: - Availability: 99.99% during build (no write pauses >100 ms beyond baseline). - Consistency: New index must be logically complete at activation; no missing entries for writes during the build. - Scale: Multi-tenant; collections up to 10 TB, billions of docs, highly skewed key distributions. - Resource isolation: Avoid noisy-neighbor effects; protect query latency SLOs (p99 < 50 ms read, p99 < 80 ms write). - Time-to-ready: Reasonable build time (target: <= 24 hours for 10 TB with backfill throttling). ## b) Options & Trade-offs Considered 1) Offline build with maintenance window - Pros: Simplest; faster backfill; no dual-write complexity. - Cons: Downtime; unacceptable for customers. 2) Read-only mode during build - Pros: Simplifies correctness guarantees. - Cons: Blocks writes; still unacceptable. 3) Online build using snapshot + change-capture (chosen) - Pros: Zero write downtime; correctness via snapshot plus catch-up on changes. - Cons: More complex; requires dual-write or log tailing; careful throttling. Key trade-offs: - Consistency vs. throughput: Aggressive backfill risks lagging behind live writes; we chose correctness-first, with adaptive throttling. - Complexity vs. operational risk: Introduced explicit build states and idempotent write paths to reduce blast radius. - Centralized coordination vs. fully decentralized: Chose shard-local workers plus a lightweight coordinator for global progress to avoid single points of contention. ## c) Final Approach & Architecture We used a three-phase online build with MVCC-based snapshotting and change capture. Build phases: 1) Snapshot/backfill: Scan a stable snapshot of collection data and populate the new index. 2) Catch-up: Apply changes that occurred during backfill by consuming the write-ahead log (WAL) or oplog, ensuring no gaps. 3) Activation: Atomically mark index as ready and route queries to use it. Components: - Index Coordinator: Tracks build state (INIT → BACKFILL → CATCHUP → READY), holds checkpoints, enforces concurrency limits, and orchestrates shard progress. Backed by a consensus metadata store (Raft) to ensure a single leader and durable state. - Shard Workers: Per-shard executors doing backfill scans and applying change events. Implement idempotent upserts into the index. - Change Stream Tailer: Reads WAL/oplog from each shard starting at the snapshot timestamp to feed catch-up. - Throttler & Governor: Enforces IOPS/CPU/QPS budgets per tenant and cluster-wide. Adaptive based on tail latency and replica lag. - Index Storage Engine: LSM-backed structure for fast writes and compactions (chose LSM over B-Tree for better write amplification under heavy backfill and catch-up). Data flows: - Backfill: Range-scan by primary key with MVCC at snapshot_ts → transform doc → index key(s) → idempotent write to index store. - Dual-path for writes during build: Live writes emit change records (insert/update/delete) captured by tailer; workers apply them to the building index. - Activation: Two-phase protocol per shard: (prepare) fence at log position L, apply all changes ≤ L, fsync metadata; (commit) flip index visibility bit atomically. Coordinator requires quorum acks before global READY. Key algorithms/patterns: - Snapshot isolation using MVCC timestamps. - Idempotent index writes keyed by (index_key, doc_id) to handle retries. - Backpressure: Pause/reduce backfill when p99 latency or replica lag exceeds thresholds. - Checkpointing: Persist progress every N MB or M seconds; resume after failure without re-scanning. Technology choices: - Language: Go for systems-level concurrency and tooling consistency. - Storage: LSM-based engine (RocksDB-like) for index segments; compression enabled; tuned compaction strategy for long runs. - Coordination: Raft-backed metadata service for index state and leases. - RPC: gRPC for coordinator/worker control plane. - Telemetry: OpenTelemetry, Prometheus, and distributed traces. Capacity planning (simplified): - If D = data size (bytes), r = sustainable backfill read rate (bytes/s), w = index write throughput (bytes/s), and α = throttling factor (0–1), then backfill time T ≈ D / min(α·r, α·w). - Example: D = 10 TB, r = 250 MB/s per shard × 4 shards = 1 GB/s, w = 800 MB/s, α = 0.3 (to protect SLOs) → T ≈ 10,240 GB / min(0.3·1, 0.3·0.8) GB/s = 10,240 / 0.24 ≈ 42,667 s ≈ 11.8 hours. ## d) Validation, Risks, and Metrics Validation: - Correctness: Shadow index verification (sampled point lookups compare query plans with/without index), checksum of index key cardinality, replay of synthetic change streams with known ground truth. - Performance: Load tests with realistic skew (Zipfian k≈1.2), chaos tests for worker restarts, tailer hiccups, and coordinator failover. - Rollout: Feature flag per tenant; canary on 1% of collections; gradual ramp with automatic rollback on SLO breach. Risks & mitigations: - Risk: Latency regressions for hot tenants → Mitigation: Per-tenant governors, dynamic throttling from SLO dashboards, separate compaction IO class. - Risk: Incomplete catch-up due to log gaps → Mitigation: Fencing at L, strict monotonic log position checks, alerts on tailing lag. - Risk: Duplicate index entries on retries → Mitigation: Idempotent writes keyed by (index_key, doc_id), and unique constraint at the index layer. - Risk: Coordinator split-brain → Mitigation: Raft quorum and lease checks on every phase transition. Metrics/SLOs: - SLOs: Availability 99.99%; p99 read < 50 ms; p99 write < 80 ms; tailer lag < 5 s; error rate < 0.1%/5 min during build. - Dashboards: Build throughput (docs/s), progress %, tenant-level throttling, compaction backlog, activation success rate. Outcomes: - 0 downtime required; 97% of builds completed within 24 hours at 10 TB scale. - <3% median and <5% p99 latency regression during builds; zero data inconsistencies detected in post-build audits. - Reduced operational toil: index builds no longer required manual maintenance windows, saving ~12 engineer-hours/week. ## e) What I’d Do Differently - Precompute tenant-specific resource envelopes using historical workload models to choose better initial throttling α, reducing convergence time of adaptive throttling. - Earlier chaos experiments on log retention boundary cases to surface tailer gap handling sooner. - Build a declarative scheduler that co-optimizes compactions and backfill IO to minimize interference. # 3) Meaningful Failure - Incident: We rolled out the index build feature to a mid-sized tenant with mixed OLTP/OLAP traffic. I approved a configuration that allowed up to 4 concurrent backfill workers per shard. During peak hours, compaction plus backfill caused an IO spike, and p99 write latency breached SLO for ~18 minutes. - My contribution: I argued that the adaptive throttler would react fast enough and did not cap concurrent workers during business hours. I also missed a signal in pre-production that compaction backlog grows non-linearly under skew. - Root cause: Combined effect of skewed key distribution (hot partitions), insufficient governor protections for compaction IO, and overly aggressive concurrency. The adaptive controller had a dampening window that was too slow (30 s), causing oscillations and prolonged high latency. - Impact: SLO breach for one tenant; autoscaling events and user-visible write latency spikes; no data loss. - What I learned: - Always protect the underlying storage with hard caps and separate IO classes for compaction vs. foreground writes. - Adaptive control loops need guardrails (min/max limits) and faster feedback during ramp-up. - Treat tenant peak windows as no-fly zones for concurrency increases unless proven safe. - How I’ve applied it since: - Added hard concurrency ceilings during business hours and a fast-path latency tripwire that immediately throttles backfill within 2–3 seconds. - Introduced workload-aware scheduling: backfills for hot tenants run in smaller bursts with longer cool-off intervals. - Enhanced pre-prod load testing with skew injections and compaction stress scenarios. This experience strengthened my instincts for correctness-first designs, conservative rollouts, and explicit resource isolation—habits I bring to complex, multi-tenant systems where small misconfigurations can have outsized effects.