Present project and answer behaviorals

# Slide 1 — Title & TL;DR - Project: Real-time Safety Guardrails for a Generative API - Goal: Add pre- and post-generation safety checks with <50 ms P95 latency overhead and ≥95% recall on sensitive categories (e.g., self-harm, hate, PII), to reduce safety incidents and enable enterprise adoption. - Impact: 99.95% availability, 32 ms P95 overhead, 96% recall / 99.6% precision on policy-aligned eval set, 84% reduction in safety incidents per million requests. - My role: Tech Lead (5 engineers, 10 weeks), design/implementation of safety service, thresholds, streaming checks, rollout, and on-call readiness. # Slide 2 — Problem & Constraints - Problem: Customers need robust safety guardrails on both prompts and model outputs without noticeable latency or broken streaming UX. - Why now: Increasing enterprise demand; prior incidents and manual moderation didn’t scale. - Constraints - Latency: +50 ms max P95 overhead; must preserve streaming. - Throughput: 7–10k QPS peak, multi-region, 99.9%+ availability. - Accuracy: High recall on disallowed categories with minimal false positives. - Compliance: Auditability, per-tenant policies, GDPR retention. - Team/time: 5 engineers, 10 weeks, shared on-call. # Slide 3 — Architecture (High Level) - Request path 1) Client → API Gateway → Safety Service (pre-check prompt): rules engine + fast classifier → allow/flag/block + risk scores. 2) If allow → LLM Inference Service → streaming tokens. 3) Post-check (streaming): tokens mirrored to streaming safety filter; if violation → truncate stream, return safe completion or refusal template. - Async path - Flagged events → Kafka → Async rescoring (heavier model) → moderation console → policy updates. - Storage/infra - Config in Postgres (policy-as-code), Redis for hot config + rate limiting, object storage for eval sets and logs. - Observability: OpenTelemetry tracing, SLO dashboards, error budgets, circuit breakers. - Resilience - Degrade to rules-only if classifier unhealthy; kill switch per tenant; multi-region failover. # Slide 4 — Key Decisions & Rationale - Build vs. buy: Built in-house for custom categories, streaming integration, and lower variable cost; evaluated 2 vendors (added 25–80 ms overhead, limited streaming support). - Model strategy: Hybrid rules + fast ONNX classifier (CPU) for sync path; heavier model async for calibration and appeals. - Language/runtime: Go for Safety Service (low latency, memory safety), Python for model training/offline eval; ONNX Runtime for inference. - Streaming UX: 20–40 ms hold-and-release micro-batching on the safety side to reduce mid-word truncation; token buffer with lookahead to mitigate false triggers. - Policy-as-code: YAML policies versioned in Git with dynamic reload; per-tenant overrides; signed changes for audit. # Slide 5 — Metrics of Success - Reliability/SLOs - Availability ≥99.9%, error budget ≤43 min/month. - P95 overhead ≤50 ms; P99 ≤120 ms; no long-tail spikes during GC. - Safety quality - Recall (sensitivity) ≥95% on curated eval set; precision ≥99%; per-category thresholds. - Incident rate: <1.0 per million requests; time-to-mitigate <15 min. - Business/UX - Block rate <1% overall; false positive rate (FPR) <0.5% for top customers; support tickets ↓30%. - Example definitions - Precision = TP / (TP + FP) - Recall = TP / (TP + FN) - Small example: If out of 10,000 events, 100 are truly disallowed and the system blocks 96 of them (TP=96) and wrongly blocks 4 allowed (FP=4), then precision=96/(96+4)=96%, recall=96/100=96%. # Slide 6 — Results & Validation - Performance: 32 ms P95 overhead (11 ms rules, 18 ms classifier, 3 ms I/O), 118 ms P99; 7.5k QPS sustained; 99.95% availability first quarter. - Safety: 96% recall / 99.6% precision on policy eval; incident rate 0.8 per million requests (↓84%). - UX/Business: False positives 0.4%; support tickets related to safety ↓31%; 3 high-severity incidents prevented via guardrails. - Validation/guardrails - Offline: stratified eval set (multi-language, adversarial prompts), calibration per locale. - Online: 5% canary, shadow evaluation with heavier model, auto-rollback on SLO breach. - Fallback: rules-only mode if ML degrades; per-tenant bypass with audit. # Slide 7 — Trade-offs & Alternatives - Precision vs. recall: Chose higher recall for self-harm/hate; balanced with tiered responses (warn → attenuate → block). - Sync vs. async: Sync path must be fast; heavier reviewers async for continuous improvement and appeal workflows. - Rules vs. ML: Rules offer determinism/audit; ML captures context. Hybrid minimized both false positives and misses. - Streaming complexity: Slight buffering added latency but materially improved UX; tested 20–40 ms windows. - Vendor vs. in-house: Vendor quicker to start but worse streaming/latency and higher per-request costs. # Slide 8 — My Role & Execution - Led end-to-end design, RFCs, and design reviews; co-implemented Safety Service, streaming post-filter, and config system. - Drove SLOs, dashboards, playbooks, and chaos tests; owned canary + rollback automation. - Coordinated with Product, Trust & Safety, Legal, and Support; ran weekly policy councils and AB test reviews. # Behavioral Follow-ups ## (a) Toughest challenge and how I resolved it - Situation: Early streaming checks caused mid-word truncations and false blocks in non-English content. - Action - Added token buffer with 20–40 ms lookahead to avoid premature triggers. - Introduced per-locale thresholds and calibration using Platt scaling on multilingual eval sets. - Deployed language ID to route to locale-specific policies. - Result: Reduced streaming false positives from 1.3% to 0.4%; customer complaints dropped 70% week over week. - Lesson: Streaming moderation needs UX-aware buffering and locale-aware calibration to avoid user-visible churn. ## (b) Stakeholder conflict and alignment - Conflict: Trust & Safety wanted aggressive blocking; Product pushed for minimal friction for enterprise users. - Actions - Built a tiered enforcement model (warn → block) with category-specific thresholds and customer-level risk bands. - Ran AB tests with pre-agreed guardrails and decision review cadence; instrumented FPR/recall per segment. - Clarified success metrics: incidents per million (T&S) and false positives per thousand (Product). - Outcome: Agreed policy with opt-in tiers; sustained 0.4% FPR while keeping recall ≥95% in critical categories. ## (c) A mistake and what I changed afterward - Mistake: Shipped global thresholds without per-locale calibration; Spanish/Portuguese content had 3× higher FPR. - Fixes - Rolled back to per-locale thresholds; added multilingual eval and translation-invariance checks to the CI gate. - Instituted a release checklist: canary in top 5 locales, per-locale metrics must pass before 100% rollout. - Outcome: Localized FPR normalized to ≤0.5% across top languages; no repeat incidents. ## (d) How I raised the bar or mentored others - Mentorship: Coached a junior engineer to own the async rescoring pipeline; paired on design, led their first incident retro. - Raising the bar - Authored a policy-as-code style guide and design review checklist; created a reproducible eval harness and seeded adversarial test cases. - Established on-call runbooks, synthetic monitors, and chaos experiments; reduced MTTR by 40%. # How to adapt this template to your own project - Choose a project with measurable impact and trade-offs. Frame with Situation → Task → Actions → Results. - Quantify performance and business outcomes (latency, availability, cost, adoption, incidents, tickets). - Visualize the architecture as a simple flow; call out resilience, observability, and rollout safety. - Pre-commit to metrics that balance stakeholders; use canaries and auto-rollback. - Be explicit about what you didn’t build and why (scope control shows judgment).