Walk through a project deep dive

# How to Structure a Strong 30–40 Minute Answer Suggested time split: - 3–4 min: Problem context and goals - 6–8 min: Architecture and technical approach - 5–6 min: Key decisions and trade-offs - 3–4 min: Timeline and constraints - 4–5 min: Metrics and impact - 6–8 min: Biggest challenge deep dive (diagnosis, fix, alternatives) - 3–4 min: Lessons learned - 3–4 min: Mentorship example Checklist for each section: - Be explicit about baselines and targets. - Separate team vs. your contributions. - Name alternatives and why you didn’t choose them. - Quantify outcomes (latency, availability, cost, adoption, dev velocity). - Translate technical wins into business/user value. --- ## Model Deep Dive: Centralized Authorization Service (AuthZ) for a Multi‑Tenant Platform This example shows depth, cross-team collaboration, design decisions, measurable impact, and a real challenge with a principled fix. ### 1) Problem Context and Goals - Context: 15+ microservices each implemented authorization differently (duplicated logic, inconsistent policy enforcement, no centralized audit). Compliance and least‑privilege requirements were increasing. - Pain: Authorization bugs caused 3 Sev-2 incidents/quarter; new features took longer due to bespoke auth. No audit trail to answer “who had access to what and when?” - Goals: - Unify authorization with a central service and policy layer. - P95 check latency ≤ 5 ms; availability ≥ 99.99%. - Near-real-time revocations (P95 ≤ 2 s from policy change to effect). - Complete, queryable audit logs for compliance. ### 2) My Role and Responsibilities - Role: Tech lead and primary IC (team of 3 engineers + 1 part-time SRE). - Responsibilities: - Drove discovery with security and service teams; defined API and SLAs. - Designed architecture, caching strategy, and data model. - Implemented policy evaluation service and cache invalidation mechanism. - Ran design reviews, rollout plan, and on-call readiness; owned migration tooling. ### 3) Technical Approach and Architecture - Interfaces: - Check(user, action, resource): boolean with reason codes. - ListPermissions(user): precomputed effective permissions (for UIs). - DryRun(request): returns decision + “what policy matched” (safe migrations). - Explain(decisionId): retrieves audit trail for a decision. - Policy Layer: - Adopted a hybrid RBAC + ABAC model (roles for common cases; attributes for finer control like tenant, department, data sensitivity). - Policies authored in a constrained DSL (OPA/Rego bundles) to balance expressiveness and safety. - Data Flow: - Source of truth: Postgres for role bindings and attributes. - Change log: CDC to Kafka topics. - Read path: Central AuthZ service with: - Hot data in Redis cluster (materialized effective permissions). - Per-instance LRU cache for microsecond lookups. - Invalidation: Permission updates publish to Kafka; consumers update Redis and push versioned invalidation messages to service instances. - Availability and performance: - Multi-AZ deployment behind Envoy; stateless service; autoscaling on QPS. - Fail-closed logic with nuanced allowlist for health checks/admin break-glass. - Idempotent writes and at-least-once event handlers for change propagation. ### 4) Key Decisions and Trade-offs - Centralized service vs. embedded library/sidecar: - Chose centralized service for consistent policy, unified audit, and easy rollout. - Trade-off: extra network hop. Mitigated with Redis + per-instance LRU caches and multi-AZ placement. - RBAC vs. ABAC vs. custom DSL: - Chose hybrid RBAC+ABAC with OPA bundles; avoided full custom DSL cost while keeping auditability and static analysis. - Cache consistency vs. revocation speed: - L1/L2 caches reduced latency but risked staleness. Chose versioned invalidations + short TTLs for high-risk resources. - Precompute effective permissions vs. compute on the fly: - Precomputed for ListPermissions (UI) to keep UI snappy; compute-on-demand for Check to save memory. - Dry-run mode: - Added to enable safe migrations and test policy changes before enforcement. ### 5) Timeline and Constraints - Quarter 1: Discovery, policy model POC, API design, security review; onboard 2 pilot services. - Quarter 2: Build, load test, runbooks, rollout to 10 additional services; migration tool for role mapping. - Constraints: Legacy services needed backward-compatible semantics; strict compliance requirements; limited infra budget (reuse managed Redis/Postgres; no exotic infra). ### 6) Metrics and Impact - Latency: P50 1.8 ms; P95 3.7 ms per Check. - Availability: 99.995% over first full quarter (no Sev-1/2 incidents related to AuthZ). - Revocation time: P95 480 ms from change to enforced decision; P99 1.7 s. - Developer velocity: Average PR cycle time for auth-related changes dropped 35%; net new features avoided bespoke auth code, reducing defects found in security review by 60%. - Compliance: Complete decision audit logs enabled faster evidence gathering (from days to minutes) in audits. ### 7) Biggest Challenge Deep Dive - What: Cache staleness caused rare but critical delays in revoking permissions after role changes, especially cross-AZ during Kafka consumer failover. - Why hard: Three layers of caching (per-instance LRU, Redis, client retries) plus eventual consistency on CDC meant multiple failure modes; reproducing timing-sensitive races required careful instrumentation. - Diagnosis: - Added request tracing with decision IDs, cache hit/miss tags, and user “permission epoch” to correlate decisions over time. - Built a chaos test that simulated message delay/duplication and AZ failover; reproduced stale decisions in ~1/200k revocations. - Resolution: - Introduced per-principal “revocation epoch” incremented on any permission change. - Embedded the current epoch in the auth token and stored it alongside cache entries. - Check path logic: if token.epoch > cache.epoch, bypass caches and fetch fresh from source; then update caches. - Added a short max-TTL guard (fail closed after N seconds if epoch mismatch persists) with an admin break-glass route. - Improved invalidation channel with exactly-once delivery semantics at the app layer (idempotent versioned messages). - Alternatives considered: - Disable L1 cache: too much latency and cost under peak QPS. - Ultra-short global TTLs: reduced staleness but spiked Redis load and tail latencies. - Sidecar OPA per service: great locality but lost centralized audit and raised rollout complexity. - Chosen approach balanced security (monotonic revocations) with performance and operability. - Impact of the fix: - Revocation correctness issues dropped to 0 in the following two quarters; P95 revocation time held under 500 ms even during failovers. - No need to overprovision Redis; kept infra cost flat. ### 8) Lessons Learned - Design for revocations explicitly; TTLs alone are not a consistency strategy. - Dry-run and explainability endpoints de-risk migrations and speed up incident response. - Decide early where you want observability: centralized services make audit and SLOs simpler. - Build a constrained policy model first; expand expressiveness only with clear use cases. --- ## Mentorship Example (Intern: Dry-Run API and Explain UI) - Goals (set in week 1): - Deliver a minimal DryRun API + simple Explain UI by end of week 4; production-ready by week 8. - Quality bar: >90% unit coverage for policy evaluation; integration tests in CI; P95 added latency ≤ 1 ms on Check when logging explain traces is disabled. - Guidance provided: - Onboarding guide (architecture doc, sample traces, test data) and a 90-minute design/pairing session to shape the API. - Weekly 1:1s for goal review; daily async check-ins; paired on the first integration test and the first PR. - Code reviews focused on testability, error handling, and performance; shared example PRs as references. - Measuring progress: - Sprint board with clearly defined acceptance criteria; tracked PR cycle time and test pass rates. - Mid-internship demo to 3 consumer teams; captured feedback and adjusted scope. - Outcome: - Shipped DryRun + Explain; adopted by 8 services within a quarter; cut policy migration incidents to zero. - Intern converted to full-time; they authored runbooks and added synthetic checks to CI. - What I’d do differently: - Pre-schedule stakeholder demos earlier to lock API needs; narrow initial UI scope to reduce context switching. --- ## Adapting This to Your Own Story - If you don’t have central-platform work, pick any project with clear stakes (performance, reliability, privacy/security, cost, or major user experience win). - Always quantify before/after. If you lack precise numbers, provide ranges and describe how you’d measure them now. - Bring one simple diagram (even verbal) and 2–3 core metrics. - Preemptively call out 2 alternatives you considered and why they lost. ## Guardrails - Avoid proprietary names; describe capabilities, not secrets. - If you degraded service to fail closed, explain safety nets (break-glass, rate limits, runbooks). - Validate claims: tie metrics to logs/dashboards and link them to user/business impact (e.g., audit pass, incident reduction, faster feature delivery).