Describe Past Project And Debugging Approach

# How to structure a strong answer Use a STAR-style narrative but go deeper on debugging and observability: - Situation: Project goal, constraints, and your role - Task: What you needed to achieve and the reliability targets - Actions: Design/build/run, then deep dive on the production bug (hypotheses → instrumentation → evidence → root cause → fix) - Results: Business outcomes, metrics, and learnings - Reflection: Trade-offs, what you'd change, how feedback propelled growth Time-box to 6–8 minutes. Emphasize decisions, evidence, and verification. # Model answer (software engineering example) Situation - I led an end-to-end build of a real-time payments ledger that recorded card transactions and produced daily reconciliation files for Finance. Scope included service design, data model (immutable ledger entries), integration with our payment processor, rollout plan, and SLOs: 99.95% success rate, p95 latency < 300 ms, and zero duplicate ledger entries. - Constraints: high-throughput (15k TPS peak), at-least-once delivery from our event bus, strict PII controls, and a regulatory deadline. - Stakeholders: Payments product, Finance, Risk, SRE, and Compliance. I was the tech lead and primary on-call for launch week. Production bug context - On day 2 of GA, alerts fired: p95 latency spiked to ~2.3s and Finance flagged 52 duplicate ledger entries out of ~16,700 transactions (~0.31%), breaching “zero duplicate” policy. - Detection: Alert on errors > 0.1% and a custom metric for duplicate candidate rate. Customer support also reported a few duplicate charges. - Impact: Potential double-charge and reconciliation breaks; medium severity with high reputational risk. Debugging approach 1) Initial hypotheses (prioritized by likelihood × impact) - H1: Retry path dropped idempotency keys, causing processor to treat retries as new charges. - H2: Database uniqueness constraints insufficient (missing composite index), allowing concurrent inserts. - H3: Event replays from the bus caused duplicate processing without deduping. - H4: Clock skew or network timeouts triggered out-of-order retries. 2) Instrumentation and probes - Added temporary structured logs at WARN level with correlation_id, trace_id, idempotency_key, and request_attempt. - Enabled distributed tracing for payment → ledger → notifications (OpenTelemetry) with 100% sampling for error/slow paths. - Added metrics: duplicate_candidate_count, db_conflict_rate, consumer_lag, and PSP 4xx/5xx split. - Built a focused Grafana dashboard and increased alert sensitivity for duplicate_candidate_count. 3) Evidence from logs/metrics/traces - Traces showed a subset of requests where our client timed out at 1s, retried, and the second attempt lacked the original idempotency_key in the downstream call. - Logs confirmed request_attempt=2 often had idempotency_key=null from a fallback code path. - DB showed no conflicts; our unique index was on (ledger_entry_id) but not on (merchant_id, customer_id, external_order_id) or idempotency_key. - PSP logs showed two successful charges with different ids for the same business transaction. 4) Reproduction in staging - Injected latency and intermittent timeouts with Toxiproxy. Confirmed that retries via the fallback code path dropped the idempotency header. Also verified we could concurrently produce two inserts without a conflict on our current index. 5) Root cause - A retry helper in our integration client rebuilt requests but failed to propagate idempotency_key. Combined with missing composite uniqueness constraints in our ledger table, duplicates were both created upstream and recorded downstream. Fix and verification 1) Code/config/data changes - Fix: Always propagate idempotency_key across retries and across the event pipeline. - Persistence: Added a unique index on (idempotency_key) and, as a defense-in-depth fallback, on (merchant_id, external_order_id) in the ledger table. - Messaging: Implemented dedup in the consumer using an upsert by idempotency_key (INSERT … ON CONFLICT DO NOTHING). - Retries: Switched to capped exponential backoff with jitter; stop retrying on processor-confirmed timeouts if idempotency key is present. - Observability: Kept key logs with PII redaction; added histogram metrics for retry_attempts and PSP latency. 2) Tests added - Unit tests for retry helper ensuring headers and idempotency are preserved. - Integration tests simulating timeouts, partial failures, and consumer restarts; property-based test to assert no duplicates even under concurrency. - Fault injection test suite using Toxiproxy in CI nightly. 3) Release strategy - Feature-flagged retry helper changes; deployed canary to 5% traffic. - Runbook and rollback plan in place; SRE on bridge. 4) Verification criteria - Success metrics: duplicate_candidate_count returns to zero; p95 latency < 300 ms; error rate < 0.1%; no new support tickets for duplicates. - Data reconciliation: Queried last 24h for count(idempotency_key) > 1; before fix: 52; after fix/canary: 0 over 2 hours; post full rollout: 0 over 24 hours. - Business: Finance confirmed clean reconciliation; refunded affected users and issued incident comms. Results - Resolved duplicates to zero and restored SLOs within 6 hours. No regressions over the following week. - Incident postmortem published; time-to-detect improved via new alerting; runbook and dashboards adopted team-wide. Trade-offs and alternatives - Throughput vs. safety: Unique constraints can add contention under peak load. We accepted a ~2–3% write latency increase for strong dedup guarantees; avoided “exactly-once” message-broker transactions due to operational complexity. - Logging cost vs. visibility: Scoped high-cardinality logs to error/slow paths and redacted PII; used sampling elsewhere to control cost. - Centralized idempotency service vs. local constraints: Considered a shared idempotency service but deferred to a local database approach to meet the regulatory deadline; documented an ADR for a future evolution. What I’d do differently next time - Shift-left on reliability: Define idempotency and retry semantics in the design doc with contract tests before integration; run a pre-mortem for failure injection scenarios. - Observability from day one: Mandate correlation_id and idempotency_key in all logs/traces, and create SLO dashboards pre-GA. - Data model guardrails: Start with the composite unique index and upsert pattern; automate DB schema checks in CI. Feedback and growth - I asked SRE and Finance for feedback on incident leadership. Actions from that: - Facilitated a blameless postmortem and incident review; improved comms cadence during incidents. - Created a reliability checklist (idempotency, retries, backoff, unique indexes, tracing) adopted by two other teams. - Mentored two engineers to lead the next reliability project and shared a brown-bag talk on fault injection. - Measurable growth: Reduced mean time to detect by 40% and cut duplicate issues to zero over the next quarter. # Guardrails and pitfalls to mention during interviews - Never log PII; use correlation ids and redaction. Keep debug logging targeted and time-bounded. - Validate in a lower environment with fault injection; avoid reproducing risky scenarios in prod. - Prefer idempotent operations with unique constraints/upserts over complex “exactly-once” guarantees. - Define clear success criteria and rollback triggers for any hotfix. # Quick template you can adapt - Project: What it is, scale, SLA/SLOs, your role. - Bug: Symptoms, impact, detection timeline. - Hypotheses: Top 3 and why you prioritized them. - Instrumentation: Logs/metrics/traces you added and why. - Evidence: Specific signals that confirmed/ruled out hypotheses. - Fix: Code/data/config changes; tests; rollout plan. - Verify: Metrics, canary, reconciliation, and user validation. - Trade-offs: Two alternatives and why you chose your path. - Growth: Feedback you sought and the habits you changed.