Production Incidents, Observability, And Troubleshooting

What's being tested

Interviewers are checking whether you can triage production incidents quickly and correctly using observability signals, and whether you can implement code-level fixes that prevent recurrence. They want to see practical debugging skills: forming hypotheses from metrics/traces/logs, isolating root cause in a distributed flow, making safe short-term mitigations (rollback / feature flag) and proposing durable fixes. Amazon cares because customer-facing correctness, latency, and availability depend on engineers who can fix faults safely under pressure and improve system observability.

Core knowledge

Observability fundamentals: the three pillars — structured logs, metrics, and distributed tracing — and how they complement each other when investigating failures. Know when to pivot from one to another.
Metric types: counter for event counts, gauge for current values, histogram for latency distributions; compute percentiles from histograms (use `p50`, `p95`, `p99`), watch sampling bias at low traffic.
SLO / Error budget: SLO = acceptable success rate (e.g., 99.9%); $ErrorBudget = 1 - SLO$ ; alerts should map to consuming the error budget, not raw noise.
Distributed tracing: spans, parent-child relationships, and trace_id propagation; use traces to attribute latency to specific services/DB calls. Know head-based vs tail-based sampling tradeoffs.
Correlation ids: instrument requests with a correlation id / `request_id` and include it in logs, traces, and metrics to cross-link signals for a single request.
High-cardinality vs aggregation: high-cardinality tags (user_id, order_id) cause storage and query costs; prefer coarse dimensions (region, endpoint) for alerting and dig deeper with trace/log lookup.
Logging practices: use structured JSON logs, include context fields (`user_id`, `trace_id`), avoid logging PII, use appropriate log levels, and rotate/retain logs with cost limits.
Latency debugging pattern: check service `p50`/`p95`/`p99` across endpoints, inspect traces for long spans, then drill down (DB queries, external APIs, GC, thread contention).
Instrumentation tools: be comfortable with `Prometheus` metrics, `Grafana` dashboards, `Jaeger`/`Zipkin` tracing, and profilers like `pprof` or flamegraphs for CPU/alloc hotspots.
Quick mitigations: feature flags, canary rollbacks, temporary rate-limiting, or circuit breakers; these are engineer-level actions to reduce customer impact before a full fix.
Retries and idempotency: design endpoints to be safe under retries (idempotent or use idempotency keys) and use exponential backoff with jitter to avoid thundering herds.
Root-cause vs band-aid: short-term mitigations must be followed by permanent fixes plus improved observability (new metrics/alerts, dashboards) to detect recurrence.

Worked example — "Diagnose a production latency spike in a user-facing service"

First 30 seconds: ask which endpoints are affected, the SLA (`SLO`) and timeframe, whether a deploy occurred, and whether load changed. Sketch the approach: (1) confirm signal (metrics) and scope (which endpoints/users/regions), (2) isolate layer (service, DB, external call, infra), (3) triage with traces/logs, (4) mitigate, then fix. Start by checking `p50`/`p95`/`p99` on the dashboard to see percentile divergence; if `p99` blew up but `p50` unchanged, focus on tail events (long GC pauses, retries, lock contention). Pull a few failing traces (via `trace_id`) and correlate with logs to see slow DB queries or external API latency. A key tradeoff: temporarily raising trace sampling gives more data but increases overhead; prefer targeted increased sampling for the affected endpoints. Mitigation might be to roll back the recent deploy or enable a feature flag for the change causing the spike. Close by proposing durable fixes (optimize hot DB query, add histogram metrics for the problematic call, add an alert on `p99` latency) and say: "If I had more time, I'd add unit/integration tests that reproduce the latency pattern and run a canary to validate the fix."

A second angle — "Intermittent 500s from a backend service"

Same investigation pattern but with different signals: error rate elevation instead of latency. Ask about traffic patterns, retry behavior, and whether the errors are client-side (`4xx`) or server-side (`5xx`). Start with error-rate metrics and error-aggregation logs to find common exception types, then use traces to find the failing span. Consider upstream downstream dependencies (authentication, third-party APIs) and whether a database migration or schema change coincided with the onset. Short-term mitigations include returning a graceful fallback, enabling a circuit breaker for the failing downstream, or rolling back the change. Important design decision here is whether to treat errors as transient (retries) or stateful (require rollback and data fixes). Also plan post-incident: add an alert that tracks both error rate and user-impacted fraction, and add structured error codes to make future triage faster.

Common pitfalls

Pitfall: Jumping to the most visible signal.
A single metric spike is only a symptom; avoid declaring the `DB` or network as root cause before correlating traces and logs. Instead, state hypotheses and test them quickly with focused trace samples.

Pitfall: Over-instrumenting during an incident.
Adding verbose logging everywhere increases load and can worsen the outage; prefer targeted increased sampling or temporary feature-flagged debug endpoints.

Pitfall: Fixing symptoms without observability improvements.
Applying a rollback or hotfix and closing the incident without adding metrics/alerts that would detect recurrence is a depth mistake; include a remediation checklist (code fix, tests, dashboards, alert thresholds).

Connections

Expect pivots to adjacent topics like release engineering (canary strategies, rollout policies) and SRE runbooks (on-call procedures, paging). Be ready to discuss tradeoffs between observability cost and fidelity (storage/retention vs sampling).