Reliability, Observability, and Incident Diagnostics

What's being tested

Interviewers expect you to demonstrate practical reliability engineering judgment: define measurable service health, design observability that surfaces real faults, and run diagnostics that map telemetry to root causes. They want concrete tradeoffs (cost vs. coverage, signal-to-noise), familiarity with distributed consistency and replication failure modes, and an incident-driven troubleshooting approach. For a Software Engineer role, focus on designing and instrumenting the service, choosing SLI/SLO targets, reasoning about alert thresholds, and proposing safe remediation paths—not organizational or SRE team staffing.

Core knowledge

• SLI / SLO / SLA — an SLI is a measurable signal (e.g., p99 latency); an SLO is a target for that SLI over a window; an SLA is a contractual penalty tied to SLO violations and error budget consumption.

• Error budget math — error_budget = 1 − SLO (e.g., 99.9% availability → error_budget = 0.001); use rolling windows and partition by customer tier when enforcing.

• Latency percentiles — prefer p50/p95/p99 for user-impact; calculate percentiles on aggregated distributions (HDR histograms) to avoid misinterpretation from averages.

• Metrics vs logs vs traces — metrics are numeric time-series for alerting, logs for event detail and forensic search, traces for request causality; instrument with correlation IDs for cross-signal joins.

• Instrumentation primitives — use client-side and server-side timers, structured JSON logs, and distributed context propagation (trace-id, span-id); expose metrics via Prometheus-compatible endpoints or push gateways.

• Alerting strategy — prefer alerting on symptoms with automated noise-reduction (rate-limited, aggregation windows, dynamic baselines); avoid alerting on single-instance internal counters unless they imply customer impact.

• Health checks & readiness — implement separate liveness and readiness probes; readiness gates for in-flight migrations and leader elections prevent traffic to partially initialized nodes.

• Replication & consistency — know leader-based replication (e.g., Raft, etcd) and leaderless quorum models (e.g., Dynamo-style with vector clocks), and root causes like split-brain, stale reads, and write reordering.

• Diagnostics signals — correlate replication_lag, commit-index, election-count, GC pauses, CPU steal, syscall errors, and network RTT; sudden divergence patterns often point to networking or leader mis-election.

• Automated remediation patterns — safe steps: circuit breakers, traffic-shaping, progressive rollbacks (canaries), and self-healing (auto-restart) with kill-switches and manual override paths documented in runbooks.

• Cost/coverage tradeoff — high-cardinality traces and logs are expensive; use sampling, adaptive tracing, and retain raw logs for a short time while long-term aggregates and derived metrics persist.

• Post-incident hygiene — capture timeline, hypotheses tried, root cause, corrective action, and follow-ups; quantify operational impact in SLI terms and update SLOs or instrumentation gaps accordingly.

Worked example — Explain SLI/SLO/SLA and design monitoring

Frame: start by clarifying user-visible actions and tenants — "Which API calls define customer experience? Are there tiers with different availability promises?" Declare assumptions about request routing, data consistency, and acceptable windows.

Pillars: (1) choose 3–5 core SLIs (successful request rate, p99 latency, error-rate by endpoint), (2) set SLOs per user-impact and business tolerance (e.g., 99.9% p99 latency over 30 days), (3) design alerting and dashboards that map SLI breaches to runbooks and escalation, (4) define automated short-term remediation (circuit breaker) and rollback policy.

Tradeoff: explicitly discuss balancing alert sensitivity vs. noisy paging — choose a multi-tier alert scheme (actionable page for customer-impacting SLO breach; internal tickets for degradations).

Close: state how you'd validate—run chaos tests, synthetic traffic, and measure alert precision; "If I had more time, I'd add per-customer SLI aggregation and adaptive alerts using short-term anomaly detection."

A second angle — Diagnose distributed database inconsistency

This framing shifts emphasis to low-level replication telemetry: first ask which consistency model the system promises (strict serializability vs. eventual). Organize diagnostics around (1) leader health and election logs, (2) replication offsets / watermark comparisons across replicas, (3) client-side write-path tracing and idempotency keys, and (4) network partitions and clock skew metrics.

A strong answer argues for safe mitigation: stop accepting writes to suspect partition, promote stable replica, or perform controlled reconciliation using deterministic merge or compensation operations. Highlight tradeoffs between consistency repair cost and customer-visible rollback: sometimes serving stale reads is preferable to data loss; other times write-rollback with reconciliation is better.

Common pitfalls

Pitfall: Confusing symptom and root cause.

Many candidates jump to "scale more" when p99 spikes, whereas root causes are often latency amplification from a downstream dependency or GC pauses; always correlate traces, GC, and network metrics before capacity actions.

Pitfall: Designing alerts on internal counters.

Alerting on a single queue-length or thread-count without mapping to user-facing SLI causes pager fatigue; instead alert on user-visible degradation and attach internal counters to the dashboard for diagnostics.

Pitfall: Overengineering remediation.

Proposing fully autonomous rollback with no human-in-loop is tempting, but neglects blast-radius controls; prefer staged automation (canary, circuit-breaker, manual escalation) and explicit abort criteria.

Connections

Interviewers may pivot to capacity planning (load forecasting, autoscaling policies), security & multi-tenancy (isolation of telemetry and remediation), or deeper distributed-systems theory (consensus algorithms, clocks, and anti-entropy). Be prepared to connect monitoring decisions to deploy processes and CI/CD safeguards.

Further reading

• Site Reliability Engineering (Google SRE book) — practical guidance on SLI/SLO, incident response, and postmortems.

• Designing Data-Intensive Applications by Martin Kleppmann — solid coverage of replication, consistency models, and repair strategies.