Operational Excellence, Observability, And Production Readiness

What's being tested

Interviewers are probing whether you can make a service production-ready: instrument code for observability, reason about real-user signals, and design safe deployment and failure-handling at the application level. They want to see practical choices—what metrics/logs/traces you emit, where to put timeouts/retries/idempotency, and how your code helps reduce MTTR without depending on platform teams.

Core knowledge

Observability basics: logs, metrics, and traces are complementary signals; logs for details, metrics for aggregation/alerts, traces for end-to-end latency and dependency topology. Instrument all three at the application layer.
Structured logging: emit JSON logs with stable fields like request_id, user_id, route, status, and duration_ms so downstream systems can parse and correlate easily.
Correlation IDs / context propagation: generate a request_id at edge and propagate via headers (e.g., X-Request-ID) into logs and tracing spans to join signals across services.
Metric types: use counters for event counts, gauges for current state, and histograms for latency distributions; prefer histograms for p95/p99 computation. For first-time use, instrument business and system metrics.
p99 stability: tail-percentiles need large sample sizes; rule-of-thumb: keep ≥1000 samples per evaluation window or use histogram-extrapolation to avoid noisy alerts.
Tracing: instrument important spans using OpenTelemetry-compatible libraries; capture errors and span attributes (e.g., DB host, query id) but avoid PII.
Timeouts & retries: apply per-call timeouts, exponential backoff with capped retries, and circuit breakers at client libraries to prevent cascading failures; ensure retries are idempotent or guarded.
Health endpoints: expose /healthz (liveness) and /readyz (readiness) that check local critical dependencies quickly without triggering heavy work.
Safe schema changes & migrations: perform backwards-compatible schema changes (add nullable columns, dual-writes reads, phased rollouts), and include migration toggles to rollback quickly.
Deployment strategies: prefer canary or blue/green with small cohorts and automatic rollback on predefined error-rate/latency thresholds; use feature flags for risky behavior changes.
Runbooks & alerts: alert on actionable signals with a clear owner and runbook link; avoid high-noise alerts by combining metrics (error rate × traffic) and using short burn-rate windows for fast detection.
Testing for production: unit tests + integration tests + contract tests for downstream services, plus pre-production load/smoke tests and targeted chaos tests for degradation paths.

Worked example — "Design an observable HTTP microservice that meets production readiness"

First 30s framing: ask what SLAs (latency/error targets) the service must meet, main downstream dependencies, expected traffic shape, and whether any data is PII. State assumptions (e.g., typical QPS, stateful vs stateless). Skeleton of answer: (1) instrumentation plan (structured logs, business counters, latency histograms, trace spans), (2) failure handling (timeouts, retries, idempotency, circuit breakers), (3) deployment & rollout (canary + feature flags + automated rollback), (4) health/check endpoints and runbooks. A concrete tradeoff: sampling traces reduces cost but can hide rare tail issues—pick head-based sampling plus full-trace capture on errors (error-based sampling). Implementation details you'd call out: use OpenTelemetry for traces, expose duration_ms histogram with explicit buckets, propagate request_id to logs and traces, and implement per-route timeouts. Close by saying: if more time, I'd sketch concrete alert thresholds, write example prometheus rules, and add a short runbook template for 5 common failures.

A second angle — "You own a background job processor that occasionally stalls; how do you make it observable and production-ready?"

Re-frame signals: background processors need different metrics—queue depth, job processing rate, job latency, and failure/error classes. Instrument job lifecycle events (enqueue, start, success, failure, retry) and emit histogram of processing duration. Add a liveness probe that detects if the worker thread pool is starved or stuck. For stalling, correlate queue growth with per-worker CPU/memory and external dependency latency (DB locks, blocking RPCs). Design retries to be idempotent and add a dead-letter queue with diagnostic metadata. For rollout, use versioned workers processing a small fraction of queue first. The same observability principles (structured logs, request_id propagation, histograms, runbooks) apply but with queue-specific metrics and alerting conditions (e.g., sustained queue growth × drop in consumption).

Common pitfalls

Pitfall: Emitting only error counts without context. Counting “5xx”s without per-endpoint or per-client breakdown produces noisy, un-actionable alerts. Always tag metrics by meaningful dimensions.

Pitfall: Instrumenting everything verbatim. Over-instrumentation creates high cardinality metrics that blow storage and alerting; limit high-cardinality tags (avoid per-user or per-order id in metrics).

Pitfall: Treating traces as optional. Relying solely on logs/metrics makes diagnosing latency and distributed causality far slower; capture at least sampled traces and always capture full traces for errors.

Connections

This topic commonly leads to pivots into distributed tracing & context propagation, deployment strategies (canary/blue-green), and SRE/SLI-SLO discussions; be ready to explain why you chose application-level mitigations versus platform-level controls.