Observability For Financial Services

What's being tested

Interviewers probe your ability to design and reason about practical observability for production services: choosing what to instrument, how to collect and store signals, and how to detect and diagnose incidents under real-world constraints (cost, cardinality, privacy, compliance). For a Software Engineer at a bank, the focus is on engineering tradeoffs: low-overhead instrumentation, reliable correlation across distributed components, preserving auditability and PII controls, and producing actionable SLIs/SLOs that tie to customer impact. Expect to be graded on clarity of assumptions, measurable success criteria, and tradeoff justification rather than vendor selection.

Core knowledge

Telemetry types: understand the three pillars — metrics (numeric time-series), traces (request flows, spans), and logs (events/context). Use metrics for alerting, traces for latency and root-cause, and logs for forensic details.
SLI/SLO basics: define Service Level Indicators precisely (e.g., successful payment within 500ms). SLO example: 99.9% of payments complete <500ms over 30 days. Convert to allowed error budget: $\text{error\_budget} = 1 - \text{SLO}$ and measure burn rate.
Percentiles and histograms: capture latency histograms to compute p50/p95/p99. Use HDR histograms or Prometheus histogram buckets; avoid relying on mean for tail problems.
Correlation and context: generate a correlation ID per user request (propagate via headers) and instrument spans with operation name, service, and resource. Prefer OpenTelemetry semantic conventions for interoperable traces.
Sampling strategies: apply a mix of head-based, tail-based, and adaptive sampling. Sample traces aggressively for errors and high-latency requests; downsample routine success traces to control costs, while ensuring representative tail coverage.
Cardinality control: cap label cardinality on metrics (avoid IDs in metric labels). Use tags for bounded dimensions (region, payment_method) and push high-cardinality context to logs or trace attributes with careful retention.
Structured logging & PII: emit JSON logs with schema, include only non-sensitive context or use tokenization/encryption for PII. Ensure logs are searchable but redact or token-map account numbers pre-ingest to meet PCI-DSS/SOX concerns.
Alerting design: prefer burn-rate and SLO-based alerts over purely threshold alerts. Use short-alert escalation for emergent spikes and longer windows to detect sustained degradations. Define actionable alerts with next-step runbook pointers.
Latency sources & instrumentation: instrument service boundaries, DB calls, external APIs, and queues. Record timing for queue wait time, handler processing, and downstream calls to disambiguate tail latency sources.
Storage, retention, and cost: store high-cardinality logs/traces short-term (days) and roll up metrics for long-term retention. Offload cold telemetry to object storage like S3 for audits; balance retention with regulatory needs and cost.
Time & monotonicity: ensure clocks are synchronized (NTP) and use monotonic timers for durations; record both wall-clock and monotonic timestamps in traces to avoid negative durations across reschedules.
Security & immutability: encrypt telemetry in transit and at rest (KMS), sign or Append-Only Store for audit logs, and ensure least-privilege access to observability data. Maintain immutable logs for forensic requirements.

Worked example — "Design observability for a payment-processing microservice"

First 30 seconds: clarify scale (TPS), SLA (latency and success rate targets), topology (sync vs async), compliance obligations (PCI retention/redaction). State assumptions: 10k TPS peak, SLO 99.9% p99 latency <500ms, logs retention 1 year for audits.

Skeleton answer pillars: 1) define SLIs/SLOs and error budget; 2) instrumentation: metrics (request_rate, error_rate, latency_histogram) + traces (span per service/db/external), structured logs with correlation ID; 3) collection and processing: local buffering, export via OpenTelemetry to central store; 4) alerting/dashboards tied to SLO burn-rate and key metrics; 5) compliance (PII redaction) and retention strategy.

Tradeoff to flag: full sampling of traces gives perfect diagnostics but is cost-prohibitive at 10k TPS; propose adaptive sampling — sample all errors and a small fraction of successes, plus targeted trace-on-threshold (e.g., latency >250ms). Explain implications for root-cause coverage and forensic completeness.

Close: state test plans (canary rollout, synthetic transactions), what you'd add with more time (detailed alert thresholds, runbooks, replayable trace/log pipeline, SLA verification tests), and how you’d measure observability effectiveness (MTTD/MTTR, SLO burn-rate).

A second angle — "Investigate sudden increase in p99 transaction latency"

Frame diagnostic approach: check SLO burn-rate and whether errors or latency driving the issue. Use traces sampled around p99 to identify which span dominates tail; correlate with metrics like DB queue length, CPU, GC pause metrics, and external API latencies. Look for increased contention (locks), retries, or queueing; if traces are sparse due to sampling, temporarily increase sampling for affected services or enable targeted sampling for requests exceeding threshold. Consider non-observability causes too (deployment, config change) and verify via deployment metadata in traces.

Common pitfalls

Pitfall: Averaging vs tail behavior — Engineers often monitor mean latency and miss tail spikes; always reason with percentiles (p95/p99) and histograms because customer-facing impact aligns with tails.

Pitfall: Metric label explosion — Adding user/account IDs to metric labels seems helpful but creates high cardinality and cripples storage/aggregation; keep labels bounded and push high-cardinality data to logs/traces.

Pitfall: Alerts without action — Creating alerts that aren't tied to a clear next step produces alarm fatigue; define actionable alerts (who does what) and prefer SLO burn-rate alerts over raw thresholds.

Connections

Observability often leads into adjacent topics: distributed tracing internals (sampling algorithms, span context propagation), SRE practices (SLIs/SLOs, error budgets, runbooks), and secure telemetry processing (PII masking, audit log immutability). Interviewers may pivot to system scaling, database contention analysis, or deployment/CI impacts on reliability.