Root-cause an incident and drive consensus

Below is a structured, teach-through solution that you can adapt in a real incident. --- ## 0) Immediate triage (stabilize first) - Declare incident severity; spin up a cross-functional bridge (ASR, NLU, Shopping skill, Payments, Catalog, SRE, DS/Analytics, PM). - Freeze non-essential deploys affecting voice shopping until triage completes. - Enable/verify kill switches and feature flags are operable. Why: Minimizes additional change while you isolate the cause; reduces customer impact window. --- ## 1) Precise problem definition 1. KPI definition - Voice checkout conversion = Orders completed via voice / Voice checkout sessions (or unique checkout intents). Clarify numerator/denominator and unit of analysis. - Example: If baseline is 22% and now 16%, that’s a 6 pp (≈27%) relative drop. 2. Onset and scope - When did the drop begin? Sudden (step change) vs. gradual (trend). Identify exact timestamp. - Segment by: - Locale: en-US, en-GB, de-DE, etc. - Device: Echo Dot, Echo Show, Fire TV, mobile app. - Customer cohort: New vs. returning, Prime vs. non-Prime. - Flow/intent: AddToCartIntent → StartCheckoutIntent → ConfirmPurchaseIntent. - Payment method: tokenized card, gift card, promotional credits. - Network region/ISP, ASR model version, NLU model version, skill/runtime version. 3. Validate metric integrity - Check logging/analytics pipeline health (late events, dropped logs, schema changes). - Compare redundant sources (e.g., voice telemetry vs. order ledger). If only telemetry moved, it may be a measurement issue. Deliverable: A one-pager with the exact KPI definition, timestamp of change, and heatmap of impact by segment. This narrows the search. --- ## 2) Hypothesis generation and prioritization Use impact × plausibility prioritization. Start with components that explain the largest-affected segments. A. ASR (Automatic Speech Recognition) - Hypothesis: ASR WER increased (e.g., acoustic model change, device mic firmware, background noise patterns), lowering intent capture. - Signals: Drop in ASR success rate, spike in partial utterances, increased reprompts. B. NLU (Intent/slot resolution) - Hypothesis: NLU model change or entity resolver issues misroute checkout or fail to resolve product/quantity. - Signals: Intent distribution shifts, slot-fill failure spikes, fallback intent increases. C. Catalog/Availability/Pricing - Hypothesis: OOS rates increased, invalid offers, restricted items blocked, price mismatch causing declines. - Signals: OOS/error codes rising, offer eligibility changes, locale-specific catalog anomalies. D. Payments/Tokenization/Authentication - Hypothesis: Tokenization failures, 3DS/SCA frictions, issuer declines, expired tokens, auth prompts failing. - Signals: Payment error codes spike, processor/issuer-specific patterns, auth prompt abandonment. E. Latency/Timeouts/Capacity - Hypothesis: Upstream latency leading to timeouts at critical steps. - Signals: p95/p99 latency up, retry/timeouts increased, CPU/memory saturation, throttling. F. Traffic mix/Experiments/Config - Hypothesis: A feature flag rollout or experiment changed flow logic; traffic source mix shifted to lower-intent users. - Signals: Cohort-specific drops aligned with flag version; experiment arms with worse CR. G. Address/Shipping/Compliance gates - Hypothesis: Address validation failures, shipping promise degradation, age/gating checks fail. - Signals: Address validation errors, shipping promise anomalies, compliance service errors. Prioritization example: If the drop localizes to en-GB, Echo Show, and tokenized payment users after a specific payment service deploy, prioritize Payments/Tokenization. --- ## 3) Causal graph (from utterance to order) Represent the flow as nodes (states) and edges (transitions), with confounders and observables. Nodes (simplified): - U: User utterance - ASR: ASR transcription success - NLU: Intent+slot resolution - CAT: Catalog/offer eligibility and availability - CART: Cart update success (add/modify) - AUTH: Customer authentication/consent (voice code, 2FA) - PAY: Payment tokenization/authorization - SHIP: Address validation/shipping promise - CONF: Purchase confirmation - ORDER: Order placed (primary KPI numerator) Key edges and failure modes: - U → ASR (affected by device, noise, locale) - ASR → NLU (affected by language model, entity resolution) - NLU → CAT (affected by item match, availability) - CAT → CART (API latency/errors) - CART → AUTH (requires confirmation/voice code) - AUTH → PAY (token retrieval, SCA) - PAY → SHIP (depends on success; declines loop back or drop) - SHIP → CONF → ORDER Confounders: - Time-of-day/seasonality, traffic source mix, concurrent promotions, deploys/model updates, service capacity. Observed metrics per node/edge allow break-point identification. --- ## 4) Analyses and diagnostics A. Funnel breakpoint analysis - Compute stepwise conversion: P(ASR), P(NLU|ASR), P(CAT|NLU), …, P(ORDER|CONF). - Segment by the dimensions from Section 1. Visualize pre vs. post event. - Example: If P(PAY|AUTH) dropped from 97% → 86% while earlier steps are flat, it’s a payment-stage issue. B. Latency and error telemetry - Plot p50/p95/p99 latency and timeout rates per service. Correlate with conversion by time bucket. - Regress step conversion on latency (e.g., logistic regression with latency quantiles) to test sensitivity. C. Feature flag and experiment diffs - Compare enabled vs. holdback cohorts (steady-state, same time window). Use difference-in-differences to control for time trends. - Check assignment integrity (no spillover/interference). Validate exposure is balanced across locales/devices. D. Recent deploy/model/config diffs - Pull commit and deploy timelines for ASR, NLU, Shopping skill, Payments, Catalog, Auth. - Check model SHA/version, feature store versions, config pushes, rate limit changes. - Align timestamps with KPI change; look for matched step changes in related telemetry. E. Payment deep-dive - Break down by processor, BIN ranges, issuer, 3DS/SCA step, token age, wallet provider. - Classify failure codes: tokenization vs. auth vs. issuer decline vs. network. F. Catalog/availability deep-dive - OOS rates by top items/categories; offer eligibility changes; locale-only effects. - Confirm price/promo data consistency and SLA for updates. G. ASR/NLU deep-dive - WER, substitutions/insertions/deletions; top misrecognized phrases; intent distribution shifts; slot fill rates. - Check lexicon/customization updates and acoustic model rollouts. H. Counterfactual tests: rollback/switchback - Rollback: Revert suspect service flag/model for a targeted cohort to verify recovery. - Switchback design: Alternate treatment/control by userId (or householdId) across time blocks to average out diurnal effects; minimize network interference. - Randomization unit: user/household to prevent cross-contamination. - Block length: multiples of 1–2 hours to span traffic cycles. - Guardrails: ASR/NLU error rate, latency, cancellations/returns. I. Quantification - Estimate revenue impact: ΔCR × sessions × AOV. - Example: 6 pp drop × 1.2M sessions/day × $28 AOV ≈ $2.0M/day. --- ## 5) Short-term mitigations (stop the bleed) - Rollback/disable suspect feature flags or models (ASR/NLU update, payment flow changes) for affected segments first. - Activate circuit breakers and graceful degradation: - Reduce dependency on slow/upstream services; increase timeouts only if it improves completion. - Fallback to simpler prompts or deterministic grammars for checkout confirmation. - Reroute payments to stable processor; extend token refresh; increase retry with backoff if safe. - Capacity relief: Autoscale, prioritize checkout traffic, cache catalog lookups. - Customer safeguards: Clearer error prompts, one-tap confirmation on companion app as a temporary fallback. Trigger these with pre-defined thresholds and maintain a holdback to observe counterfactuals. --- ## 6) Long-term fixes (durable solutions) - ASR/NLU - Add domain-specific lexicons and constrained grammars for checkout phrases; improve entity resolution for quantities/variants. - Continuous evaluation pipelines with WER/intent accuracy SLOs and per-locale benchmarks. - Canary + shadow deployments with holdbacks; automated rollback on guardrail breaches. - Payments - Multi-processor failover, token refresh health checks, issuer-decline retriable heuristics. - Strengthen 3DS/SCA UX for voice with adaptive risk and minimal friction. - Catalog/Offer - SLA and alerting for OOS spikes; integrity checks for price/promo feeds; eligibility rule tests. - Reliability/Latency - End-to-end SLOs per step with error budgets; bulkhead isolation; circuit-breaking tuned by step criticality. - Pre-compute/cart snapshot caching for voice flows. - Experimentation/Change management - Mandatory holdbacks for critical-path features; switchback-ready rollout plans. - Launch checklists with stepwise guardrails and owner sign-offs. - Analytics/Observability - Unified funnel telemetry schema; golden dashboards with per-step conversion and segmented error trees. - Anomaly detection with seasonality-aware models (e.g., STL + EWMAs) on primary and leading metrics. - Process - Blameless postmortems with tracked action items; DRIs per component; comprehensive runbooks. --- ## 7) Persuading stakeholders and driving alignment Decision doc structure (crisp, 2–4 pages + appendix): 1. Executive summary: What happened, customer/business impact, recommended action. 2. Problem definition: KPI, onset, affected segments, sanity checks on data quality. 3. Evidence: Funnel breakpoints, telemetry correlations, deploy/flag timelines, counterfactual tests. 4. Options considered: Rollback scope, feature toggles, targeted mitigations, do-nothing baseline. 5. Recommendation: Chosen path with rationale. 6. Impact quantification: Revenue risk/day, customers affected, expected recovery. 7. Risks and guardrails: Potential downsides, triggers, and rollback criteria. 8. Rollout and timeline: Phases, checkpoints, switchback plan. 9. Owners and next steps: RACI with names and dates. Quantify impact - Show before/after rates, confidence intervals, and dollar impact. Include sensitivity (best/base/worst). Address risks with guardrails - Holdbacks, error/latency SLO thresholds, automated rollback triggers, daily WBR for first 2 weeks. Secure alignment - Pre-brief critical owners (Payments, ASR/NLU, SRE). In the review, surface trade-offs and commit to SLAs and timelines. Owner-by-owner action items (example) - ASR lead: Revert model vX→vW; add constrained grammar for checkout; deadline T+2d. - NLU lead: Roll back entity resolver; add regression tests; T+3d. - Payments PM/Eng: Route 30% traffic to processor B; fix token refresh job; T+1d. - Catalog Eng: Validate offer feed; add OOS anomaly alerts; T+2d. - SRE: Tune circuit breaker thresholds; capacity plan; T+1d. - DS/Analytics: Maintain investigation dashboard; run switchback analysis; T+1d. - PM: Decision doc, comms, customer messaging; T+0.5d. --- ## 8) Measuring success and preventing recurrence Success metrics (primary and leading): - Primary: Voice checkout conversion returns to baseline (or within agreed error band) for affected segments. - Leading: Stepwise conversions (ASR success, NLU resolution, payment auth rate), error rates, p95 latency, abandonment. - Customer: Reprompt rate, satisfaction proxies, complaint/contact rates. Validation plan - A/A or holdback monitoring for 1–2 weeks; ensure stability across locales/devices. - Use difference-in-differences against unaffected segments to confirm recovery is causal. Dashboards and alerts - Golden path funnel dashboard segmented by locale/device. - Seasonality-aware anomaly alerts on: overall CR, step CRs, payment error codes, ASR/NLU error spikes, latency p95/p99. - Alert policies: page on-call when thresholds breached; include runbook links. Postmortem - Blameless RCA with timeline, root cause(s), contributing factors, and quantification of impact. - Concrete remediation tasks with owners, due dates, and verification criteria. - Preventive controls checklist updated (tests, canaries, guardrails, on-call playbooks). --- By defining the problem precisely, using a causal graph to focus hypotheses, validating with funnel breakpoints and controlled rollbacks/switchbacks, and pairing immediate mitigations with durable fixes and clear ownership, you can both recover the KPI and reduce the chance of recurrence while bringing stakeholders along with quantified, risk-aware decisions.