Respond to a Reliability Regression Shortly After Deployment

A new version has been deployed, and 10–15 minutes later latency or reliability worsens. Explain how you would detect the problem, decide whether to roll back, measure customer impact, and prevent recurrence. ### Constraints & Assumptions - Treat the temporal link to deployment as evidence to investigate, not proof of the root cause. - Prioritize restoring service while retaining enough evidence to diagnose the failure. - State what signals and rollback prerequisites you need; no specific service architecture or error budget is supplied. ### Clarifying Questions to Ask - Which endpoint, region, user cohort, or dependency is affected? - Did the release include schema, data, configuration, or compatibility changes that complicate rollback? - Is the previous version known healthy, and can traffic be shifted safely? ```hint Separate mitigation from explanation The evidence required to safely restore service can be narrower than the evidence required to prove the final root cause. ``` ### What a Strong Answer Covers - Detection through latency distributions, errors, saturation, and customer-facing success signals. - Comparison of new and old versions or rollout cohorts. - A risk-based rollback or mitigation decision with explicit compatibility checks. - Customer-impact measurement and incident communication. - Root-cause analysis, targeted regression checks, and rollout/observability improvements. ### Follow-up Questions - When would you avoid an immediate binary rollback? - How would you distinguish a release regression from a coincident dependency incident?