Connect Monitoring, Circuit Breakers, Rollback, and CI/CD

Explain how you would keep a backend service reliable through monitoring, circuit breakers, production rollback, and a CI/CD process. ### Part 1 — Monitor Service Health Describe the signals that reveal user impact and help diagnose a problem. #### What This Part Should Cover Success rate, latency, traffic, saturation, dependency behavior, and actionable alerts tied to service objectives. ### Part 2 — Contain Dependency Failures Explain how a circuit breaker interacts with timeouts, retries, and fallback behavior when another service fails. #### What This Part Should Cover Breaker states, recovery probes, bounded work, and avoidance of retry amplification. ### Part 3 — Recover with Rollback Explain when rollback is useful and what can make it unsafe or insufficient. #### What This Part Should Cover Version and data compatibility, deployment scope, verification, and alternatives for irreversible effects. ### Part 4 — Implement CI/CD Describe a practical flow from a code change through validation, artifact creation, deployment, and post-deployment checks. #### What This Part Should Cover Reproducible artifacts, meaningful gates, staged rollout, observability, and a tested recovery path. ### Constraints No deployment platform, team process, or reliability target is prescribed. State a coherent baseline and connect the four parts; do not promise that monitoring or a circuit breaker alone prevents all outages. ### Clarifying Questions - Which user operations and dependencies are critical, and what objectives apply? - Can application versions coexist, and are schema changes backward compatible? - Which failures require immediate rollback versus a forward fix or data repair? ```hint Rehearse the recovery path A rollback button is useful only if the old version can still operate with the current schema, data, and surrounding services. ``` ### What a Strong Answer Covers - Signals that detect user impact and support diagnosis. - Dependency protection with explicit timeout, retry, and fallback policies. - Safe rollback reasoning and a CI/CD flow that verifies the deployed result. ### Follow-up Questions - What would you do if a rollback restores code but not corrupted data? - How would you avoid every instance probing a recovering dependency at once?