Lessons from System Failures

Study failures as chains of dependencies: what triggered the incident, what spread its effects, what operators could observe, and what was required to recover. Independent probes and recovery paths help only when they avoid the dependencies that failed.

Introduction

Even widely used services experience failures, which can disrupt both individuals and businesses. System designers must understand why mature services built by experienced teams still experience outages. This chapter examines major failures in widely used services and the techniques used to mitigate them.

Two useful starting points are change and interaction. They are not an exhaustive list of failure causes:

• Change: New requirements, deployments, configuration changes, and capacity additions can expose defects. Unchanged software can also fail as traffic, data, certificates, hardware, or external dependencies change.

• Interaction: A component can behave as designed and still contribute to a system failure. For example, many clients retrying a slow dependency can create enough extra load to prevent its recovery.