A frequently requested key expires and many application instances simultaneously miss the cache, overwhelming the backing store. Explain the failure mode and design a mitigation that preserves correctness during expiration, refresh failure, and sudden popularity changes. ### Constraints & Assumptions - The backing store is correct but cannot absorb a synchronized burst for the hot key. - Some bounded staleness may be acceptable, but it must be explicit. - The design runs across multiple application instances. ### Clarifying Questions to Ask - How stale may the response be, and are missing values cacheable? - Can the backing computation be made idempotent? - Is the hot-key set known or does it emerge dynamically? ### What a Strong Answer Covers - The distinction among penetration, breakdown on one hot key, and broad cache avalanche. - Request coalescing or a distributed lease with safe timeout behavior. - Soft and hard expirations with stale-while-revalidate when policy permits. - TTL jitter, negative caching, rate limits, and degraded responses. - Metrics for miss bursts, refresh ownership, backend load, staleness, and failed refreshes. ### Follow-up Questions - What if the refresh owner crashes while holding the lease? - How would you prevent a stale value from being written after a newer refresh? - How does the answer change when stale data is never acceptable?