Architecting a RAM-Based Key-Value Cache: Eviction, Expiry, Consistency and Sharding

In a technical round for a product manager role, the interviewer asks you to design a cache that keeps frequently requested data in memory, so applications can serve it much faster than by going to the database or service behind it. Explain what the cache needs to do, how it works internally, how it behaves when memory runs out or the underlying data changes, and how it would scale past a single machine. Tie the main design choices back to what users and the business would notice. ```hint Two moments shape the design Ask what the cache should do when it is full, and what it should do when the data behind it changes. Most of the internal design and its trade-offs follow from those two answers. ``` ### Clarifying Questions - Is this a cache inside a single application process, or a shared cache that many services reach over the network? - What kind of data will it hold, how large are typical values, and how stale may a read be before users would notice or be harmed? - What read and write volumes and total data size should the design support? - Should entries also expire after a fixed time, or leave the cache only when memory is needed? ### What a Strong Answer Covers - Requirements, including the cache's interface, its latency and availability goals, and the staleness each kind of data can tolerate. - Internal data structures and an eviction policy, with the cost of reads and writes and how concurrent access is handled. - How cached data stays correct relative to its source: write strategy, expiration and invalidation. - Scaling and resilience: partitioning, replication, hot keys, stampedes and node failures. - Metrics such as hit rate and tail latency, and the trade-offs between memory cost, freshness and load, linked to user impact. ### Follow-up Questions - A single key suddenly receives a huge share of all reads. What breaks, and how do you fix it? - After a cache node restarts, the database behind it is overwhelmed. Why does this happen, and how would you prevent it? - How would you decide between building this cache and adopting an existing open-source store?