Design a Distributed Blob Cache With Index Metadata Kept in a Database

Design a distributed caching service for a storage infrastructure team. What sets this design apart is that the service must also keep **index metadata** about what it stores. The expected design is along the lines of keeping the index metadata in a database while the data blobs themselves are stored in the cache. ### Constraints and Clarifications - The index metadata lives in a database; the data blobs live in the distributed cache. - Not given: exactly what the index metadata contains, whether the blobs also have a durable copy elsewhere, blob sizes, data volume, request rates, latency targets, and how fresh reads must be. Ask, or state your assumptions. ### Clarifying Questions - What does the index metadata hold: only which keys exist and where their bytes are, or also versions, sizes, checksums, expiry and support for listing keys? - Is there a durable backing store behind the cache, so that a lost or evicted blob can be fetched again, or is the cache the only copy? - How large are the blobs, and how skewed is access across keys? - After a key is updated, may a reader briefly see the previous version, or must every read return the latest one? - What is the ratio of reads to writes? ### Part 1 — API, index metadata and blob layout Define the client API and the records kept in the metadata database and in the cache, and walk through the read path and the write path. ```hint Give each store one job Decide which store answers whether a key exists and which version is current, and whether a cached blob should ever need to be told that its key changed. ``` #### What This Part Should Cover - The schema of the index metadata, and why it belongs in a database rather than in the cache - How blobs are named in the cache, and how that choice affects staleness after an update - The read path, including a cache miss, and the write path ### Part 2 — Distributing the cache Explain how blobs are spread across cache nodes, how nodes join and leave, how memory is bounded, and how hot blobs are handled. ```hint Placement is a choice too Compare computing a blob's node from its identifier with recording the node in the metadata, and count the database writes each approach costs on every eviction and rebalance. ``` #### What This Part Should Cover - The partitioning scheme and the effect of adding or losing a node - Replication, especially for hot blobs - An eviction policy for blobs of very different sizes, and whether the metadata tracks what is cached - Protecting the backing store, if there is one, from a burst of misses ### Part 3 — Keeping metadata and blobs consistent A write touches two systems that can fail independently. Show what a reader sees after a crash at any point during a write, after two concurrent writes to the same key, after a delete, and after a cache node or the metadata database fails. Also explain how the metadata database keeps up with read traffic, since every read consults it. ```hint Order the two writes Choose the order so that a crash between the two writes leaves, at worst, bytes that nothing refers to, and never metadata that points at bytes that do not exist. ``` #### What This Part Should Cover - Write ordering, the commit point, and cleanup of orphaned blobs - Concurrent updates and deletes - Scaling metadata reads without returning inconsistent results - Failure handling for cache nodes and the database, and the metrics that reveal problems ### What a Strong Answer Covers - Clarifies what the index metadata contains and whether the cache is backed by durable storage - A clear division of authority: the database decides what exists and which version is current, and the cache only holds bytes - A write order and blob naming scheme that rule out stale or mixed reads, with orphan cleanup - Concrete partitioning, replication and eviction choices with their trade-offs - Failure handling and observability: hit ratio, miss load, metadata latency, orphaned blobs ### Follow-up Questions - A single blob is larger than one cache node's memory. How do you store and serve it? - Why not keep the index metadata in the cache as well? What would break? - After a full restart of the cache cluster, how do you warm it without overwhelming the backing store? - Some keys are rewritten many times per second. How does your design behave for them?