Design a Distributed Blob Storage System for Build Artifacts

This system design round was set in the interviewing team's own domain: design a distributed storage system for **build artifacts**, the outputs of the build pipeline (compiled binaries, packages, archives and similar files) that later pipeline stages download in order to test and deploy them. The interviewer described it as similar to designing a distributed NoSQL database, with two differences: the stored values are opaque blobs rather than documents or key-value records, and the ACID requirements are less strict than a database's. Design the system end to end: the API that build jobs and downstream consumers use, how artifacts and their metadata are stored and located, how data is replicated and kept durable, how the system scales for both uploads and downloads, and exactly which consistency guarantees you keep and which you relax. ```hint Separate what never changes from what must be current Look at which data is written once and never modified, and which small records must always be up to date, and give each kind only the consistency it actually needs. ``` ```hint Follow a popular artifact Trace what happens when many test jobs request the same freshly built artifact within a minute, and find where that load lands in your design. ``` ### Clarifying Questions - What is the size distribution of artifacts (many small files, a few multi-gigabyte images, or both), and how many are produced per day? - Who reads artifacts, how soon after upload, and how many readers fetch the same artifact? - Are artifacts immutable once published, or can a name be overwritten, for example a "latest build of this target" pointer? - How long must artifacts be kept, and do some, such as release builds, need longer retention than routine CI outputs? - How are artifacts looked up: by content hash, by build ID, or by commit and target name? - Does the system run in one data center or several regions, and what durability and availability targets apply? - May the design build on an existing object store, or is the storage layer itself part of the question? ### What a Strong Answer Covers - Requirements and back-of-the-envelope estimates for stored volume, upload and download bandwidth, and object counts - A clean split between an immutable blob layer and a strongly consistent metadata layer, with a clear key scheme - An upload protocol that makes an artifact visible only after all of its bytes are durable and verified - Chunking, placement, replication or erasure coding, and repair after disk or node failure - A read path that stays healthy when one artifact is suddenly in high demand - Retention, deduplication and safe garbage collection - An explicit statement of which transactional guarantees are kept, which are relaxed, and why relaxing them is safe for build artifacts ### Follow-up Questions - Two builds of the same commit produce byte-identical artifacts. How does deduplication work, and how does it interact with garbage collection? - Release artifacts must be kept far longer than routine CI outputs. How do you implement retention policies and make deletion safe? - A storage node holding copies of a hot artifact fails during a burst of downloads. What do clients see, and how does the system recover? - A build publishes a set of artifacts that must become visible together or not at all. How do you support that without general multi-object transactions?