Design a Merchant Catalog Update Pipeline with Bulk Files and Image Resizing

Design a system that lets merchants upload and update product attributes (price, stock status, description, and images) that annotate product pins. Merchants upload original images, and the server must resize them. The system must support both single-item updates and bulk updates, where a merchant uploads a file covering an entire catalog. Image updates have lower priority. Price and stock status must take effect as quickly as possible, while images can be delayed. ### Constraints Numbers given by the interviewer: | Dimension | Value | | --- | --- | | Merchants | 50,000 | | Products | 1 billion | | Write rate | Spiky, from 0 up to 100,000 QPS at peak | | Read rate | About 100,000 QPS, roughly constant | | Bulk jobs | About 100 | | Regular attribute payload | About 1 KB | | Image size | About 500 KB on average | ### Clarifying Questions - Are the roughly 100 bulk jobs running at the same time, or is that a daily count? - How many images does a product have, and how many resized versions are needed per image? - Who reads the attributes at 100,000 QPS (pin rendering, search indexing, or both), and how stale can each reader's data be? - What does "as quickly as possible" mean for price and stock: seconds, or a stricter bound? - If a bulk file leaves out a product the merchant listed before, should that product stay unchanged or be removed? ### Part 1 — Architecture and Data Store Design the ingestion paths for single updates, bulk catalog files, and images, and the storage behind them. Which database would you choose for product attributes, and why? ```hint Size the data before choosing the store Estimate the attribute data volume, the peak write bandwidth, and the access pattern (mostly lookups by product ID) before picking a database. ``` #### What This Part Should Cover - Separate paths for time-sensitive attributes, bulk files, and low-priority image processing. - Capacity estimates based on the given numbers. - A database choice justified by data volume, access pattern, write spikes, and consistency needs. ### Part 2 — Conflicts Between Single and Bulk Updates A single-item update and a bulk update change the same product at the same time. How do you resolve the conflict? ```hint Decide what "newer" means Consider when a bulk file's contents were produced compared with when they are applied, and what information each write would need to carry so the two can be compared. ``` #### What This Part Should Cover - A clear ordering rule for concurrent writes to the same product. - How the store enforces that rule atomically. - Behavior when a bulk job is partly applied or is retried. ### What a Strong Answer Covers - Prioritization so that price and stock changes are never stuck behind bulk files or image processing. - Absorbing write spikes from 0 to 100,000 QPS without slowing the steady read traffic. - Bulk processing that is idempotent, can resume after failure, validates each row, and reports errors to merchants. - Image storage, asynchronous resizing, and how the product record points to finished images. - Monitoring of how long updates take to take effect, bulk job progress, and failure rates. ### Follow-up Questions 1. How would you keep one merchant's very large catalog file from starving other bulk jobs? 2. How would readers see a consistent product when the new price is live but the new image is still being resized? 3. How would you let a merchant roll back a bulk upload that contained wrong prices?