Design a CI system that runs one million tests in ten minutes

Design a continuous integration (CI) system for a large codebase whose test suite has about 1,000,000 tests. When a developer submits a change, the system must build it, run the tests and report which tests failed, and a full run of the suite must complete within 10 minutes. Explain how the design meets that budget, and argue for each choice through its trade-offs rather than by listing components. ```hint Do the arithmetic first Before drawing any boxes, divide the total test time by the budget. The number of parallel workers this implies, and the duration of the single slowest test, drive most of the later decisions. ``` ```hint What sets the finish time When the work is spread over thousands of machines, ask which machine's work decides when the result is ready, and what you can do about it. ``` ### Constraints and Clarifications - About 1,000,000 tests per full run. - A full run must finish within 10 minutes of wall-clock time. - Test durations, change volume and the cost budget are not given. Ask for them, or state your assumptions explicitly. ### Clarifying Questions - Does the 10-minute budget include building the code and getting machines ready, or only running the tests? - What do test durations look like: mostly sub-second unit tests, or a long tail of multi-minute integration tests? Is per-test duration history available? - Must every change run all 1,000,000 tests, or may the system run only the tests a change can affect? - Are tests hermetic (no shared state, no network calls), and how often are tests flaky? - How many changes are submitted per hour at peak, and is there a limit on machine cost? ### What a Strong Answer Covers - A capacity estimate that links the test count, per-test duration and the 10-minute budget to the number of parallel executors and machines - Partitioning and scheduling of tests so that no shard finishes after the budget, including very long tests and slow machines - How build outputs reach thousands of executors quickly, and what is reused across runs - Running every test versus a selected or cached subset, and how missed regressions are caught - Flaky tests, executor failures and retries handled without breaking the budget - Cost versus latency (warm capacity versus autoscaling), and the metrics that show the 10-minute target is being met ### Follow-up Questions - One test alone takes 12 minutes. What do you do? - Fifty changes are submitted in the same minute. How does each still finish within 10 minutes without 50 times the machines? - How would you detect that a test has become flaky, and what happens to a change it fails? - How do you check that test selection or result caching is not silently skipping tests that should have run?