Testing nondeterministic software: invariants, controlled randomness and statistical evals

How would you test a system that can give different outputs for the same input? ```hint Split the randomness Sort the sources of variation into ones you can pin down in a test and ones you cannot, then decide what "correct" means when the same input can legitimately produce several different outputs. ``` ### Clarifying Questions - Where does the variation come from: a model that generates text or predictions, timing and concurrency in a distributed system, or randomness built in on purpose? - Does each input have one right answer, as a payroll calculation does, or a range of acceptable answers, as a drafted message does? - Is the question about testing before release only, or also about checking behavior once real customers use it? - How costly is a wrong output here: a slightly awkward suggestion, or a wrong paycheck or access permission? ### What a Strong Answer Covers - The sources of non-determinism, and which of them can be controlled or isolated in tests - Correctness defined through properties, invariants and acceptance ranges instead of exact expected outputs - Statistical testing: repeated runs, pass-rate thresholds and evaluation sets for model-driven behavior - Layers of testing from deterministic components through pre-release evaluation to staged rollout and production monitoring - Test rigor matched to the cost of a failure, with human review or deterministic fallbacks where errors are expensive ### Follow-up Questions - A test suite passes on 97 runs out of 100 on the same build. Is the build broken, and how would you decide? - The vendor behind a model you depend on updates it without notice. How would your testing catch a regression? - An engineer proposes retrying failing tests automatically until they pass. How do you respond?