Design a Test Strategy for a System That Counts the Leaves on a Large Tree

In a QA engineer interview, the testing scenario was: how would you test a system that counts the number of leaves on a large tree? The prompt is deliberately underspecified. Part of the exercise is establishing what the system is before designing tests for it. ```hint Find the oracle Decide how you would know the correct count for a test input. Most of the strategy follows from that. ``` ```hint Vary one thing at a time List the properties of the tree and of the input that could change the count or break the counter, and design inputs that vary them deliberately. ``` ### Clarifying Questions - Is it a physical tree photographed or scanned by a vision system, or a software tree data structure whose leaf nodes are counted? - What accuracy is acceptable: an exact count, or a count within some tolerance? - How large is "large", and are there time or memory limits? - What are the inputs and outputs: images or a stream of nodes, a single number or a per-branch breakdown? ### What a Strong Answer Covers - Clarifying questions that pin down the system, its inputs and the accuracy requirement - A way to obtain ground truth (a test oracle) for each kind of input - Functional test cases, including boundary and degenerate inputs - Non-functional testing: scale, performance and robustness - Test levels and automation: unit, integration, end-to-end and regression ### Follow-up Questions - You cannot count the leaves of a real large tree by hand. How do you still estimate the system's accuracy? - The tree is too large for one machine's memory and is stored across machines. What new failure modes do you test? - How do you set release criteria for a count that is never exactly right?