Evaluate an In-Platform Ad Video Editor: Experiment Design and 2% Adoption

An ads platform's self-serve ads manager lets advertisers upload video creatives while they build a campaign. Today, when an uploaded video has a problem such as the wrong resolution, the advertiser has to leave the platform, fix the video offline and upload it again. The team plans to add an in-platform video editor so advertisers can fix the video without leaving the campaign-creation flow. How would you evaluate the impact of this feature? The interviewer stated that the ultimate business goal is to increase revenue by helping advertisers complete campaign creation more smoothly. ### Clarifying Questions - Is the editor offered to everyone who uploads a video, or only when an upload has a problem it can fix? - Can several people, such as team members or agency staff, work in the same advertiser account? - Which revenue counts: spend from campaigns created during the test, or the advertiser's total spend? - Does editing carry its own risk, for example edited videos being rejected in ad review or performing worse once delivered? ### Part 1 — Design the experiment Design the experiment. Should you randomize by campaign, by advertiser, or by the users of the tool? How much traffic should go to treatment and to control, and why would you, or would you not, use a 50/50 split? ```hint Who touches one account Think about everyone who works on one advertiser's campaigns, and what happens if they see different versions of the upload step. ``` ```hint Not every upload needs the editor Only uploads with a problem can benefit. Ask what that means for the effect size you can detect and for whom you analyze. ``` #### What This Part Should Cover - The three candidate units compared on consistency of experience, interference, and match with the unit that generates revenue - Why randomizing only advertisers who use the editor breaks the comparison - The traffic split weighed between statistical power and launch risk, with a ramp plan - Sample size and duration given heavy-tailed spend and the lag from submission to revenue ### Part 2 — Choose the metrics Which primary and guardrail metrics would you choose, and how would you measure whether the campaign-submission flow has improved? ```hint Revenue comes last Revenue arrives only after a campaign is submitted, approved and delivered. Ask which step the editor changes first. ``` #### What This Part Should Cover - A primary metric tied to the revenue goal but sensitive enough for the test - Submission-flow metrics for advertisers who hit a video problem - Guardrails on ad review outcomes, creative quality, tool reliability and overall spend ### Part 3 — Decide after low adoption After the experiment, only 2% of the treatment advertisers actually used the editor, and the overall lift is close to zero. The 2% is the feature's usage rate, not a revenue lift. Would you still recommend launching? ```hint Expected size of the overall effect If only a small share of the treatment group could be affected, ask how large an overall lift you should have expected even if the editor works well for them. ``` #### What This Part Should Cover - Distinguishing advertisers offered the feature, those who needed it, and those who used it - What a near-zero overall result means when 98% of the treatment group never used the feature - Estimating the effect on users without comparing users with non-users - A launch recommendation conditioned on guardrails, cost and the analysis of advertisers who needed the editor ### What a Strong Answer Covers - A clear chain from the editor to smoother creation, more submitted campaigns and revenue, with a metric at each link - A randomization unit and analysis that avoid self-selected comparisons - Traffic allocation reasoned from power and risk rather than habit - A correct reading of a diluted intent-to-treat result - A recommendation with explicit conditions and next steps ### Follow-up Questions - How would you estimate the editor's effect on the advertisers who used it, and what assumption does that estimate rely on? - Advertisers who used the editor spent more than those who did not. Why does that not show the editor caused the difference? - How would you redesign the test to have enough power for the advertisers who actually hit a video problem? - If adoption is low because advertisers never notice the editor, what would you test next?