Practice designing and evaluating an in-app purchase pricing A/B test for a mobile game. The solution covers pricing hypotheses, profit-based metrics, conversion and LTV trade-offs, statistical power, skewed revenue, promotional costs, guardrails, and follow-up experiment design.
##### Question
TikTok is running an A/B test on two in-app purchase pages for a mobile game.
State your hypotheses for why Page A (higher price, fewer buyers) could outperform Page B (lower price, more buyers) and vice-versa.
List the primary and secondary metrics you would track (e.g., conversion rate, ARPU, LTV, ROI) and explain why.
Describe the statistical approach you would use to determine significance and how you would check if the current sample size provides enough power.
Explain how you incorporate promotional cost ("spend X, get bonus Y") into your ROI calculation.
Propose a follow-up experiment based on the initial results.
Quick Answer: Practice designing and evaluating an in-app purchase pricing A/B test for a mobile game. The solution covers pricing hypotheses, profit-based metrics, conversion and LTV trade-offs, statistical power, skewed revenue, promotional costs, guardrails, and follow-up experiment design.
mediumProduct ManagerOnsiteProduct / Decision Making
16
0
Product Experiment Prompt: In-App Purchase A/B Test Design and Evaluation
You are testing two purchase pages in a mobile game. Page A presents a higher price with fewer buyers. Page B presents a lower price with more buyers. Assume user-level randomization, a 50/50 split, and a 14-28 day observation window.
The goal is to decide which page maximizes long-term profit while maintaining user experience.
Constraints & Assumptions
Optimize for profit and long-term value, not just purchase conversion rate.
Include platform fees, variable costs, refunds, promotions, and downstream retention where relevant.
Keep randomization stable at the user level across sessions.
Consider sample size, statistical significance, and practical significance.
Clarifying Questions to Ask Guidance
Are prices for the same product bundle or different bundles?
What platform fees, taxes, bonus currency, and variable costs apply?
Is the observation window long enough to estimate LTV, or do we need a proxy?
Are users exposed once or repeatedly across sessions?
Are there regional, payer-status, or player-level segments that should be pre-specified?
Part 1 - Hypotheses
Explain why Page A could outperform Page B and why Page B could outperform Page A.
What This Part Should Cover Guidance
Price elasticity, conversion rate, average order value, payer quality, and repeat purchase.
User experience and perceived value.
Cannibalization, anchoring, refund risk, and long-term willingness to pay.
Retention and monetization spillovers.
Part 2 - Metrics
List primary and secondary metrics and explain why.
What This Part Should Cover Guidance
Primary metric such as gross profit per exposed user or long-term LTV proxy.
Secondary metrics such as purchase conversion, ARPPU, ARPU, repeat purchase, retention, refund rate, and support contacts.
Guardrails for user experience, churn, session length, progression fairness, and policy compliance.
Part 3 - Statistical Approach and Test Design
Explain how to determine statistical significance, whether the sample has enough power, and how to analyze results.
What This Part Should Cover Guidance
Unit of randomization, eligibility, exposure logging, and stable assignment.
Minimum detectable effect, variance, sample size, and duration.
Confidence intervals, hypothesis tests, CUPED or covariate adjustment if appropriate.
Segment analysis without overfitting.
Decision rules for ship, iterate, or run longer.
Part 4 - Promotions and Follow-up Experiment
Explain how to incorporate promotional cost, such as "spend X, get bonus Y," into ROI and propose a next experiment.
What This Part Should Cover Guidance
Net revenue after platform fees, bonus cost, variable cost, refunds, and support cost.
Profit per user and payback.
Follow-up tests based on the result: price ladder, personalized offer, bundle framing, limited-time promo, or segmentation.
What a Strong Answer Covers Guidance
A strong answer chooses a profit-based primary metric, not conversion alone, and evaluates pricing through statistical power, user experience guardrails, promotional economics, and longer-term retention or LTV.
Follow-up Questions Guidance
What if Page B wins conversion but loses profit?
What if Page A increases ARPPU but hurts retention?
How would you handle whale-driven variance?
How would you estimate long-term LTV from a 14-day test?