Design A/B Test for Subscription Price Increase Effectiveness
Company: Google
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
B2B SaaS company considering raising subscription prices.
2) Website sign-up funnel exploring button color and placement changes.
##### Question
Your PM asks for a two-week A/B test to evaluate a subscription price increase. How would you design the experiment? Should pricing be A/B-tested at all, and how would you decide if the new price is beneficial?
A sign-up page currently shows a red button at the top. The team wants to test red vs blue and top vs bottom to lift click-through. How would you structure and analyze this multivariate test?
##### Hints
Discuss metrics (conversion, LTV, churn), sample-size, test length, randomization, interaction effects, user cross-bucket risk, and alternative before-after designs.
Quick Answer: Evaluates experiment design for SaaS subscription pricing and a multivariate sign-up CTA test. Strong answers address pricing-test risks, LTV metrics, factorial design, interaction effects, and guardrails.
A/B Testing a Subscription Price Increase and Sign-up CTA
A B2B SaaS company is considering two experiments: raising subscription prices and improving a sign-up page where the current CTA button is red and placed at the top. The team wants to test button color, red versus blue, and placement, top versus bottom.
Design the pricing test and the multivariate sign-up test.
Constraints & Assumptions
Pricing experiments can create fairness, trust, and long-term LTV risks.
A two-week window may be too short to observe churn and full customer value.
For the CTA test, test color and placement as factors, not only separate pairwise tests.
Define primary metrics and guardrails before launch.
Clarifying Questions to Ask Guidance
Is the pricing change for new prospects only or existing customers too?
What billing cycle, sales cycle, and conversion lag apply?
Is the signup page traffic large enough for a 2x2 factorial test?
What is the primary business objective: conversion, revenue, profit, or LTV?
Part 1 - Subscription Pricing Test
How would you evaluate a subscription price increase over two weeks? Should pricing be A/B-tested, and how would you decide if the new price is beneficial?
What This Part Should Cover Guidance
Discuss whether A/B testing price is appropriate given ethics, customer trust, sales process, and long-term effects.
If testing, limit to new prospects or eligible cohorts and randomize at account or company level.
Measure conversion, revenue, expected profit, average contract value, sales-cycle progression, refund/complaint rates, and predicted LTV.
Use longer-term holdouts, geo/cohort tests, conjoint, willingness-to-pay research, or phased rollout as alternatives.
Decide using profit or LTV, not conversion alone.
Part 2 - Multivariate CTA Test
How would you structure and analyze a multivariate test for color and placement?
What This Part Should Cover Guidance
Use a 2x2 factorial design: red/top, red/bottom, blue/top, blue/bottom.
Randomize users or accounts consistently and avoid cross-device contamination.
Estimate main effects and interaction effects for color and placement.
Track click-through, signup completion, lead quality, downstream conversion, latency, accessibility, and bounce rate.
Follow-up Questions Guidance
What if the price increase raises short-term revenue but lowers long-term retention?
How would you avoid users discovering different prices?
What if CTA clicks increase but paid conversions do not?