##### Scenario B2B SaaS company considering raising subscription prices. 2) Website sign-up funnel exploring button color and placement changes. ##### Question Your PM asks for a two-week A/B test to evaluate a subscription price increase. How would you design the experiment? Should pricing be A/B-tested at all, and how would you decide if the new price is beneficial? A sign-up page currently shows a red button at the top. The team wants to test red vs blue and top vs bottom to lift click-through. How would you structure and analyze this multivariate test? ##### Hints Discuss metrics (conversion, LTV, churn), sample-size, test length, randomization, interaction effects, user cross-bucket risk, and alternative before-after designs.

Evaluates experiment design for SaaS subscription pricing and a multivariate sign-up CTA test. Strong answers address pricing-test risks, LTV metrics, factorial design, interaction effects, and guardrails.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Google.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Google during technical interviews.

Design A/B Test for Subscription Price Increase Effectiveness

A B2B SaaS company is considering two experiments: raising subscription prices and improving a sign-up page where the current CTA button is red and placed at the top. The team wants to test button color, red versus blue, and placement, top versus bottom.

Design the pricing test and the multivariate sign-up test.

Constraints & Assumptions

Pricing experiments can create fairness, trust, and long-term LTV risks.
A two-week window may be too short to observe churn and full customer value.
For the CTA test, test color and placement as factors, not only separate pairwise tests.
Define primary metrics and guardrails before launch.

Clarifying Questions to Ask

Is the pricing change for new prospects only or existing customers too?
What billing cycle, sales cycle, and conversion lag apply?
Is the signup page traffic large enough for a 2x2 factorial test?
What is the primary business objective: conversion, revenue, profit, or LTV?

Part 1 - Subscription Pricing Test

How would you evaluate a subscription price increase over two weeks? Should pricing be A/B-tested, and how would you decide if the new price is beneficial?

What This Part Should Cover

Discuss whether A/B testing price is appropriate given ethics, customer trust, sales process, and long-term effects.
If testing, limit to new prospects or eligible cohorts and randomize at account or company level.
Measure conversion, revenue, expected profit, average contract value, sales-cycle progression, refund/complaint rates, and predicted LTV.
Use longer-term holdouts, geo/cohort tests, conjoint, willingness-to-pay research, or phased rollout as alternatives.
Decide using profit or LTV, not conversion alone.

Part 2 - Multivariate CTA Test

How would you structure and analyze a multivariate test for color and placement?

What This Part Should Cover

Use a 2x2 factorial design: red/top, red/bottom, blue/top, blue/bottom.
Randomize users or accounts consistently and avoid cross-device contamination.
Estimate main effects and interaction effects for color and placement.
Track click-through, signup completion, lead quality, downstream conversion, latency, accessibility, and bounce rate.

Follow-up Questions

What if the price increase raises short-term revenue but lowers long-term retention?
How would you avoid users discovering different prices?
What if CTA clicks increase but paid conversions do not?

A/B Testing a Subscription Price Increase and Sign-up CTA

Design the pricing test and the multivariate sign-up test.

Constraints & Assumptions

Pricing experiments can create fairness, trust, and long-term LTV risks.

A two-week window may be too short to observe churn and full customer value.

For the CTA test, test color and placement as factors, not only separate pairwise tests.

Define primary metrics and guardrails before launch.

Clarifying Questions to Ask

Is the pricing change for new prospects only or existing customers too?

What billing cycle, sales cycle, and conversion lag apply?

Is the signup page traffic large enough for a 2x2 factorial test?

What is the primary business objective: conversion, revenue, profit, or LTV?

Part 1 - Subscription Pricing Test

How would you evaluate a subscription price increase over two weeks? Should pricing be A/B-tested, and how would you decide if the new price is beneficial?

What This Part Should Cover

Discuss whether A/B testing price is appropriate given ethics, customer trust, sales process, and long-term effects.

If testing, limit to new prospects or eligible cohorts and randomize at account or company level.

Measure conversion, revenue, expected profit, average contract value, sales-cycle progression, refund/complaint rates, and predicted LTV.

Use longer-term holdouts, geo/cohort tests, conjoint, willingness-to-pay research, or phased rollout as alternatives.

Decide using profit or LTV, not conversion alone.

Part 2 - Multivariate CTA Test

How would you structure and analyze a multivariate test for color and placement?

What This Part Should Cover

Use a 2x2 factorial design: red/top, red/bottom, blue/top, blue/bottom.

Randomize users or accounts consistently and avoid cross-device contamination.

Estimate main effects and interaction effects for color and placement.

Track click-through, signup completion, lead quality, downstream conversion, latency, accessibility, and bounce rate.

Follow-up Questions

What if the price increase raises short-term revenue but lowers long-term retention?

How would you avoid users discovering different prices?

What if CTA clicks increase but paid conversions do not?

Design A/B Test for Subscription Price Increase Effectiveness

Quick Overview

Design A/B Test for Subscription Price Increase Effectiveness

Constraints & Assumptions

Clarifying Questions to Ask

Part 1 - Subscription Pricing Test

What This Part Should Cover

Part 2 - Multivariate CTA Test

What This Part Should Cover

Follow-up Questions

Write your answer

Design A/B Test for Subscription Price Increase Effectiveness

Quick Overview

Design A/B Test for Subscription Price Increase Effectiveness

Constraints & Assumptions

Clarifying Questions to Ask

Part 1 - Subscription Pricing Test

What This Part Should Cover

Part 2 - Multivariate CTA Test

What This Part Should Cover

Follow-up Questions

Write your answer