How would you test swapping two CTA buttons?

Q: How would you test swapping two CTA buttons?

This question evaluates A/B test design, causal inference, funnel metric selection and diagnostic/guardrail thinking within Analytics & Experimentation for a Data Scientist role, and is commonly asked to gauge the ability to reason about user behavior, conversion trade-offs, and risk management in sensitive domains like fintech.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

You work on the Airwallex marketing/product landing page. There are two primary CTA buttons:

“See a demo”
“Get started”

A proposal suggests swapping their positions (e.g., left/right or primary/secondary styling stays the same but the order changes).

Questions:

What user/business problem might this change be trying to solve? Provide 2–3 plausible hypotheses.
Design an experiment (or evaluation plan) to decide whether to ship it:
- What is the primary success metric and why?
- What secondary/diagnostic metrics would you track to understand tradeoffs across the funnel?
- What guardrail metrics matter (especially for a fintech onboarding flow)?
What are key pitfalls and biases to watch for (e.g., novelty effects, cannibalization between CTAs, heterogeneous effects by segment)?

Assume you can run an A/B test on page traffic. Define the unit of randomization and the analysis approach.

How would you test swapping two CTA buttons?

Quick Overview

Solution

Comments (0)