How would you test swapping two CTA buttons?
Company: Airwallex
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
You work on the Airwallex marketing/product landing page. There are two primary CTA buttons:
- **“See a demo”**
- **“Get started”**
A proposal suggests **swapping their positions** (e.g., left/right or primary/secondary styling stays the same but the order changes).
**Questions:**
1) What user/business problem might this change be trying to solve? Provide 2–3 plausible hypotheses.
2) Design an experiment (or evaluation plan) to decide whether to ship it:
- What is the **primary success metric** and why?
- What **secondary/diagnostic metrics** would you track to understand tradeoffs across the funnel?
- What **guardrail metrics** matter (especially for a fintech onboarding flow)?
3) What are key pitfalls and biases to watch for (e.g., novelty effects, cannibalization between CTAs, heterogeneous effects by segment)?
Assume you can run an A/B test on page traffic. Define the unit of randomization and the analysis approach.
Quick Answer: This question evaluates A/B test design, causal inference, funnel metric selection and diagnostic/guardrail thinking within Analytics & Experimentation for a Data Scientist role, and is commonly asked to gauge the ability to reason about user behavior, conversion trade-offs, and risk management in sensitive domains like fintech.