Scenario

You are evaluating whether to launch a B2C chatbot for retailers on a commerce messaging platform. The chatbot automates merchant–customer responses (e.g., FAQs, order status, returns, product info), is available 24/7, and can hand off to humans when needed. Direct A/B testing is not feasible due to contractual or product constraints.

Assume you have a simple pre/post visualization showing trends such as average response time and resolution rate for early adopters versus the historical period, but not a full experimental control.

Task

1. Articulate the value this chatbot provides to:
   • (a) Retailers
   • (b) The platform
   • State whether it is worth launching and under what conditions.
2. Define and justify key metrics from three perspectives:
   • (a) User/customer
   • (b) Business/merchant and platform
   • (c) Model/operations
3. Since A/B testing is not possible, describe how you would:
   • (a) Size the opportunity
   • (b) Identify launch signals and guardrails
   • (c) Use quasi-experimental methods (e.g., synthetic control, matched markets, interrupted time series) to infer impact
4. Given a simple visualization (as described above), critique its strengths and weaknesses and suggest improvements.

Hints

• Discuss retention, merchant response time, revenue lift, cost savings, quasi-experiments or synthetic controls when A/B is impossible.