Define Success Metrics for Euro-Chat Customer-Service Chatbot
Overview
This question evaluates a data scientist's competency in defining product success metrics, selecting guardrail metrics, and designing A/B experiments for a customer-service chatbot, covering metric definition, statistical power, randomization, and monitoring.
Meta
Jul 12, 2025, 6:59 PM
Data Scientist
Onsite
Analytics & Experimentation
13
0
Scenario
An e-commerce company has deployed a customer-service chatbot ("euro-chat") to handle B2C support inquiries across web/app chat. The bot can answer questions and escalate to human agents when needed.
Task
Define how you would measure success for euro-chat, what guardrail metrics you would track, and how you would design an experiment to test whether the chatbot improves customer experience.
Requirements
- Propose a single primary success metric (clear definition and measurement window).
- List guardrail metrics and why they matter.
- Describe an A/B test design including: control vs treatment, randomization unit, success threshold, sample size approach, and monitoring/stop criteria.
Hints to consider
- Resolution/containment rate, CSAT, handle time.
- Control/treatment, randomization unit, success threshold, sample sizing, and monitoring.
Solution
Login required
Show