Design Metrics to Track and Analyze Spam Impact

Q: Design Metrics to Track and Analyze Spam Impact

This question evaluates a data scientist's ability to define proxy metrics, interpret noisy behavioral signals, and design statistically powered experiments for rare events, testing skills in product analytics, causal inference, A/B testing, and metric instrumentation.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

A messaging product team wants to reduce spam without harming normal user experience. You do not have access to a ground-truth spam classifier table.

Tasks

Metrics without a classifier: What proxy metrics would you track to monitor spam prevalence and overall user experience?
Interpreting a decline: If the user report rate (reports per message) declines, what plausible causes could explain it, and what additional metrics would you examine to disambiguate?
Experiment design with rare spammers: When running an anti-spam A/B test where spammers are rare, how would you select test and control groups to ensure power and minimize interference?

Hint: Consider message volume, unique senders, reports per message, acceptance rate of message requests, stratified sampling, and power.

Design Metrics to Track and Analyze Spam Impact

Scenario

Tasks

Solution

Comments (0)