How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Tubi.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Tubi during technical interviews.

Handle multiple experiment metrics and communication

Quick Overview

This question evaluates a data scientist's competency in experimental design and analysis, including hypothesis hierarchy, multiple-testing control, pre-registration practices, guardrail-based gating, trade-off assessment, and concise stakeholder communication.

Multi‑Outcome Experiment: Decision Framework, Multiplicity, Pre‑registration, and Communication

Context

You ran an A/B test with:

Primary metric: conversion
Secondary metric: ARPU
Guardrails: p95 latency and crash rate

Observed effects (two‑sided p‑values):

Conversion: +1.1 percentage points, p = 0.04
ARPU: −3%, p = 0.08
p95 Latency: +2%, p = 0.20
Crash rate: +0.15 percentage points, p = 0.03

Assume independent randomization, standard frequentist testing, and that guardrails are safety (worsening is bad).

Tasks

(a) Propose a principled decision framework: specify hypothesis hierarchy, whether/how to gate on guardrails, and acceptable Type I error rates.

(b) Choose and justify a multiple‑testing control (e.g., Holm, Hochberg, Benjamini–Hochberg for FDR). Apply it to the given results and note which conclusions change.

(d) Draft a ≤120‑word message to the PM with the decision and trade‑offs in plain language.

(e) If you must ship despite a guardrail hit, propose a mitigation and follow‑up plan.

Quick Overview

Context

You ran an A/B test with:

Primary metric: conversion

Secondary metric: ARPU

Guardrails: p95 latency and crash rate

Observed effects (two‑sided p‑values):

Conversion: +1.1 percentage points, p = 0.04

ARPU: −3%, p = 0.08

p95 Latency: +2%, p = 0.20

Crash rate: +0.15 percentage points, p = 0.03

Assume independent randomization, standard frequentist testing, and that guardrails are safety (worsening is bad).

Tasks

(a) Propose a principled decision framework: specify hypothesis hierarchy, whether/how to gate on guardrails, and acceptable Type I error rates.

(b) Choose and justify a multiple‑testing control (e.g., Holm, Hochberg, Benjamini–Hochberg for FDR). Apply it to the given results and note which conclusions change.

(d) Draft a ≤120‑word message to the PM with the decision and trade‑offs in plain language.

(e) If you must ship despite a guardrail hit, propose a mitigation and follow‑up plan.

Handle multiple experiment metrics and communication

Quick Overview

Multi‑Outcome Experiment: Decision Framework, Multiplicity, Pre‑registration, and Communication

Context

Tasks

Solution

Comments (0)

Handle multiple experiment metrics and communication

Quick Overview

Multi‑Outcome Experiment: Decision Framework, Multiplicity, Pre‑registration, and Communication

Context

Tasks

Solution

Comments (0)