Investigate SMS delivery-rate drop at Attentive
Company: Attentive
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Attentive’s US SMS delivery rate (Delivered/Attempted) drops by 6.5 percentage points starting March 10, 20:00–22:00 ET, more pronounced on two major carriers; CA/UK unaffected. A new link-shortener domain and updated 10DLC registrations rolled out March 9. Build a triage and validation plan: 1) Define exact delivery, acceptance, and failure metrics using carrier error codes; include guardrails (CTR, unsubscribe, complaint rate). 2) List the precise slices you’ll run: by carrier, sender type (short code/toll-free/10DLC), campaign/template, message length/character set, URL domain, vertical (SHAFT-sensitive), send-time, client, region, and new-vs-existing senders. 3) Describe artifact checks (duplicate sends, clock skew/timezone, logging gaps, retry policy changes, throughput throttles, MPS caps, carrier queue backoffs). 4) Prioritize hypotheses and how you’d test each: (a) carrier filtering—use error-code mix shift, acceptance vs delivery delta; (b) link-domain reputation—A/B old vs new domain with stratification by carrier and sender; (c) 10DLC registration/brand-score issues—audit TCR status and vetting outcomes; (d) content/template changes—token-level diff and spam-score models. 5) Specify the experiment design (randomization unit, sample sizing/MDE for delivery-rate uplift, sequential monitoring, holdouts), and the statistical model you’d use for attribution (hierarchical logistic regression controlling for carrier×sender×template×hour). 6) Provide an action and rollback plan, escalation criteria to carriers, and what evidence would be sufficient to declare the incident resolved.
Quick Answer: This question evaluates diagnostic analytics and incident‑triage competencies, including delivery‑metric definition, carrier error‑code interpretation, hypothesis prioritization, experiment design, and hierarchical statistical attribution relevant to messaging systems.