Diagnose metric anomalies and evaluate new algorithm

Context

You work on a consumer product that includes an AI calling feature (users trigger calls; the system places AI-assisted calls). The team monitors operational and product metrics daily.

Assume you have access to:

• Event logs (requests, successes/failures, latency)
• User/device/app version, geo, acquisition channel
• Experiment assignments / feature flags
• Recent change log (deploys, config changes, marketing campaigns)
• Basic dashboards and the ability to query raw data

Part A — Sudden spike

Today you notice AI call count is much higher than normal (e.g., +60% day-over-day).

1. What is your step-by-step investigation plan for identifying the cause?
2. How do you determine whether it’s a real product change vs a data/measurement issue ?
3. What follow-up actions would you recommend depending on the root cause (e.g., rollback, rate limits, alerting changes, comms)?

Part B — Sudden drop

Another day you notice a key metric has dropped sharply (pick a concrete example such as conversion rate, call success rate, revenue per user, or retention).

1. How do you triage the issue (what do you check first and why)?
2. How do you localize the problem by segment (geo, app version, device, cohort, channel, experiment cell)?
3. What are common confounders and false alarms you’d guard against (seasonality, reporting lag, instrumentation changes, Simpson’s paradox)?

Part C — Launching a new algorithm

You plan to launch a new algorithm (e.g., ranking, routing, spam detection, call-quality model).

1. How do you decide whether the new algorithm is “better”?
2. Propose offline metrics and online metrics , including a primary metric , diagnostic metrics , and guardrail metrics .
3. Describe an online evaluation plan (e.g., A/B test or phased rollout), including:
   • Eligibility and randomization unit
   • Success criteria and stopping rules
   • Handling delayed outcomes and interference/network effects
   • What you would do if metrics move in opposite directions (trade-offs)

Provide your reasoning, assumptions, and concrete checks you would run.