Measure App Store success and debug funnel anomaly

## Part A — Product case: measuring success for a new App Store Shopify is launching a **Shopify App Store** where merchants can browse/install apps built by third-party developers (some paid, some free). You are the Data Scientist supporting the launch. ### 1) Define success Propose a success measurement framework with: - **Primary (north-star) metric(s)** - **Input/leading metrics** (activation, engagement) - **Diagnostic metrics** (funnel rates, segment cuts) - **Guardrails** (latency, merchant churn, refunds/chargebacks, spam/fraud, support burden) Be explicit about *whose success* you’re optimizing for (merchants, developers, Shopify) and how you’d balance tradeoffs. ### 2) Data + instrumentation Specify what data you’d need and where it comes from. - List key **event streams** (e.g., clickstream/browse/search, install/uninstall, subscription/billing, app usage, support tickets). - Propose a minimal **data model** (example fact/dimension tables) that would support the metrics. Assume events arrive in near-real-time; define any time windowing (e.g., daily in UTC) and identity rules (merchant_id, app_id, developer_id, session_id). ### 3) Experimentation plan Design at least one experiment to improve App Store outcomes (e.g., ranking algorithm, pricing surfaces, recommendation modules, onboarding prompts). Include: - Unit of randomization (merchant vs session), eligibility, and duration - Primary/secondary/guardrail metrics - Key threats to validity (network effects, interference, novelty effects, selection bias) - How you’d analyze (e.g., CUPED, stratification) and make a ship/no-ship decision --- ## Part B — Data interpretation + visualization: traffic spike with worse funnel You’re given a dataset with **3 years of daily metrics** for the App Store. You notice: - A **large traffic spike** that is not explained by normal seasonality. - **Add-to-cart (ATC) rate drops sharply** during the spike. - **Conversion rate drops slightly**. Assume the table below (you may create derived fields like YoY, WoW, and rolling averages): ### Table: `daily_app_store_metrics` - `date` (DATE) - `sessions` (INT) — total visits to the App Store - `product_views` (INT) - `add_to_cart` (INT) - `purchases` (INT) - `revenue` (NUMERIC) - `channel` (STRING) — e.g., organic, paid_search, email, affiliate, referral - `device_type` (STRING) — desktop/mobile/tablet - `geo` (STRING) - `merchant_tier` (STRING) — e.g., trial/basic/plus - `landing_page` (STRING) - `app_category` (STRING) - `is_bot_suspected` (BOOL) — if available ### Tasks 1) List plausible hypotheses that could cause **sessions ↑** while **ATC rate ↓** and **conversion ↓/flat** (cover both product and data-quality causes). 2) Propose the *most useful charts* you would build (in Python or Google Sheets) to validate/refute your hypotheses. 3) Explain what follow-up data you would request if the dataset is insufficient. *Define rates as:* - `ATC_rate = add_to_cart / product_views` (or justify an alternative) - `CVR = purchases / sessions` Output expected: a structured investigation plan plus the key visualizations you’d generate.