Measure Shopify App Store Success
Company: Shopify
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
You are a Data Scientist working on the Shopify App Store, a marketplace that connects Shopify merchants with third-party app developers. Merchants use the app store to discover, install, and pay for apps that help them run their businesses. Developers publish apps, acquire merchants, and earn revenue. Shopify operates the marketplace and wants to grow it while maintaining trust and quality.
Design a framework to measure the success of the Shopify App Store.
Your answer should cover:
1. The product goal and a possible north-star metric.
2. Success metrics for the three sides of the marketplace: merchants, developers, and Shopify.
3. Guardrail metrics and tradeoffs.
4. The datasets you would need, including primary keys and how the datasets would join.
5. How you would evaluate a new app-store ranking, search, or recommendation feature.
Quick Answer: This question evaluates a data scientist's competency in product-metrics design for a multi-sided app marketplace, including defining a north-star metric, per-side success and guardrail metrics, required dataset schemas and join keys, and criteria for evaluating ranking/search/recommendation experiments.
