Determine Player Preference for Local Game Creators
Quick Overview
Roblox analytics prompt on measuring player preference for local creators, covering engagement metrics, availability adjustment, observational causal methods, selection bias, matching, fixed effects, and guardrails.
Determine Player Preference for Local Game Creators
Company: Roblox
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Roblox analytics team wants to verify whether players prefer games created by local creators.
##### Question
Define primary and secondary metrics to measure player preference for local creators. If an A/B test is impossible, outline an analysis plan that credibly estimates the causal effect while mitigating selection bias.
##### Hints
Discuss time-per-session, sessions count, matching, IV, DID.
Overview: Roblox analytics prompt on measuring player preference for local creators, covering engagement metrics, availability adjustment, observational causal methods, selection bias, matching, fixed effects, and guardrails.
A Roblox analytics team wants to understand whether players prefer games created by local creators, such as creators who share the player's country, region, or language.
Assume you can label each game session or impression with whether the creator is local to the player, and you have standard engagement and retention telemetry.
Constraints & Assumptions
Define "local" explicitly and discuss data quality for location and language.
Separate preference from availability and ranking exposure.
If an A/B test is infeasible, propose observational methods and state assumptions.
Address selection bias: players may choose local games for reasons unrelated to locality.
Clarifying Questions to Ask Guidance
Is locality based on country, region, language, or creator community?
Are local games equally available in every market?
Are local games already favored by ranking or discovery surfaces?
What business decision depends on the analysis?
What a Strong Answer Covers Guidance
Primary metrics such as local-game CTR, playtime per session, completion/return rate, retention, or availability-adjusted local preference index.
Secondary metrics such as sessions per player, D1/D7 retention, spend, social co-play, creator outcomes, diversity, and satisfaction.
Guardrails for overall engagement, content quality, creator displacement, latency, and safety.