Analyze aggregator lender page flows
Company: Upstart
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
A comparison page (like NerdWallet) lists multiple loan providers (e.g., SoFi, Upstart). You must assess user behavior and partner performance across two application flows: (A) users complete an on-site prequalification/application module; (B) users click out to a partner's flow. Design a plan to: (1) instrument the page and both flows end-to-end (identify all events, ids, cross-domain tracking, and how to stitch sessions to partner outcomes under ITP/cookie loss); (2) define success metrics and guardrails for the page and for each partner (e.g., CTR-to-partner, on-site completion rate, qualified rate, funded-loan rate, revenue per session), with precise numerator/denominator definitions; (3) generate three non-obvious insights from the visible differences in partner info (e.g., APR ranges, fees, eligibility messaging) and hypothesize how they could shift user mix and downstream funding; (4) propose an experiment to compare on-site vs click-out entry points for a given partner, including experimental unit, randomization, stratification, sample-size/power assumptions, pre-registered decision rule, and how to mitigate selection bias and attribution mismatches; (5) outline how you'd measure partner cannibalization and mixed-traffic contamination across listings (e.g., user revisits, tab hoarding, last-click bias). Provide specific event names, join keys, and a minimal metric table schema you would create to drive weekly reviews.
Quick Answer: This question evaluates a candidate's competency in analytics instrumentation, cross-domain attribution under identity/cookie loss, experimentation design, precise metric specification, and detection of partner cannibalization for a loan-comparison product.