Evaluate channels and allocate budget

Quick Overview

This question evaluates competency in marketing analytics and experimentation for a Data Scientist role, covering funnel analysis, conversion and CAC metrics, attribution modeling, response-curve-based budget allocation, and incremental test design using channel-level spend and outcome data.

Evaluate channels and allocate budget

Company: Upstart

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

You are given a marketing-by-channel dataset with daily aggregates: channel, spend, visits, form_starts, form_completes, loan_funded, revenue. Tasks: (1) build a funnel with stage-to-stage conversion rates and CAC at each meaningful milestone; (2) compute ROAS and payback under both last-click and 7-day first-touch attribution (describe how you would re-attribute if you only had raw click/impression logs); (3) recommend next-month budget shifts by channel using a simple response curve (e.g., log or Hill function) and estimate marginal ROAS; (4) design an incrementality test (geo or time-based holdout) to validate your recommendations, including unit of randomization, sample size, contamination risks, and success criteria; (5) flag channels that look effective but likely non-incremental due to retargeting or brand spillovers, and propose diagnostics to detect this. Provide concrete formulas, any assumptions you must make, and how you would communicate the recommendation with risk bounds.

Quick Answer: This question evaluates competency in marketing analytics and experimentation for a Data Scientist role, covering funnel analysis, conversion and CAC metrics, attribution modeling, response-curve-based budget allocation, and incremental test design using channel-level spend and outcome data.

|Home/Analytics & Experimentation/Upstart
Upstart logo
Upstart
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
13
0

Marketing Analytics Case: Funnel, Attribution, Budget Optimization, and Incrementality

You are given a daily-by-channel dataset with the following columns:

  • date
  • channel
  • spend
  • visits
  • form_starts
  • form_completes
  • loan_funded
  • revenue

Assume each row aggregates that day’s outcomes for a channel. Revenue is recognized at funding (no refunds/chargebacks) and is net of promotions but gross of media cost unless otherwise stated.

Tasks:

  1. Funnel and CAC
    • Build a funnel with stage-to-stage conversion rates: visits → form_starts → form_completes → loan_funded.
    • Compute cost-per-stage ("CAC at each milestone"): cost per visit, per start, per complete, and per funded loan.
  2. ROAS and Payback under Two Attribution Models
    • Compute ROAS and payback using: a) Last-click attribution b) 7-day first-touch attribution
    • Describe how you would re-attribute using only raw click/impression logs if the provided revenue is not already attributed as required.
  3. Next-Month Budget Shifts via Response Curves
    • Fit a simple channel response curve (e.g., logarithmic or Hill function) to link spend to outcomes and estimate marginal ROAS.
    • Recommend how to shift next-month budget by channel based on marginal ROAS.
  4. Incrementality Test Design
    • Propose a geo- or time-based holdout experiment to validate recommendations. Specify unit of randomization, sample size approach, contamination risks, success criteria, and analysis plan.
  5. Diagnose Likely Non-Incremental Channels
    • Flag channels that look effective but may be non-incremental (e.g., retargeting, brand spillovers).
    • Propose diagnostics to detect non-incrementality.

Provide concrete formulas, any assumptions you must make, and how you would communicate the recommendation with risk bounds.

Loading comments...