Quantify build-vs-buy training decision

Quick Overview

This question evaluates proficiency in quantitative decision analysis, causal inference for experimentation, KPI definition, and risk‑adjusted financial modeling within the Analytics & Experimentation domain, assessing skills relevant for a Data Scientist making operational trade‑offs.

Quantify build-vs-buy training decision

Company: Amazon

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Take-home Project

Your company plans a new employee training program and must choose among: (1) External vendor—Standard package: fastest to implement, not customizable, training data stored in the vendor’s database; (2) External vendor—Premium package: customizable, higher data security controls, longer to implement; (3) Internal build: fully tailored, longest to implement. Propose a quantitative decision framework that: a) defines the core KPIs (e.g., completion rate, performance uplift) and how you would causally estimate the incremental impact of training versus baseline (describe an experiment or quasi-experiment with success metrics and power considerations); b) constructs a risk‑adjusted NPV model over horizon H with variables for implementation time T, upfront cost C0, annual run cost Ct, annual benefit Vt, breach probability p and loss L, and a discount rate r. Write the exact decision rule/inequality that chooses among the three options, explicitly modeling the value lost while implementation is in progress; c) specifies a sensitivity analysis (which parameters, ranges, and stopping rules) and a pilot plan to update priors for Vt and p; d) given only ordinal information that Standard < Premium < Internal on customization and that Standard is fastest, defend which option you’d select and what observable threshold would cause you to switch.

Quick Answer: This question evaluates proficiency in quantitative decision analysis, causal inference for experimentation, KPI definition, and risk‑adjusted financial modeling within the Analytics & Experimentation domain, assessing skills relevant for a Data Scientist making operational trade‑offs.

|Home/Analytics & Experimentation/Amazon
Amazon logo
Amazon
Oct 13, 2025, 9:49 PM
hardData ScientistTake-home ProjectAnalytics & Experimentation
2
0

Quantitative Decision Framework for Selecting a New Employee Training Program

Context

Your company must choose one of three ways to launch a new employee training program:

  • (1) External vendor — Standard package: fastest to implement, not customizable, training data stored in vendor’s database.
  • (2) External vendor — Premium package: customizable, higher data security controls, longer to implement.
  • (3) Internal build: fully tailored, longest to implement.

Assume the goal is to maximize long‑run business value while managing risk. Implementation times differ, so benefits start later for slower options.

Task

(a) Define core KPIs (e.g., completion rate, performance uplift) and describe how to causally estimate the incremental impact of training versus baseline (experiment or quasi‑experiment), including success metrics and power considerations.

(b) Construct a risk‑adjusted NPV model over horizon H with variables: implementation time T, upfront cost C0, annual run cost Ct, annual benefit Vt, breach probability p, breach loss L, discount rate r. Write the exact decision rule/inequality that chooses among the three options, explicitly modeling value lost while implementation is in progress.

(c) Specify a sensitivity analysis (parameters, ranges, and stopping rules) and a pilot plan to update priors for Vt and p.

(d) Given only ordinal information that customization increases from Standard < Premium < Internal and that Standard is the fastest to implement, defend which option you would select now and specify an observable threshold that would cause you to switch.

Loading comments...