Stabilize LLM inference and estimate needed repeats
Company: Citadel
Role: Data Scientist
Category: ML System Design
Difficulty: medium
Interview Round: Technical Screen
You run an LLM-based sentiment model to score a fixed dataset of texts. Because the inference API doesn’t let you set `temperature` (and outputs are stochastic), the model produces slightly different score vectors on different days.
- Day 1 inference output is a vector \(y_1\) (one score per item).
- Day 2 inference output is \(y_2\).
- The observed Pearson correlation is \(\mathrm{corr}(y_1, y_2) = 0.95\).
Tasks:
1. **System/ML design:** How would you make inference outputs more reproducible (or at least stable) in production given limited decoding controls?
2. **Modeling question:** Propose a reasonable statistical model for this randomness and derive how many independent inference runs (e.g., days) you’d need to aggregate so that the correlation between aggregated outputs from two independent aggregations exceeds **0.99** (state assumptions clearly).
Quick Answer: This question evaluates skills in designing reliable LLM inference pipelines and in statistical modeling of stochastic outputs, including reproducibility engineering, uncertainty quantification, and the use of correlation metrics (e.g., Pearson) to measure stability.