Compute Expected Calibration Error Across Data Shards
Quick Overview
Compute Expected Calibration Error on sharded predictions using mergeable bin statistics, with correct weighting, confidence boundaries, and empty-data checks.
Compute Expected Calibration Error Across Data Shards
Company: Inception
Role: Software Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
Implement an aggregation strategy for Expected Calibration Error (ECE) when prediction data is split across shards. Explain what each shard emits and how the final metric is computed without gathering all predictions on one machine.
### Constraints & Assumptions
Use top-label ECE: each example has a confidence in [0,1] and a correctness indicator. Fix the same bins on every shard. For this practice formulation use B equal-width bins, with confidence 1 assigned to the final bin. The source does not specify B or a different calibration variant.
### Clarifying Questions to Ask
- Is confidence the probability of the predicted class or a per-class probability?
- Are bins fixed globally, and how are values on bin boundaries assigned?
- Should empty total input return an undefined result or raise an error?
```hint Aggregate sufficient statistics
A shard's final ECE loses information. Think about the quantities needed to reconstruct each global bin's accuracy and average confidence.
```
### What a Strong Answer Covers
- Per-bin counts, confidence sums, and correct-prediction sums.
- Global merging before taking absolute differences.
- Consistent boundaries, empty bins, empty input, and confidence validation.
- Complexity in examples, shards, and bins.
### Follow-up Questions
- Why is a weighted average of shard ECE values generally incorrect?
- How would quantile-based bins change the distributed computation?
Overview: Compute Expected Calibration Error on sharded predictions using mergeable bin statistics, with correct weighting, confidence boundaries, and empty-data checks.
Implement an aggregation strategy for Expected Calibration Error (ECE) when prediction data is split across shards. Explain what each shard emits and how the final metric is computed without gathering all predictions on one machine.
Constraints & Assumptions
Use top-label ECE: each example has a confidence in [0,1] and a correctness indicator. Fix the same bins on every shard. For this practice formulation use B equal-width bins, with confidence 1 assigned to the final bin. The source does not specify B or a different calibration variant.
Clarifying Questions to Ask Guidance
Is confidence the probability of the predicted class or a per-class probability?
Are bins fixed globally, and how are values on bin boundaries assigned?
Should empty total input return an undefined result or raise an error?
What a Strong Answer Covers Guidance
Per-bin counts, confidence sums, and correct-prediction sums.
Global merging before taking absolute differences.
Consistent boundaries, empty bins, empty input, and confidence validation.
Complexity in examples, shards, and bins.
Follow-up Questions Guidance
Why is a weighted average of shard ECE values generally incorrect?
How would quantile-based bins change the distributed computation?