Compute Expected Calibration Error Across Data Shards

Quick Overview

Compute Expected Calibration Error on sharded predictions using mergeable bin statistics, with correct weighting, confidence boundaries, and empty-data checks.

Compute Expected Calibration Error Across Data Shards

Company: Inception

Role: Software Engineer

Category: Machine Learning

Difficulty: medium

Interview Round: Technical Screen

Implement an aggregation strategy for Expected Calibration Error (ECE) when prediction data is split across shards. Explain what each shard emits and how the final metric is computed without gathering all predictions on one machine. ### Constraints & Assumptions Use top-label ECE: each example has a confidence in [0,1] and a correctness indicator. Fix the same bins on every shard. For this practice formulation use B equal-width bins, with confidence 1 assigned to the final bin. The source does not specify B or a different calibration variant. ### Clarifying Questions to Ask - Is confidence the probability of the predicted class or a per-class probability? - Are bins fixed globally, and how are values on bin boundaries assigned? - Should empty total input return an undefined result or raise an error? ```hint Aggregate sufficient statistics A shard's final ECE loses information. Think about the quantities needed to reconstruct each global bin's accuracy and average confidence. ``` ### What a Strong Answer Covers - Per-bin counts, confidence sums, and correct-prediction sums. - Global merging before taking absolute differences. - Consistent boundaries, empty bins, empty input, and confidence validation. - Complexity in examples, shards, and bins. ### Follow-up Questions - Why is a weighted average of shard ECE values generally incorrect? - How would quantile-based bins change the distributed computation?

Overview: Compute Expected Calibration Error on sharded predictions using mergeable bin statistics, with correct weighting, confidence boundaries, and empty-data checks.

Read the full Inception Software Engineer interview experience this question came from

Inception logo
Inception
Aug 7, 2026
mediumSoftware EngineerTechnical ScreenMachine Learning
0
0

Implement an aggregation strategy for Expected Calibration Error (ECE) when prediction data is split across shards. Explain what each shard emits and how the final metric is computed without gathering all predictions on one machine.

Constraints & Assumptions

Use top-label ECE: each example has a confidence in [0,1] and a correctness indicator. Fix the same bins on every shard. For this practice formulation use B equal-width bins, with confidence 1 assigned to the final bin. The source does not specify B or a different calibration variant.

Clarifying Questions to Ask Guidance

  • Is confidence the probability of the predicted class or a per-class probability?
  • Are bins fixed globally, and how are values on bin boundaries assigned?
  • Should empty total input return an undefined result or raise an error?

What a Strong Answer Covers Guidance

  • Per-bin counts, confidence sums, and correct-prediction sums.
  • Global merging before taking absolute differences.
  • Consistent boundaries, empty bins, empty input, and confidence validation.
  • Complexity in examples, shards, and bins.

Follow-up Questions Guidance

  • Why is a weighted average of shard ECE values generally incorrect?
  • How would quantile-based bins change the distributed computation?