OpenAI Applied Scientist Interview Experience — A General Round and a New ML Calibration Question

RS interview: one general round and one ML round.

General: Elevator. It was exactly the same as the one on the forum. I did two parts and ran out of time, so I never saw the third part.

ML: I got a new question this time, see below.

MNIST dataset, 10 classes.

Single example:
if predict correct:
loss lower bound
loss upper bound
if predict incorrect:
loss lower bound
loss upper bound

Implement ACE metric:
Accuracy - prob. Group them in bins

numpy is not required
Write down test cases
Why is the ACE curve spiky? How should we smooth it?
Increase the number of bins
etc