I bugged a former coworker for a referral, and then got rejected right at the phone screen. The round was a system design interview, a classic ML inference design question. I felt okay about it after the interview and don't know why I got rejected. There were a few points where I think I either matched what the interviewer wanted, or was way off. The interviewer gave basically no real-time feedback.
Caching might not have been necessary — I argued that caching didn't make much sense here since the hit ratio wouldn't be high either, and the interviewer at least seemed to agree on the surface.
The batching strategy really was the key focus.
He had me calculate the theoretical max throughput and then estimate the actual throughput. In my own work, real-time inference throughput ends up far below the max throughput that offline inference can hit, because you have to account for latency. The interviewer didn't seem to get my point here and instead asked why it would be so far off from the theoretical max. Maybe I just don't have enough experience — in the online systems I've worked on, you can get to about 70% of offline throughput, and p99 latency is generally not great.
For the GPU server load-balancing strategy, I mentioned random round robin, and the interviewer didn't seem too convinced.
I prepared for a long time, got a perfect 600/600 on the OA, and then got rejected at the phone screen. That stings a bit.
Discussion
Loading comments…