OP here. A recruiter reached out to me, and I was already thinking about switching jobs, so I went for the interview. The team is really interesting. The main team is in Tel Aviv, Israel, and they want to hire people in the US to build up a team on this side too. The Israeli team came from a company that Crusoe acquired. The team members are fairly young, and they all looked like they had less than 5 years of work experience.
Phone Screen: it was with the team's Hiring Manager. He mainly asked what inference-related work I had done, and he also introduced what the team does. The interview was scheduled for 6:45am, which I thought was strange. Why schedule an interview that early? Right then I felt this hiring manager did not really want to hire anyone.
VO: there were only two technical rounds in total. It happened that people from the Tel Aviv team were visiting the US, so the same two interviewers showed up for both rounds, one asking questions while the other sat to the side taking notes. They asked a lot of basic inference concepts, for example the difference between the Prefill and Decoding stages, and how an Inference Engine handles a request. They also asked how I would debug it if an inference system hit a bottleneck, and, in a prefill/decode disaggregation (PD separation) setup, how I would debug a latency spike on my prefill server when it goes from 23 to 24 qps. The second round leaned a bit toward system design: how to design an auto-scaling inference system.
I have always worked on product teams, so I haven't had much exposure to distributed inference systems. Mostly I have optimized a single inference service (speculative decoding, quantization, routing, load balancing) and things like that. I feel there is still a gap between that and the tech stack the companies that do inference for a living use now, so it was no surprise that I got rejected this time.
Discussion
Loading comments…