Dynamic Batching Decode Loop for a Mock Language Model with Slot Refilling
Company: xAI
Role: Machine Learning Engineer
Category: Software Engineering Fundamentals
Difficulty: medium
Interview Round: Onsite
Overview: An LLM inference coding question: write the generation loop for a mock language model that returns one next token per sequence in a batch. Requests finish at different steps on a stop token or a max-token limit, and freed batch slots must be refilled from a waiting queue, testing state tracking and batching logic.