Design a Chat Assistant Inference Service: Streaming, Free-Tier Limits, GPU Scheduling
Company: OpenAI
Role: Software Engineer
Category: System Design
Difficulty: hard
Interview Round: Onsite
Overview: Design the serving system for a chat assistant built on a large language model: stream replies token by token, rate-limit a free tier alongside paid users, schedule requests onto GPUs efficiently, and meet latency targets. It tests streaming protocols, batching, GPU memory limits, rate limiting and overload handling.