Design an AI Gateway
Company: Decagon
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Technical Screen
Design an AI gateway service that sits between product teams and multiple third-party model providers. The gateway should expose a unified API for capabilities such as text generation and embeddings, while handling authentication, per-tenant quotas, routing, retries, streaming responses, observability, and cost controls.
Discuss the request lifecycle, core components, data model, failure handling, model selection, rate limiting, prompt or policy management, and how the system should scale. Assume the interview format allows the candidate to use AI coding tools while implementing part of the service.
Quick Answer: This question evaluates system design competencies for a multi-tenant AI gateway, including architectural design, unified API and authentication patterns, routing and retries, rate limiting and quotas, observability, failure handling, model selection, and cost-control mechanisms.