System Design: Background Processing Backend for LLM Prompts

Context

Design a multi-tenant backend that processes large language model (LLM) prompts asynchronously. Clients submit prompts via an API and later poll for status/results or receive callbacks via webhooks. The system must support reliability, scale, and cost controls.

Requirements

1. APIs
   • Submit prompts (with idempotency keys), poll job status, fetch results, register webhooks/callbacks.
2. Job orchestration
   • Queueing, prioritization (e.g., realtime vs bulk), worker pools, retries, dead-letter queues (DLQ).
3. Model routing
   • Route requests to appropriate model/provider based on policy (latency/cost/quality/capacity).
4. Prompt versioning
   • Manage template versions and the exact prompt/model context used for reproducibility.
5. Idempotency
   • Ensure duplicate submissions do not create duplicate work/charges.
6. Retries and DLQ
   • Automatic retry with backoff; poison message handling.
7. Result storage
   • Store inputs/outputs/metadata, enable polling and callback delivery; set retention policies.
8. Observability
   • Metrics, logs, traces; per-tenant dashboards, alerting, audits.
9. Non-functionals
   • Scaling and capacity planning, cost control, rate limiting, PII/security, and SLAs/SLOs.
10. Follow-up

• Support streaming partial outputs and cancellation of in-flight jobs.

Describe the architecture, data flows, and key design choices. Provide concrete API designs and operational policies.