System Design: Minimal ChatGPT-like Service With Reusable Presets

Context

Design a multi-tenant conversational AI service that supports reusable presets (system prompts) at user and team scope. A preset can be versioned, shared, approved, applied to a chat session, and soft-deleted. The service should expose public APIs, manage session state within context windows, and operate reliably and cost-effectively at scale.

Requirements

Specify the following:

1. Core components and their responsibilities.
2. Data model for presets:
   • Fields and metadata.
   • Ownership and scope (user, team, org, public).
   • Versioning and soft delete.
   • How presets are referenced in a chat session.
3. Public APIs:
   • CRUD for presets, list and search, apply-to-session, clone and share.
   • Request and response examples.
   • Auth and RBAC considerations.
4. Session state handling and context-window management:
   • How messages and presets combine into prompts.
   • Truncation and summarization strategy.
5. Storage choices for messages and presets, indexing, and caching.
6. Scalability plan:
   • Stateless workers, queueing and backpressure, autoscaling.
   • Latency and SLO targets.
   • Cost controls.
7. Multi-tenant isolation, rate limiting, and quota.
8. Safety and privacy controls:
   • PII redaction, preset approval flow.
   • Audit logging and observability.
9. Failure modes and fallbacks:
   • Missing preset, model timeout, partial outage.
10. Extensibility:
    • A B testing different presets, preset libraries, shareable links.
11. Capacity estimates for 100k DAU.
12. Sequence of calls for:
    • (a) Create a preset.
    • (b) Send a message using a preset.