Design an Event Ingestion and Subscription System

Context

You are asked to design a horizontally scalable platform where producers send high-volume events (fire-and-forget, up to 1,000,000 events/second). Consumers register content-based subscription rules and webhooks to receive matching events.

Assume events are JSON objects with a tenant_id and metadata (id, timestamp, type). The system must isolate tenants and provide durable, reliable delivery to subscribers.

Requirements

1. Event producers
   • Fire-and-forget ingestion of JSON events.
   • Up to 1M events/sec sustained, with burst tolerance.
2. Subscribers
   • Register subscription rules (content-based filtering) and webhook endpoints to receive matching events.
3. Define the following:
   • Rule expression language (syntax, supported operators, limits).
   • Registration APIs for subscriptions and webhook config.
   • Webhook delivery protocol, including retries, exponential backoff, idempotency, and error handling.
   • Ordering and filtering guarantees.
   • Backpressure strategy for slow webhooks and overloaded tenants.
   • Sharding and scalability strategy across services and storage.
   • Multi-tenant isolation, rate limiting, and quotas.
   • Security: authentication, authorization, encryption, data isolation, and webhook verification.
4. Non-functional targets (state clearly in your design):
   • Durability guarantees.
   • Delivery semantics (at-least-once vs exactly-once).
   • Latency SLOs (ingest to delivery).
   • Operational observability and failure handling.