Design high-throughput event subscription system

Quick Overview

This question evaluates a candidate's ability to design scalable, fault-tolerant event ingestion and subscription architectures, covering distributed systems, high-throughput messaging, content-based filtering, multi-tenant isolation, API/webhook design, delivery semantics, backpressure, sharding, security, and operational observability.

Design an Event Ingestion and Subscription System

Context

You are asked to design a horizontally scalable platform where producers send high-volume events (fire-and-forget, up to 1,000,000 events/second). Consumers register content-based subscription rules and webhooks to receive matching events.

Assume events are JSON objects with a tenant_id and metadata (id, timestamp, type). The system must isolate tenants and provide durable, reliable delivery to subscribers.

Requirements

Event producers
- Fire-and-forget ingestion of JSON events.
- Up to 1M events/sec sustained, with burst tolerance.
Subscribers
- Register subscription rules (content-based filtering) and webhook endpoints to receive matching events.
Define the following:
- Rule expression language (syntax, supported operators, limits).
- Registration APIs for subscriptions and webhook config.
- Webhook delivery protocol, including retries, exponential backoff, idempotency, and error handling.
- Ordering and filtering guarantees.
- Backpressure strategy for slow webhooks and overloaded tenants.
- Sharding and scalability strategy across services and storage.
- Multi-tenant isolation, rate limiting, and quotas.
- Security: authentication, authorization, encryption, data isolation, and webhook verification.
Non-functional targets (state clearly in your design):
- Durability guarantees.
- Delivery semantics (at-least-once vs exactly-once).
- Latency SLOs (ingest to delivery).
- Operational observability and failure handling.

Design high-throughput event subscription system

Quick Overview