Design of a Pub-Sub System

First design

Producers write to topics, and consumers subscribe to topics to read messages. As new messages are appended to the end, we can use distributed messaging queues.

The system requires the following components:

• Topic queue: Holds publications accepted from producers.

• Subscription database: Records which subscriptions match a topic and their lifecycle state. A relational model is one option for maintaining this structured metadata.

• Message director: Reads publications, resolves matching subscriptions, and forwards them to subscription queues.

• Subscription queue: Holds the deliveries for one independent subscription, which may have multiple competing workers.

• Subscription service: Handles subscription creation and removal.

When a subscription is created, persist its metadata and starting position. The message director, also called the dispatcher here, copies matching publications into its queue. Record dispatch progress so a director restart neither silently skips subscriptions nor creates uncontrolled duplicate deliveries.