Topic Subscription Push Systems

What's being tested

Candidates must design a scalable, robust topic subscription push system that maps publishers to subscribers while meeting latency, ordering, and delivery guarantees. Interviewers probe tradeoff reasoning: throughput vs consistency, fan-out strategies, backpressure, and failure/retry semantics — all skills central to backend systems engineering at low-latency firms.

Core knowledge

• Push vs pull: Push actively delivers to subscribers (lower latency) but requires backpressure handling; pull gives subscriber-controlled pacing at the cost of higher tail latency and more load on brokers.

• Fan-out math: If publishers emit rate $R$ and average fan-out is $f$, delivery rate ≈ $R\times f$. Use this to size brokers, network, and persistence layers.

• Partitioning & sharding: Partition by topic or subscriber; per-partition ordering needs sticky-key assignment. Partition count should match consumer parallelism; oversharding increases coordination cost.

• Broker choices: Kafka suits high-throughput durable logs with consumer pull semantics; NATS/Redis/RabbitMQ can do lower-latency push or lightweight pub/sub with different persistence and delivery semantics.

• Delivery semantics: Understand at-most-once, at-least-once, and exactly-once; exactly-once requires idempotency + transactional sinks or deduplication tokens, expensive at scale.

• Ack & retry model: Use explicit acks with exponential backoff and capped retries; store dead-letter messages after retry budget exhaustion to preserve observability.

• Ordering guarantees: Strict per-subscriber ordering ≈ single-threaded delivery per partition; parallelism trades throughput for ordering complexity and requires sequence numbers and reordering buffers.

• Backpressure & buffering: Implement per-subscriber queues with bounded buffers and policies: drop-oldest, drop-new, or slow-consumer splintering (move to offline store). Monitor p99 queue sizes and delivery latency.

• Rate limiting: Enforce per-subscriber sliding-window limits using token buckets or leaky buckets; implement on edge servers to avoid broker overload.

• Deduplication: Use message IDs and a sliding-time hash (bounded by retention) or stable offsets; memory/time bounds determine dedup window and false-positive risk.

• State & persistence: Use durable storage (Kafka, Postgres) for retention; ephemeral in-memory queues for low-latency delivery with checkpointing for recovery.

• Security & auth: Use mutual TLS or token-based auth, and topic ACLs for publisher/subscriber authorization; enforce quotas to prevent noisy-neighbor effects.

Worked example — Design a subscription push service

Start by clarifying SLAs: expected message rate, average fan-out, delivery latency SLO and ordering requirements. Declare assumptions: durable storage required, subscribers reachable over HTTP/2 gRPC, and per-subscriber delivery limit of X messages/sec. Organize the solution into three pillars: ingest & persistence, fan-out & delivery, and reliability/observability. For ingest, propose a front-end API that writes to a durable append-only log (Kafka) partitioned by topic; this decouples producers and delivery. For fan-out, use worker pools that read partitions, expand messages to subscriber lists via a subscription index (stored in Redis or Postgres), and push asynchronously over connection pools with per-subscriber rate limiting. For reliability, implement ack/retry with per-delivery state, a deduplication cache (bounded TTL), and a dead-letter store for permanent failures. Flag the tradeoff: synchronous push to live subscribers gives low latency but requires expensive connection management and complex backpressure; an alternative is hybrid push/pull where offline or slow subscribers pull from a retained queue. Close by saying: if I had more time I'd prototype end-to-end failure scenarios, quantify latencies at target load, and sketch monitoring dashboards for p99 delivery times and retry rates.

A second angle — Design a topic-based news subscription system

This framing emphasizes relevance filtering, per-subscriber delivery limits, and de-duplication for similar news items. Keep the same pipelines but add a pre-fan-out filtering stage that evaluates user preferences and dynamic relevance scores (stateless worker or cacheable rules). Implement sliding-window rate limits per subscriber using token buckets and de-duplication across near-duplicate headlines using content hashes and short retention. The ordering requirement may be relaxed (news are often independent), allowing more aggressive parallel fan-out and partitioning by topic for throughput. Here, storage retention may be shorter and deduplication accuracy matters more than strict exactly-once delivery.

Common pitfalls

Pitfall: Assuming push means you can ignore backpressure. Without bounded per-subscriber queues and drop/retry policies, a few slow subscribers can exhaust memory or block delivery workers system-wide.

Pitfall: Designing for average fan-out, not tail. Systems sized to average load collapse under bursts; always plan capacity for $p95$–$p99$ spikes and use admission control or throttling.

Pitfall: Chasing exactly-once by default. Exactly-once semantics add coordination and latency; prefer idempotent delivery and deduplication unless business correctness mandates stronger guarantees.

Connections

Interviewers may pivot to adjacent topics: stream processing (windowing, stateful operators, Flink/Kafka Streams) for in-pipeline filtering and aggregation, or load testing & benchmarking to validate throughput/latency claims. They might also ask about client-side designs (reconnect/backoff, exponential jitter) and deployment strategies (blue/green, canary).

Further reading

• [Designing Data-Intensive Applications — Martin Kleppmann] — deep treatment of messaging, logs, and replication tradeoffs, useful for consistency/ordering reasoning.

• Kafka documentation — foundational for append-only log semantics, partitioning, retention, and consumer-group delivery models.