Design Real-Time Analytics Pipeline with Kafka and Flink
Overview
This question evaluates a Data Scientist's competency in designing real-time streaming architectures, covering event-time semantics, stateful stream processing, fault-tolerant checkpointing, partitioning and data modeling across Kafka, Flink, and downstream warehouse or lakehouse systems.
Real-Time Clickstream Analytics Pipeline with Kafka and Flink
Scenario
You need to design a real-time pipeline that ingests website click events via Kafka, processes them using Apache Flink, and writes queryable aggregates to a data warehouse or lakehouse for downstream analytics.
Assume the business wants near real-time (<1 minute) aggregate metrics (e.g., page views per URL, unique users, funnels) with correctness guarantees suitable for critical decisioning. Click events are append-only and can arrive out of order.
Task
Describe the end-to-end design, addressing:
- Kafka topic and partitioning strategy (keys, partition count, retention, compaction).
- Flink processing: event-time windowing, watermarks, allowed lateness, and handling late data.
- State management: state backend, sizing, TTLs, checkpointing, and savepoints.
- Data modeling in the warehouse/lakehouse for downstream consumption (raw, sessionized, and aggregates).
- Failure handling and back-pressure strategies.
- Semantics and evolution: at-least-once vs exactly-once, checkpointing, sink guarantees, and schema evolution.
Keep the design practical and call out trade-offs and key configuration choices.
Solution
Login required