Real-Time Clickstream Analytics Pipeline with Kafka and Flink

Scenario

You need to design a real-time pipeline that ingests website click events via Kafka, processes them using Apache Flink, and writes queryable aggregates to a data warehouse or lakehouse for downstream analytics.

Assume the business wants near real-time (<1 minute) aggregate metrics (e.g., page views per URL, unique users, funnels) with correctness guarantees suitable for critical decisioning. Click events are append-only and can arrive out of order.

Task

Describe the end-to-end design, addressing:

1. Kafka topic and partitioning strategy (keys, partition count, retention, compaction).
2. Flink processing: event-time windowing, watermarks, allowed lateness, and handling late data.
3. State management: state backend, sizing, TTLs, checkpointing, and savepoints.
4. Data modeling in the warehouse/lakehouse for downstream consumption (raw, sessionized, and aggregates).
5. Failure handling and back-pressure strategies.
6. Semantics and evolution: at-least-once vs exactly-once, checkpointing, sink guarantees, and schema evolution.

Keep the design practical and call out trade-offs and key configuration choices.