Design a Data Stream Processor
Company: Atlassian
Role: Software Engineer
Category: System Design
Difficulty: easy
Interview Round: Technical Screen
Design a real-time data stream processing system for application events.
Each incoming event contains fields such as `event_id`, `creator_id`, `event_type`, `event_time`, and `payload`. The system must ingest events at high throughput, validate them, discard bad events, and forward valid events to downstream consumers.
Examples of bad events include:
- missing or malformed required fields
- invalid or blocked creators
- timestamps outside an allowed window
- duplicate `event_id` values
Your design should cover:
- how producers send events into the system
- the ingestion and buffering layer
- validation and filtering logic
- deduplication strategy
- partitioning and ordering
- failure handling and replay
- backpressure and scaling
- storage of valid and invalid events
- monitoring and operational visibility
Quick Answer: This question evaluates the ability to design real-time data stream processing systems, testing competencies such as high-throughput ingestion, event validation and filtering, deduplication, partitioning and ordering, failure handling and replay, backpressure and scaling, storage strategies, and operational monitoring.