Design streaming error-log counting with moving average
Company: Amazon
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Take-home Project
Design a core component in a streaming system:
Input:
- Multiple upstream services continuously emit log events.
- Each event includes at least: `service_id`, `timestamp`, `log_level`, `message`.
Tasks:
1) Filter and output only **error logs**.
2) Maintain **real-time per-service error count**.
3) Maintain a **moving average** of error count per service over a sliding time window.
4) Trigger an **alarm** when a service’s error rate/moving average crosses a threshold.
Describe the architecture, state management, windowing approach, and how you handle late events, scale, and fault tolerance.
Quick Answer: This question evaluates understanding of real-time stream processing concepts such as event filtering, stateful per-service counting, sliding-window moving averages, late-event handling, scalability, and fault-tolerant state management for monitoring and alerting.
