Design an AI-Agent Security Detection Pipeline for 1 TB/Day of Streaming Telemetry

Design a system in which an AI agent works on a stream of security data and runs security detection on it. The incoming traffic is about 1 TB per day. The question came from a team working on security products in the agentic-AI space, and it was presented as a traditional system design question: the focus is the architecture, data flow, scaling and failure handling of the whole service, not how the underlying model is trained. ```hint Size it per second first Turn 1 TB per day into bytes and events per second before picking components. Then ask which stages must touch every event and which only ever need to see a small fraction of them. ``` ```hint The input is written by your adversary Part of any security log is controlled by the attacker you are trying to catch. Consider what that means for a component that reads event content and then reaches a verdict or takes an action. ``` ### Constraints and Clarifications - The only reported number is the volume: about 1 TB of incoming security data per day. - What the stream contains, whether 1 TB is the total or per customer, the detection latency target, retention, and what the agent is allowed to do are not specified. Ask, or state your assumptions. - Treat the AI model as a component with an API, a cost per call and a latency; you do not need to design its training. ### Clarifying Questions - What does the stream contain: cloud API audit logs, network flow records, DNS queries, host or process events, or records of what other AI agents are doing? - Is 1 TB per day the total across all customers or per customer, and is the service multi-tenant? - What is the agent's job: detecting threats itself, triaging and investigating findings produced by the detection services, or also taking response actions such as revoking a credential? - How fast must a detection reach the customer: seconds, minutes, or is hourly acceptable? - How long must raw events, findings and the agent's investigation records be kept, and must a past verdict be reproducible? ### What a Strong Answer Covers - Clarifies what the stream contains, tenancy, the agent's role and permitted actions, latency and retention before designing - Converts 1 TB per day into throughput, event rate, storage and model-processing cost, and lets those numbers shape the architecture - Ingestion and stream processing: a durable partitioned log, normalization and enrichment, stateful detection with event-time windows - Where the agent sits in the flow, which tools and data it can use, and how its cost, latency and output quality are bounded - Agent safety: attacker-controlled input, tenant isolation of the agent's tools, approval for response actions, and an audit trail - Failure handling and observability: replay, idempotent findings, backpressure, behavior when the model or a detector is down, and detection-quality metrics ### Follow-up Questions - The volume grows to 100 TB per day. Which components change, and which stay the same? - An attacker plants text in a log field that is meant to convince the agent a case is benign. Where in your design is that stopped? - How do you roll out a new detection rule without flooding customers with false positives? - The agent recommends disabling a credential. When should the system act on that automatically, and when must a human approve?