Detailed Design of Scalable Data Infrastructure for AI/ML

Follow one dataset through the five layers and define how each layer publishes recoverable output. The named technologies are implementation choices; the design depends on their configured guarantees and the contracts between them.

1. Data ingestion layer

The data ingestion layer handles two primary data categories using dedicated services:

• Real-time events: High-velocity events (e.g., clicks, payments) are published to a distributed message queue like Apache Kafka. A stream processing engine (Apache Flink or Spark Structured Streaming) consumes, validates, and enriches these events before writing them to object storage. This supports real-time features and analytics.

• Batch data: Large-volume data (e.g., database snapshots) arrives at scheduled intervals. A workflow orchestrator, such as Apache Airflow, manages the extraction, cleaning, and formatting of this data before loading it into the raw zone of the object storage. This ensures consistent datasets for training.