High-Level Design of Scalable Data Infrastructure for AI/ML

Trace data through ingestion, storage, processing, feature management, and access. The main contract is that a model can identify the data and feature definitions it used, both during training and at prediction time.

High-level design

The five layers separate responsibilities. They do not by themselves guarantee freshness, reproducibility, or recovery; those properties depend on the publication and versioning rules between layers.

The high-level workflow operates as follows: