Audit and Extend a Time-Series ML Pipeline

You are given an ML pipeline using time-series data from a CSV. Audit it for bugs and possible feature leakage, adjust model parameters using a valid evaluation process, and extend the pipeline so it can perform inference on new data. ### Constraints & Assumptions - The source does not provide the dataset, code, target, or confirmed bug. Treat leakage as a hypothesis to investigate, not an established diagnosis. - Define the prediction time, target horizon, and available information before validating features. - This is a pipeline engineering and modeling discussion, involving file input, data transformation, model fitting, and inference rather than a single read-only query or deterministic algorithm console. - If data contains several entities, preserve entity boundaries as well as time ordering. ### Clarifying Questions to Ask - What outcome is predicted, at what timestamp, and how far into the future? - Which fields are actually available at that prediction time, including reporting delays? - Is this one series or a panel of entities? Can the same entity appear in all time splits? - Must inference run in batches or online, and what history is available for lagged features? ### Part 1 — Audit data and features Describe how to find implementation bugs and determine whether features contain future information. #### What This Part Should Cover - Schema, timestamps, sorting, missingness, duplicates, and entity grouping. - Target construction, lag/rolling windows, as-of joins, and transform-fit boundaries. - Reproduction of a suspected bug with a small traceable example. ### Part 2 — Tune and evaluate Explain how to compare parameter changes without leaking information from the evaluation period. #### What This Part Should Cover - Chronological or walk-forward splits appropriate to the target horizon. - Training-only fitting of learned preprocessing and validation-only parameter selection. - A simple baseline, appropriate metrics, and untouched final evaluation. ### Part 3 — Add inference Define the artifacts and data contract required to reproduce training-time feature logic on new inputs. #### What This Part Should Cover - Saved model, preprocessing state, feature order, schema, and version information. - Historical context and timestamp-correct feature construction. - Missing-feature behavior, prediction outputs, and training/inference parity checks. ```hint Reconstruct one prediction timestamp For a suspect feature, list every row and field used to compute it and ask whether each was available when that prediction would have been made. ``` ### What a Strong Answer Covers - A complete audit, tuning, and inference plan that respects time and information availability. - Evidence for any claimed leakage bug and separation of bug fixes from parameter improvements. - Reproducible evaluation and a deployable feature/model contract. ### Follow-up Questions - Why can a rolling mean still leak after a chronological train/test split? - What would you check if offline metrics were strong but inference performance deteriorated immediately?