Feature Leakage Interview Exercise: Audit a Dataset Before Training

A model that predicts hospital readmission with 99% accuracy may be useless if one feature is “discharge disposition” recorded after the prediction should have been made. The metric looks excellent because the dataset quietly contains the answer.

In a feature leakage interview exercise, resist the urge to train first. Reconstruct the decision clock, audit each column's provenance, and test whether validation reproduces deployment. Practise the diagnosis with Detect Data Leakage in Supervised Learning Pipelines.

Evidence boundary: Scikit-learn's official guidance establishes that test data must not influence fitting or preprocessing and recommends pipelines to keep transformations inside the training process. The dataset and audit below are original interview exercises. PracHub records are practice prompts, not evidence of any employer's current questions.

Start by writing the prediction contract

Before looking at correlations, write four fields: the entity, prediction timestamp, outcome window, and action. For example: “At 08:00 on the day after signup, predict whether this account will cancel within 30 days so the retention team can choose an offer.”

That sentence decides which facts are legal. Events after 08:00 cannot be features for that prediction. A cancellation reason is part of or downstream from the label. A thirty-day activity total overlaps the outcome window. A “current plan” value is ambiguous unless you know which snapshot it came from.

Also define when a field becomes available to the production system. An event may occur before the cutoff but reach the warehouse later. A manually reviewed fraud flag may have an earlier business date and a later publication time. Point-in-time correctness requires the value the serving path could actually retrieve, not the value a completed warehouse table eventually shows.

Audit this dataset before training

Assume each row predicts churn at prediction_time:

Immediately quarantine account_status and last_support_resolution until their timestamps and semantics are proven. Rename aggregate columns to include direction and boundary, such as ticket_count_[t-30d,t). A friendly column name is not a lineage guarantee.

Distinguish four common leakage mechanisms

Target leakage occurs when a feature contains the label or a consequence of it. A chargeback outcome, final diagnosis, cancellation reason, or post-review status can make the task trivial.

Temporal leakage occurs when a feature uses information later than the prediction cutoff. The value may be legitimate in another application; it is illegal for this decision time.

Split leakage occurs when related observations cross train and validation boundaries. Duplicate rows, repeated users, neighboring frames from the same video, or multiple visits from one patient let the model recognize an entity instead of generalizing.

Preprocessing leakage occurs when the test fold influences an imputer, scaler, vocabulary, feature selector, target encoder, or hyperparameter choice. Scikit-learn's common-pitfalls documentation explicitly advises splitting before preprocessing and fitting transformations only on training data. A Pipeline helps enforce that order during cross-validation.

Use lineage, not correlation, as the primary test

A suspiciously high correlation is a useful alarm, but low correlation does not prove safety. Several weak post-outcome features can combine into a strong shortcut. A hashed status code can leak just as clearly as a readable label. Conversely, a legitimate feature can be highly predictive.

For each candidate feature, record its source table, event timestamp, ingestion timestamp, transformation, window, owner, and serving path. Ask whether the same computation exists online or in the production batch. Re-run the join with a strict as-of condition and compare row counts.

Then inspect simple signals. Measure single-feature validation performance, missingness by label, cardinality, and value changes around the outcome. Search feature names and transformation code for outcome vocabulary. These tests prioritize investigation; the final decision still depends on causality and availability.

Repair the feature query at the time boundary

Suppose an incorrect query joins every payment ever observed for a user:

SELECT p.user_id, SUM(pay.amount) AS paid_total, p.churned
FROM predictions AS p
LEFT JOIN payments AS pay ON pay.user_id = p.user_id
GROUP BY p.user_id, p.churned;

The corrected version binds each event to the prediction clock and a stated lookback:

SELECT p.user_id,
       p.prediction_time,
       COALESCE(SUM(pay.amount), 0) AS paid_total_prior_30d,
       p.churned_within_30d
FROM predictions AS p
LEFT JOIN payments AS pay
  ON pay.user_id = p.user_id
 AND pay.event_time >= p.prediction_time - INTERVAL '30 days'
 AND pay.event_time <  p.prediction_time
GROUP BY p.user_id, p.prediction_time, p.churned_within_30d;

The half-open interval excludes events exactly at or after prediction. In a real system, also use the availability timestamp if late arrival matters. Preserve multiple prediction rows per user by grouping on the full prediction key rather than user_id alone.

Keep preprocessing inside the validation fold

This pattern leaks because the scaler sees the full dataset before the split:

X_scaled = StandardScaler().fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

Move learning steps into a pipeline and split first:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
model = make_pipeline(
    SimpleImputer(strategy="median"),
    StandardScaler(),
    LogisticRegression(max_iter=1000),
)
model.fit(X_train, y_train)
score = model.score(X_test, y_test)

During cross-validation, the pipeline is fitted separately on each training fold. Feature selection and target encoding belong there too. If an entity appears repeatedly, use a grouped splitter; if the product changes over time, retain an out-of-time test. A random split is not fixed merely by wrapping preprocessing in a pipeline.

Design tests that fail when time travel returns

Create a tiny fixture with two prediction timestamps for one user and events on both sides of each cutoff. Assert the exact aggregate for each row. Add an event at the boundary to confirm the intended < or <= rule. Add a late-arriving event whose occurrence time is legal but availability time is not.

For split integrity, assert that the intersection of entity IDs between train and validation is empty when entity generalization is the goal. For preprocessing, fit the pipeline on a training fixture and confirm that changing only held-out values does not change learned statistics.

Add a schema-level denylist only as defense in depth. Blocking columns named label will not catch a renamed outcome proxy. The durable controls are point-in-time joins, lineage metadata, fold-contained transformations, and review of how the production decision is made.

How should you interpret a dramatic metric drop?

If removing a suspect feature drops AUC from 0.99 to 0.71, do not declare the second model “bad.” The earlier estimate may have been invalid. Compare against a simple baseline and inspect precision, recall, calibration, and segment performance under the corrected split.

Run ablations in a controlled order: remove direct proxies, enforce time, group entities, and move preprocessing into folds. Record which change affects the score. This turns “there was leakage” into a reproducible diagnosis.

Sometimes the suspicious feature is legal for a later decision point. Then create a separate model contract rather than silently using it. A feature available after manual review may help prioritize follow-up, but it cannot justify the earlier automated decision.

Present the audit as an interview answer

Lead with the contract, then enumerate risks by mechanism. Point to two concrete columns and explain why their timestamps or meanings are unsafe. Propose a corrected query, split strategy, and fold-contained pipeline. Finish with tests and the expected metric change.

Avoid promising that one automated tool “detects leakage.” Static checks can flag names, windows, and overlap; domain review is still needed to know whether a variable is downstream from the label. The best answer combines code-level controls with product knowledge.

Practice with related PracHub questions

These records are practice material and reported context, not guaranteed questions for a future assessment.

The central habit is simple: freeze the world at prediction time. Every feature, join, transformation, and validation split must respect what was knowable then.

Run a two-model leakage drill

Train a transparent baseline twice: once on every supplied feature and once after the audit. Keep the algorithm and hyperparameters fixed so the comparison isolates the data contract. Report the metric, confusion matrix, and the top coefficients or feature importances only as diagnostic evidence. If the “all features” model is nearly perfect because account_status encodes cancellation, explain why its score is invalid rather than celebrating the gain.

Next shift the validation window forward. A feature can pass a random holdout yet fail when computed with real historical cutoffs. Compare row counts, missingness, feature distributions, and label rates by time. If the corrected model degrades sharply, investigate population change and data availability before selecting a more complex algorithm.

Finally, write a one-page audit record: prediction contract, prohibited fields, point-in-time join rule, split unit, fold-contained transformations, tests, and owner. This artifact turns the interview exercise into a production control. It also makes later feature proposals reviewable: a new column must state provenance and availability before it enters training.

Ask one final counterfactual: if the label column were deleted today, could the team still calculate every feature at the original decision time? If the answer depends on a completed case, retrospective warehouse correction, or human outcome review, the feature needs investigation. This catches indirect proxies that a column-name scan misses. Save the answer beside the feature definition so reviewers can repeat the check after pipeline changes.

Sources and Further Reading

• scikit-learn: data leakage and inconsistent preprocessing
• scikit-learn: Pipeline
• Google: Rules of Machine Learning
• scikit-learn: cross-validation and model selection