Pandas And NumPy ML Data Manipulation

What's being tested

This tests vectorized data manipulation with pandas and NumPy: filtering, joining, reshaping, aggregation, missing-value handling, and array broadcasting. The candidate must choose idiomatic operations over Python loops, preserve row alignment, avoid leakage or unintended mutation, and reason about time and memory costs in an ML pipeline.

Patterns & templates

• Boolean filtering with df.loc[mask, columns]; combine conditions with & and |, parenthesize each expression, and use .isin() for membership.

• Group aggregation with df.groupby(keys, observed=True).agg(...); named aggregations improve readability, while .transform() preserves original row count for features.

• Safe joins with pd.merge(..., how="left", validate="many_to_one"); inspect key uniqueness to prevent accidental Cartesian row explosions.

• Missing values using .isna(), .fillna(), and .dropna(); fit imputers on training data only, then reuse them for validation and serving.

• Vectorized NumPy operations use broadcasting and functions such as np.where, np.clip, np.nanmean, and np.select; avoid row-wise .apply() when arrays suffice.

• Reshaping with pivot_table, melt, and stack; explicitly define aggregation when duplicate index-column pairs exist.

• Performance favors columnar operations, categorical keys, and selective columns; estimate memory because a 10M-row merge can exceed available RAM.

Common pitfalls

Pitfall: Using df["flag"] == True or chained indexing can create ambiguous behavior; use .loc and explicit boolean expressions.

Pitfall: groupby().mean() silently excludes missing values, while joins may duplicate records; validate row counts and null rates afterward.

Pitfall: axis=0 operates down rows and axis=1 across columns; confirm shapes before relying on broadcasting.

Practice these

The practice cards below cover the canonical variants—solve all of them and time yourself.