Explore and Clean Unexplained 2D Image Arrays with NumPy

In a 45-minute technical interview for a machine learning data engineer role on a team that works heavily with image data, you are given several 2D NumPy arrays with little explanation of what they represent. The interviewer suggests plotting the data as an image in Python to see what it shows and how its values are distributed, and then guides you step by step through cleaning it. You may use NumPy, pandas, Matplotlib and SciPy. The report does not say what the arrays contain or which problems the interviewer pointed out, so in this version you drive both steps yourself, explain what you see, and confirm your reading of the data with the interviewer before you change it. ### Clarifying Questions - Is each array one grayscale image, one channel of a multi-channel image, or a grid of measurements of another kind? - What units and valid range do the values have, and is any special value (such as NaN, zero or a fixed fill value) used to mark missing or invalid pixels? - Do all arrays share one shape and orientation, and are they meant to line up pixel for pixel? - What will consume the cleaned arrays (model training, a visualization, a measurement), and should invalid pixels be repaired or only masked? ### Part 1 — Find out what the arrays hold Inspect the arrays and plot them so that you can explain what the data represents and how its values are distributed. Say what in the plots looks like real signal and what looks like a defect. ```hint Look before you summarize Check shape, dtype and value range first, then choose a plot whose color scale is not dominated by a handful of extreme values. ``` #### What This Part Should Cover - Structural checks: shape, dtype, minimum and maximum, count of non-finite values, most frequent values - Image plots with a color bar and a robust color scale, plus a histogram of values - Reading the plots out loud: structure, background, artifacts, suspicious values - Questions back to the interviewer about what the observed patterns mean ### Part 2 — Clean the data Using what the plots showed, clean the arrays: decide which values are invalid, handle them, and leave the data in a consistent form for downstream use such as model training. Write the cleaning as a reusable, vectorized NumPy function and show how you would check that it did what you intended. ```hint Mask first Decide how a downstream consumer will know which pixels you changed before you decide what to change them to. ``` #### What This Part Should Cover - Detecting invalid values: non-finite values, fill values, out-of-range values, isolated spikes, constant rows or columns - Repairing from valid neighbours versus masking, and keeping the mask - A vectorized implementation that never modifies the raw input - Verification by re-plotting, per-rule counts and a check that valid pixels are untouched ### What a Strong Answer Covers - Explores before cleaning and lets what the plots show drive each step - Uses vectorized NumPy rather than Python loops over pixels - Keeps an explicit validity mask and never silently changes valid data - Explains each cleaning choice and its effect on downstream machine learning - Handles dtype pitfalls such as integer wraparound and keeps thresholds confirmable rather than guessed ### Follow-up Questions - How would you run the same cleaning over many thousands of large arrays that do not fit in memory together? - How would you tell a real small bright feature from a defective pixel? - Two arrays should line up but are shifted by a few pixels. How would you detect and correct the shift? - How would you test the cleaning function so that a later change cannot silently alter valid data?