Merge four CSVs locally, robustly and efficiently
Company: Capital One
Role: Data Scientist
Category: Data Manipulation (SQL/Python)
Difficulty: medium
Interview Round: Technical Screen
You receive four CSV files that must be merged locally on a laptop with 8 GB RAM, without relying on cloud services:
- products.csv: product_id, category, product_name
- purchases.csv: purchase_id, product_id, price, stars
- ads_events.csv: date (YYYY-MM-DD), user_id, ad_id, watch_seconds, clicks
- categories.csv: category, category_display_name
Constraints and quirks: files may use different delimiters (comma/semicolon), encodings (UTF-8/Windows-1252), and may contain duplicate header rows and stray BOMs; purchases.csv can exceed 10 million rows. Task: Produce two outputs—(A) category_min_price.csv containing, for every category in products.csv, the lowest price among purchases with stars > 4 (or 0 if none), and (B) weekly_watchtime.csv containing, for every (ISOYear, ISOWeek, category), the total watch_seconds from ads_events.csv after joining ads_events to products via AdID→product_id is not available; instead, a lookup file ad_product_map.csv is provided at runtime with columns ad_id, product_id. Describe and then implement (in pseudocode or Python) a robust merge pipeline that: detects and normalizes encodings and delimiters; streams large files in chunks; de-duplicates by natural keys; handles missing/NULL stars; computes ISO weeks correctly (state the formula or library call); avoids out-of-memory errors (e.g., via chunked pandas + on-disk store, or SQLite/DuckDB with CREATE TABLE AS SELECT); validates row counts and key coverage; and writes both outputs sorted and reproducible. Include exact join keys, data types you will enforce, and how you would unit test the correctness on a 1000-row synthetic sample.
Overview: This question evaluates skills in scalable data ingestion, robust data cleaning and normalization, memory-efficient joins and aggregations, and handling heterogeneous file encodings and delimiters within SQL/Python workflows; Domain: Data Manipulation (SQL/Python).
Minimum 5-star price per category (include categories with no qualifying purchases)
You are given product and purchase data loaded into SQL tables.
Create a result that contains **one row per category that appears in `products`**, with the **lowest purchase price** among purchases where **stars > 4**.
Rules:
- Join `purchases` to `products` using `product_id`.
- Only consider purchases with `stars > 4`.
- `stars` can be NULL; NULL should not count as > 4.
- If a category has **no qualifying purchases**, return **0.00** for that category.
- Output columns: `category`, `min_price`
- Sort by `category` ascending.
Tables
products(product_id INT, category VARCHAR(50), product_name VARCHAR(200))
purchases(purchase_id BIGINT, product_id INT, price DECIMAL(10,2), stars INT)
ads_events(event_date DATE, user_id VARCHAR(50), ad_id VARCHAR(50), watch_seconds INT, clicks INT)
ad_product_map(ad_id VARCHAR(50), product_id INT)
categories(category VARCHAR(50), category_display_name VARCHAR(100))
Hints
- A LEFT JOIN from products to purchases ensures categories with no purchases still appear.
- Use an aggregate with a condition (FILTER or CASE) and COALESCE to return 0.00 when there are no qualifying rows.
Weekly (ISO year/week) watch time by category via ad-to-product mapping
You are given ad event data and a runtime mapping from `ad_id` to `product_id`.
Compute total watch time per **(ISO year, ISO week, category)**.
Rules:
- Join `ads_events` to `ad_product_map` on `ad_id`.
- Join to `products` on `product_id` to get `category`.
- Exclude ad events that do not have a match in `ad_product_map`.
- Use ISO week definitions (Monday-based weeks; week 1 is the week with the year's first Thursday). In PostgreSQL, you can use `EXTRACT(ISOYEAR FROM event_date)` and `EXTRACT(WEEK FROM event_date)`.
- Output columns: `iso_year`, `iso_week`, `category`, `total_watch_seconds`
- Sort by `iso_year`, `iso_week`, `category` ascending.
Tables
products(product_id INT, category VARCHAR(50), product_name VARCHAR(200))
purchases(purchase_id BIGINT, product_id INT, price DECIMAL(10,2), stars INT)
ads_events(event_date DATE, user_id VARCHAR(50), ad_id VARCHAR(50), watch_seconds INT, clicks INT)
ad_product_map(ad_id VARCHAR(50), product_id INT)
categories(category VARCHAR(50), category_display_name VARCHAR(100))
Hints
- Use an inner join to the mapping table to drop unmapped ad_ids.
- In PostgreSQL, EXTRACT(ISOYEAR ...) and EXTRACT(WEEK ...) produce ISO week-based groupings.