Load Daily JSONL LLM Chat Logs Into a Warehouse: Schema, Validation, Dedup, Idempotency
Company: Cohere
Role: Data Engineer
Category: Data Manipulation (SQL/Python)
Difficulty: medium
Interview Round: Onsite
Overview: A data engineering take-home: load daily JSONL files of LLM conversation logs from object storage into a warehouse, with schema DDL, parsing, validation and deduplication in pseudocode. The walkthrough probes NULL versus zero for missing values, global dedup, idempotent loads for conversations spanning two days, and Spark data skew.