Implement robust word counts and min/max
Company: Amazon
Role: Data Scientist
Category: Data Manipulation (SQL/Python)
Difficulty: medium
Interview Round: Onsite
You receive a 50GB UTF-8 text corpus on disk. Implement a Python solution that:\n- Streams the file without loading it fully into memory.\n- Counts case-insensitive word frequencies, treating "e-mail" and "email" as the same token, stripping punctuation, and normalizing Unicode (e.g., accented forms). Specify your tokenization rules for hyphens, apostrophes, and emojis.\n- Ignores a provided stopword list of ~500 words.\n- Emits the top 10 words with counts and their percentage of total tokens.\n- Reports time and memory complexity in Big-O and any practical optimizations (e.g., mmap, chunking, generators).\nThen, implement the same without collections.Counter using only dicts. Finally, implement safe_min(iterable, key=None, default=sentinel) and safe_max(...) that:\n- Work with NaNs present (treat NaN as greater than all numbers for max, less than all for min) and mixed comparable types via a key function.\n- Return default if iterable is empty, else raise ValueError when default is not supplied.\n- Are stable with equal keys. Explain edge cases you tested.
Overview: This question evaluates proficiency in large-scale text processing, Unicode-aware tokenization and normalization, memory-efficient streaming and frequency counting (including implementing counting with plain dicts), and designing robust safe_min/safe_max semantics to handle NaNs, mixed comparable types, and stability concerns.
Read the full Amazon Data Scientist interview experience this question came from
