Make Cache Fetch Thread-Safe and Single-Flight

Quick Overview

Design a thread-safe cache fetch operation that coalesces concurrent misses for the same key while allowing different keys to load in parallel. Define linearization points, generation-aware publication, lock scope, timeout and error handling, deletion races, and stale completion cleanup.

Company: Tubi

Role: Software Engineer

Category: Software Engineering Fundamentals

Difficulty: medium

Interview Round: Technical Screen

# Make Cache Fetch Thread-Safe and Single-Flight A TTL cache adds fetch(key, loader, ttl): return a valid cached value immediately; otherwise call the loader, cache a successful result, and return it. Loader errors must not be cached. Explain how to make this operation safe and efficient when many threads request the same key. ### Constraints & Assumptions - The loader can take 100 milliseconds or longer and may fail or time out. - Different keys should usually load concurrently. - The cache supports put, get, delete, and fetch. - A key can be replaced or deleted while a load is in flight. - Practice assumption: values are immutable once published. ### Clarifying Questions to Ask - Should concurrent callers share the same loader result and error? - Does delete invalidate an in-flight fetch? - Are loaders cancelable, and is stale-on-error allowed? - Is the cache in one process or distributed across processes? ### Part 1: Thread-Safe Core State Define synchronization for the value map, TTL metadata, and any recency or cleanup structures. Explain the linearization point of each operation. #### Hints - Correctness involves coordinated structures, not only a thread-safe dictionary. #### What This Part Should Cover - Atomic state transitions - Bounded lock scope - Expiration and replacement races ### Part 2: Fetch and Key Conflicts Prevent a miss storm for one key while preserving concurrency across keys. Handle loader success, error, timeout, delete, and replacement. #### Hints - The thread performing slow I/O should not hold the global cache lock. - Consider how a waiter identifies the exact generation it joined. #### What This Part Should Cover - Per-key request coalescing - Generation-aware publication - Explicit error and cancellation semantics ### What a Strong Answer Covers - A clear concurrency contract and linearization points - Fine-grained or sharded synchronization - Single-flight loading without global-lock I/O - Correct handling of stale completions, errors, and cleanup ### Follow-up Questions - How would you extend single-flight across a service fleet? - Should a transient loader error be negatively cached? - How do you prevent unbounded per-key lock objects? - What changes if values can be mutated by callers?

Overview: Design a thread-safe cache fetch operation that coalesces concurrent misses for the same key while allowing different keys to load in parallel. Define linearization points, generation-aware publication, lock scope, timeout and error handling, deletion races, and stale completion cleanup.