Reuse Work Across Multiprocess Image Pipelines

# Reuse Work Across Multiprocess Image Pipelines You have several image-processing pipelines, each made of ordered stages. A working version runs different pipelines in separate processes. Optimize it so that if two pipelines share the same stage prefix, the intermediate result after any shared stage can be reused safely rather than recomputed. Explain the implementation and its correctness under concurrency. ### Constraints & Assumptions - A stage is deterministic only when its code version, parameters, and input bytes are identical. - Intermediate images may be large. - Worker processes do not share ordinary heap memory. - Pipelines may fail, retry, or be canceled. - A stale result from an older stage version is incorrect. ### Clarifying Questions to Ask - Are stages pure, or can they depend on time, randomness, or external state? - What latency, throughput, and storage limits matter most? - May cached intermediates survive process or host restarts? - How often do pipeline definitions and stage versions change? ### Part 1 - Identity and execution graph Transform pipeline lists into a shared execution graph and define a cache key that proves two stage outputs are reusable. #### What This Part Should Cover - Prefix or DAG representation - Content, parameters, and implementation version in identity - Immutable intermediate references - No reuse across impure stages ### Part 2 - Concurrent execution and storage Explain ownership, single-flight behavior, process communication, result publication, memory limits, and cleanup. #### What This Part Should Cover - Atomic claim and publish protocol - Shared memory versus disk/object storage trade-offs - Reference counts or leases - Failed and abandoned computation recovery ### Part 3 - Validation and measurement Show how you would prove equivalence to independent execution and determine whether reuse is worthwhile. #### What This Part Should Cover - Byte or semantic equality checks - Race and version-invalidation tests - Hit rate, avoided compute, and transfer cost - Memory, disk, and end-to-end latency metrics ```hint Make intermediate identity content-addressed A reusable stage result needs to bind the previous result, stage parameters, and implementation version, not merely a human-readable stage name. ``` ### What a Strong Answer Covers - A shared prefix/DAG model and collision-resistant result identity - Safe process-level single flight and immutable publication - Resource-aware storage, lifetime, retry, and invalidation - Correctness tests and measurements that include transfer overhead ### Follow-up Questions 1. What changes if one stage uses nondeterministic GPU kernels? 2. How would two hosts share intermediates without turning storage into the bottleneck? 3. When can recomputation be cheaper than reading a cached image?