Find top co-purchased products
Overview
This question evaluates algorithmic problem-solving and data engineering skills, focusing on computing and ranking co-occurrence frequencies, choosing efficient data structures for pairwise counts, and contrasting offline batch implementation with streaming updates.
You receive purchase logs where each order is represented as an integer array of product IDs bought together in a single transaction. Implement a function top_k_related(product_id, k= 10) that returns the k products that most frequently co-occur with the given product_id across all orders. Break ties by smaller product ID. Discuss data structures to support streaming updates (orders arriving continuously), memory considerations for very large product ID spaces, and the time/space complexity of building and querying the co-occurrence counts. Provide code for the offline batch case and outline an approach for the streaming case.