Design log-query stream processor evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
##### Question
Design and implement a function that processes a mixed input stream of queries (prefix "Q:") and logs (prefix "L:"), assigns incremental IDs to new queries, outputs an acknowledgment for each query, and tags each log line with the IDs of matching queries as shown in the example. How would you modify your design to efficiently handle a very large volume of data? How would you support deletion of queries in your current implementation, and what inefficiencies need to be addressed?
Overview: Design log-query stream processor evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Stream Processor: Query Registration and Log Tagging
Context
You are designing a streaming component that ingests a single mixed stream of messages. Each message is either:
A query registration (prefix "Q:"), which defines a string pattern to search for in future log lines.
A log line (prefix "L:"), which must be tagged with the IDs of all queries whose pattern appears in the log text.
Assume a query is a case-sensitive substring pattern (extendable to regex later). Query IDs are assigned incrementally starting at 1, in the order queries arrive. The system outputs an acknowledgment when a query is registered, and for each log it emits the list of matching query IDs (sorted ascending).
Example
Input stream:
Q: error
Q: timeout
L: database timeout after 5s
L: ERROR: connection reset
Q: reset
L: timeout reset error
Expected outputs:
For 1) → ACK 1
For 2) → ACK 2
For 3) → L: database timeout after 5s | matches: [2]
For 4) → L: ERROR: connection reset | matches: [3] (note: case-sensitive, so "ERROR" doesn't match "error")