Streaming Python Parser for Key-Value Log Lines with Quoted and Escaped Values

Host machines and IoT edge devices, such as network cameras, door-access controllers and gateways, continuously write operational logs. Every line starts with the timestamp of the event, followed by any number of key-value pairs joined by `=`: ```text <timestamp> <key1>=<val1> <key2>="<val 2 with spaces>" <key3>=<val3> ... ``` Write a Python program that reads such a log file and parses every line into a `dict`. The dict holds the line's timestamp under the key `'timestamp'`, plus one entry per field. Log files can be several gigabytes, so the program must stream the file through a generator rather than load it. **Line format** - **Timestamp.** Starts the line. It is either an ISO-8601 value with no spaces, such as `2026-09-29T22:15:30Z`, or a standard date and time separated by a single space, such as `2026-09-29 22:15:31.450`. Store it, as written, under `'timestamp'`. - **Key.** One or more letters, digits, underscores, hyphens or dots (matching `[\w.-]+`), immediately followed by `=`. - **Unquoted value.** Contains no whitespace and ends at the next space or at the end of the line, for example `host=cam-01`, `status=failed` or `retries=3`. - **Quoted value.** Wrapped in a matching pair of double quotes. Inside the quotes it may contain spaces, runs of several spaces, `=` signs (as in `details="reboot=true"`) and escaped double quotes, written as a backslash followed by a quote (`\"`). The output drops the outer quotes and turns each escape back into the character it stands for. - **Empty value.** `note=""` produces `{'note': ''}`. - **Separators.** Consecutive pairs are separated by one or more spaces or tabs. - **Blank lines.** Skip lines that are empty or contain only whitespace. **Examples** Mixed value types: ```text Input: 2026-09-29T22:15:30Z host=cam-01 event="camera offline" status=failed retries=3 Output: {'timestamp': '2026-09-29T22:15:30Z', 'host': 'cam-01', 'event': 'camera offline', 'status': 'failed', 'retries': '3'} ``` Runs of spaces between pairs, and a timestamp that contains a space: ```text Input: 2026-09-29 22:15:31.450 host=gw-99 ip=192.168.1.1 msg="handshake failed: connection reset" code=500 Output: {'timestamp': '2026-09-29 22:15:31.450', 'host': 'gw-99', 'ip': '192.168.1.1', 'msg': 'handshake failed: connection reset', 'code': '500'} ``` Escaped quotes, an inner `=` and an empty value: ```text Input: 2026-09-29T22:15:33Z event="firmware update" details="version=\"2.4.1\" reboot=true" note="" Output: {'timestamp': '2026-09-29T22:15:33Z', 'event': 'firmware update', 'details': 'version="2.4.1" reboot=true', 'note': ''} ``` ```hint Tokenizing Splitting on whitespace, or on every `=`, fails on the quoted values in these examples. Think about what a tokenizer has to remember while it is inside a quoted value. ``` ```hint Where the timestamp ends One timestamp format contains a space. Decide how you will tell where the timestamp stops and the first key begins. ``` ```hint Memory Ask which ways of reading a file hold all of it, or a list of every parsed record, in memory at once. ``` ### Constraints and Clarifications - Do not read the whole file at once: no `read()` and no `readlines()`. Yield one parsed record at a time from a generator, so that auxiliary memory stays constant with respect to file size. - Each line is one record. - All values are returned as strings, exactly as the examples show (`retries=3` becomes `'3'`). The timestamp is also returned as a string, not converted to a `datetime`. ### Clarifying Questions - What should happen to a malformed line, such as one with no recognizable timestamp, an unterminated quote, or a token without `=`: skip it, raise an error, or report it with its line number? - If a key appears twice on one line, which value wins? - Besides `\"`, which backslash escapes can appear inside quoted values (for example `\\`), and how should an unknown escape be treated? - Can a field be named `timestamp`, and if so, how should it coexist with the leading timestamp? - Is `key=` with nothing after the `=` legal, and does it mean an empty string? - What encoding are the files in, and what should happen to bytes that do not decode? ### What a Strong Answer Covers - Correct tokenizing of quoted values: spaces and `=` inside quotes, escaped quotes, and empty strings - Reliable separation of both timestamp formats from the first key-value pair - A generator that streams the file with memory independent of file size, and that skips blank lines - A stated policy for malformed lines and duplicate keys that the code actually follows - Tests built from the three examples plus edge cases such as tabs, trailing whitespace and a quoted value at the end of a line - Linear time per line, and the trade-off between a regular expression and a hand-written scanner ### Follow-up Questions - How would you report malformed lines without stopping the stream, so that an operator can find them in a multi-gigabyte file? - How would you parse one large file faster using several CPU cores, given that byte offsets do not line up with line boundaries? - How would you extend the parser so that selected fields, such as `retries` or `code`, are converted to typed values according to a schema? - What changes if a quoted value can contain an escaped newline, so that one record spans several lines?