# Implement a Longest-Match Tokenizer The source reports longest matching, unknown-token handling, and an optional unknown-merging extension. The exact function flag and output payload below are a deterministic practice contract. Implement: ```python def tokenize( text: str, vocabulary: dict[str, int], unknown_id: int, coalesce_unknowns: bool = False, ) -> list[tuple[int, str]]: ... ``` Scan `text` from left to right. At each position, emit the longest vocabulary token that matches there as `(token_id, exact_token_text)`. If no token matches, consume exactly one Unicode code point as unknown. - With the default `coalesce_unknowns=False`, emit one `(unknown_id, character)` tuple per unmatched code point. - With `coalesce_unknowns=True`, merge each maximal consecutive run of unmatched code points into one `(unknown_id, substring)` tuple. A recognized token always ends an unknown run. Vocabulary keys are unique because the input is a dictionary, so two different equal-length strings cannot both match the same position. Token IDs may repeat and `unknown_id` may equal a vocabulary ID; the consumed-text field keeps the output unambiguous. Matching is case-sensitive. Concatenating the second field of every result must reproduce `text` exactly. ## Constraints and Errors - `text` is a string and every vocabulary key is a string. - Vocabulary tokens must be nonempty. An empty token or a non-Boolean `coalesce_unknowns` raises `ValueError` before producing output. - Empty input returns `[]`; an empty vocabulary makes the whole nonempty input unknown according to the selected mode. - Operate on Python Unicode code points, not encoded bytes. Discuss a direct vocabulary scan and a trie-based implementation. Test overlapping tokens, an entire unknown string in both modes, recognized text between unknown runs, duplicate token IDs, an unknown ID shared with a known token, and empty input.