Explain how tokenization choices interact with sparse and dense retrieval in a search or retrieval-augmented generation system. Compare word, character, and subword tokenization; compare a BM25-style sparse retriever with an embedding-based dense retriever; and propose a hybrid retrieval and evaluation plan. ### Constraints & Assumptions - The corpus contains natural-language text, rare product or error codes, and some multilingual queries. - Retrieval quality must be measured before generation quality. - Latency and index-update cost matter alongside relevance. ### Clarifying Questions to Ask - Which languages, scripts, identifiers, and domain vocabulary dominate the corpus? - Is exact lexical matching or semantic paraphrase more important for the use case? - How quickly must new documents become searchable? - Are relevance judgments available at query-document level? ```hint Preserve complementary signals Sparse retrieval is strong on exact rare terms; dense retrieval can bridge vocabulary mismatch. Hybrid systems should combine scores or ranks after each component retrieves candidates. ``` ### What a Strong Answer Covers - Vocabulary, out-of-vocabulary, sequence-length, and morphology trade-offs among tokenizers. - Sparse lexical scoring and dense semantic similarity strengths and failures. - A concrete hybrid candidate-generation and reranking design. - Retrieval metrics such as recall, MRR, and nDCG at specified cutoffs. - Segment analysis for rare identifiers, languages, freshness, and latency. ### Follow-up Questions - Why can aggressive subword splitting hurt exact identifier retrieval? - How would you calibrate scores from sparse and dense systems? - What negative examples are useful when training a dense retriever? - How would you detect that generation quality is hiding poor retrieval recall?