Bi-Encoders vs Cross-Encoders and the Search Methods Used in RAG

Answer two questions on retrieval for language-model applications: what the difference is between a bi-encoder and a cross-encoder, and when each should be used; and which types of search are used in retrieval-augmented generation (RAG), with the advantages and disadvantages of each. ### Clarifying Questions - Is the corpus mostly prose, or does it also contain identifiers, codes, tables or several languages? - What are the corpus size and the latency budget for retrieval? - Should the answer stop at retrieval, or also cover reranking and how results are passed to the generator? ### Part 1 — Bi-encoder versus cross-encoder What is the difference between a bi-encoder and a cross-encoder, and when should each be used? ```hint What can be precomputed Ask which parts of each model's computation can happen before the query arrives. ``` #### What This Part Should Cover - How each architecture encodes and scores a query-document pair - Per-query cost, and which one can use a precomputed index - Accuracy, typical roles in a pipeline, and how each is trained ### Part 2 — Search types in RAG What types of search are used in RAG, and what are the advantages and disadvantages of each? ```hint Exact words versus meaning Compare a query containing a rare product code with one that paraphrases a concept, and ask which retriever handles each. ``` #### What This Part Should Cover - Lexical, dense and hybrid search, plus reranking and structured filtering - Concrete strengths and failure modes of each - How to combine them and how to evaluate retrieval quality ### What a Strong Answer Covers - Correct architectures and scoring for both encoder types, and why only one of them can be served from an index - A sense of the cost difference per query - A taxonomy of RAG retrieval methods with concrete strengths and failure modes - How methods are combined (hybrid fusion, retrieve-then-rerank) and how retrieval is evaluated - Practical concerns such as chunking, metadata filters, index updates and latency ### Follow-up Questions - How would you fuse BM25 and dense results when their scores are on different scales? - How would you adapt a bi-encoder to a specialized domain? - How would you tell whether wrong answers come from retrieval or from generation? - What is late interaction, and where does it sit between the two encoder types?