Embeddings for semantic search

Use text embeddings to compare documents and queries numerically. Learn what cosine similarity measures, how a text becomes one vector, and what the supplied API examples are intended to demonstrate.

A prompt describes the task. A retrieval system also needs a way to find relevant text. Embeddings represent text as vectors that can be compared for that purpose.

Earlier we used token embeddings inside a language model. Here the goal is different: obtain a fixed-size vector for a word, passage, or query so an application can search or group texts.

A quick recap

Start by distinguishing the tokenizer’s units from the text-level vector returned by an embedding service.