The CLIP Encoder and Multimodal Bridges

A text query and an image use different representations. CLIP, short for Contrastive Language-Image Pretraining, learns encoders whose outputs can be compared in a shared vector space. This supports tasks such as retrieving images from a caption or ranking candidate labels for a picture. It does not itself generate a caption or execute a content policy.

Separate inputs, comparable outputs

Text enters as a token sequence. A digital image enters as a grid of sampled pixel values. Both are numerical inputs, but their raw coordinates do not provide a useful semantic comparison.