Definition

An embedding is a way to represent discrete content (a word, a sentence, an image) as a list of continuous numbers, called a vector. These vectors typically have hundreds to thousands of dimensions. No single dimension has a clear meaning on its own, but together they encode what the content means.

The key property: content with similar meaning ends up close together in vector space. “Cat” and “kitten” sit near each other; “cat” and “stock market” sit far apart. That lets computers judge similarity with distances and angles instead of relying on exact wording.

How It Works

Early word vectors. Methods like Word2Vec built on a simple observation: words that appear in similar contexts tend to mean similar things. Models were trained to predict surrounding words from a word, or the reverse, and each word ended up with a fixed vector. These vectors even captured some analogies. The drawback was one vector per word, so they couldn’t tell apart different senses of the same word.

Context-dependent representations. Transformer-based models compute each word’s vector from the entire sentence, so the same word gets different representations in different sentences, reflecting context more accurately.

Sentence and document embeddings. In practice, it’s more common to produce one vector for a whole passage. Dedicated embedding models are usually trained with contrastive learning: given many semantically related text pairs (questions and answers, titles and bodies), the model pulls related vectors together and pushes unrelated ones apart.

Multimodal embeddings. Images and text can be mapped into the same space too, so “a dog running on grass” lands near a matching photo. That enables text-to-image search.

Common similarity measures include cosine similarity (do the vectors point the same way?), dot product and Euclidean distance.

A few practical rules:

  • Compare vectors from the same model only. Different models produce incompatible spaces.
  • Switching models means re-embedding. Upgrading your embedding model usually requires regenerating vectors for all existing data.
  • Dimension affects cost. Higher dimensions mean more storage and compute, and quality doesn’t necessarily scale with them.

Typical Applications

  • Semantic search: finding content by meaning rather than keywords.
  • Retrieval-augmented generation (RAG): locating the most relevant passages for an LLM.
  • Recommendations: suggesting items based on similarity between user and item vectors.
  • Clustering and classification: automatically grouping similar reviews or tickets.
  • Deduplication and anomaly detection: spotting near-duplicates or outliers.
  • Cross-lingual retrieval: multilingual embedding models place same-meaning sentences in different languages close together.
  • Vector database: purpose-built storage and search for embeddings.
  • RAG: relies on embeddings for semantic retrieval.
  • Transformer: the dominant architecture for modern embedding models.
  • Token: inside an LLM, each token is first mapped to an embedding vector.
  • Multimodal: cross-modal embeddings put images and text in one comparable space.

Papers: Efficient Estimation of Word Representations in Vector Space (Word2Vec), Sentence-BERT