Embeddings Explained

An embedding is a list of numbers that represents the meaning of something - a word, a sentence, an image, a product. Numbers that are close together in that list correspond to things that are similar in meaning. Embeddings are the foundational primitive that powers semantic search, RAG, recommendation systems, and clustering - any task where you need a machine to understand "similar" rather than just "equal".

What embeddings are

A computer cannot compare the meaning of two strings directly. The strings "automobile" and "car" are lexically dissimilar but semantically identical; "bank" (financial) and "bank" (river) are lexically identical but semantically different. Traditional search systems treat text as bags of tokens and fail at both cases.

Embeddings solve this by mapping text to points in a high-dimensional geometric space such that semantic similarity corresponds to geometric proximity. If you embed "dog" and "cat", the resulting vectors will be close in Euclidean distance and have high cosine similarity. If you embed "dog" and "quantum mechanics", the vectors will be far apart.

This geometric structure is not manually designed - it emerges from training a neural network to predict context. Words that appear in similar contexts end up in similar regions of the space.

How models produce them

Modern text embedding models are Transformer encoders trained with a contrastive objective: pairs of semantically similar sentences are pulled together in the embedding space; dissimilar pairs are pushed apart. The model learns to compress the full meaning of a sentence into a fixed-size vector.

• Tokenisation - the input text is split into subword tokens (using BPE or WordPiece). "embeddings" might become ["embed", "##dings"].
• Transformer encoding - the token sequence is processed through attention layers. Each token attends to all others, capturing long-range dependencies and word-sense disambiguation.
• Pooling - the per-token representations are collapsed into a single vector. Common strategies: mean pooling (average all token vectors) or using the [CLS] token representation.
• Normalisation - the output vector is L2-normalised to unit length, which makes cosine similarity equivalent to dot product and stabilises training.

The history here matters: Word2Vec (Mikolov et al., 2013) showed that word analogies could be solved with vector arithmetic - "king - man + woman = queen". Modern sentence embedding models extend this to full sentences and paragraphs, with dramatically better semantic precision.

Embedding models

Choosing the right embedding model is a key architectural decision. Latency, dimension count, maximum input length, and quality on your specific domain all vary significantly across models.

The MTEB (Massive Text Embedding Benchmark) leaderboard at Hugging Face is the authoritative comparison for embedding model quality across retrieval, clustering, and classification tasks.

Dimensionality

Dimensionality is the number of floating-point values in each embedding vector. Higher dimensions generally mean higher quality - the model has more capacity to encode subtle distinctions - but at a cost:

A 1536-dimension float32 vector takes 6 KB of storage. At 10 million documents that is 60 GB of raw vectors before the index overhead. For large datasets, two compression techniques reduce this significantly:

• Matryoshka Representation Learning (MRL) - some models (OpenAI text-embedding-3) support truncating vectors to a smaller dimension (e.g. 512) with only a modest quality drop. This is the most practical option for most teams.
• Scalar quantisation - convert float32 to int8 per dimension. 4x storage reduction, ~1% quality loss. Supported natively by Cohere and most vector databases.
• Product quantisation (PQ) - aggressively compress to 32-256 bytes per vector. Used in IVF+PQ indexes for billion-scale datasets where HNSW RAM usage is prohibitive.

Similarity and distance

Once you have two embedding vectors, measuring their similarity is a single arithmetic operation:

• Cosine similarity = dot product divided by the product of magnitudes. Returns 1.0 for identical direction, 0 for orthogonal (unrelated), -1.0 for opposite. The standard metric for text.
• Dot product = equivalent to cosine when vectors are unit-normalised (which most embedding APIs produce by default). Faster to compute; preferred in production.
• Euclidean distance (L2) = straight-line distance in the high-dimensional space. Sensitive to vector magnitude; less common for text embeddings.

Practical uses

Pitfalls

Embeddings are powerful but have failure modes that catch engineers off guard:

• Domain mismatch - a general-purpose embedding model trained on web text may produce poor results for specialised domains (legal contracts, medical notes, code). Evaluate on your specific data before committing to a model. Fine-tuned models (or domain-specific ones from Hugging Face) often outperform general models significantly on narrow domains.
• Input length truncation - most models have a token limit (256-8192 tokens). Text beyond the limit is silently truncated, producing embeddings that represent only the beginning of the document. Always chunk before embedding; never embed full documents.
• Model version lock-in - if you upgrade your embedding model, all previously stored vectors become incompatible. You must re-embed your entire corpus. Plan for this in your architecture: version your indexes, and avoid frequent model switches.
• Semantic compression loss - a single 1536-dimensional vector cannot capture every nuance of a long document. Two documents with the same general topic but opposite conclusions will have similar embeddings. For high-stakes retrieval, combine embeddings with keyword search (hybrid search) and reranking.
• Bias in the embedding space - embedding models inherit biases from training data. Stereotyped associations between demographic terms and concepts are encoded geometrically. Audit your model on bias benchmarks if your application is user-facing.

For how embeddings are stored and searched at scale, see Vector Databases. For how they power the full retrieval pipeline in production AI systems, see RAG Explained.