How to Build Semantic Search With Embeddings and Open Models (2026 Guide)

Quick answer: Embedding models turn text into fixed-length vectors so that texts with similar meaning end up close together. That powers semantic search, deduplication, clustering and the retrieval step of RAG. The shortest path: encode your documents with an open model such as Qwen3-Embedding, bge-m3 or EmbeddingGemma through sentence-transformers, store the vectors in an index, and rank by cosine similarity.

Keyword search breaks when users and documents use different words for the same thing. Someone types "cancel my plan" and the help article says "end your subscription". Embeddings close that gap, which is why they sit under most modern search boxes, recommendation features and RAG chatbots.

This guide covers two closely related Hugging Face tasks: feature extraction (producing the vectors) and sentence similarity (comparing them). It shows which open models to use, working code, how to serve embeddings at scale, and how to measure retrieval quality.

What are embeddings and sentence similarity?

The feature extraction task page covers models that convert raw input into numerical features. For text, the output is one vector per input, typically a few hundred to a few thousand dimensions. The sentence similarity task page covers the step on top: given a source sentence and a list of candidates, score how close each candidate is in meaning, usually with cosine similarity.

Most text embedding models are bi-encoders. A transformer encodes each text on its own, then a pooling step (mean pooling, the CLS token, or the last token for LLM-based models) reduces the token vectors to one sentence vector. The model is trained contrastively, so that a query and its relevant passage score higher than unrelated pairs. The approach comes from the Sentence-BERT paper and is implemented in the sentence-transformers library.

Because documents are encoded independently of the query, you compute document vectors once and store them. At query time you embed only the query and run a nearest-neighbour search, which is what makes embeddings fast enough to search millions of passages.

Some models do more than produce one dense vector. bge-m3, for example, supports dense, sparse (lexical weights) and multi-vector (ColBERT-style) retrieval from one model, and several newer models accept a task instruction on the query side.

What can you build with it?

• Semantic search over help centres, wikis, tickets or product catalogues.
• The retrieval step of RAG: find the passages an LLM should read before answering.
• Duplicate and near-duplicate detection for support tickets, bug reports or forum questions.
• Clustering and topic discovery in feedback, survey answers or call notes.
• Recommendations such as "related articles" or "similar products" based on text content.
• Cross-meeting search: find every discussion of a topic across months of transcripts, even when people phrased it differently.

Which models should you use?

These open models cover small and fast through large and accurate, English-only and multilingual. Check each model card for its licence, maximum input length and recommended prompts.

• sentence-transformers/all-mpnet-base-v2: a long-standing English baseline for sentences and short paragraphs, suited to clustering and semantic search. Easy to run on CPU.
• BAAI/bge-small-en-v1.5: a small English BGE model for low-latency or high-volume workloads. The FlagEmbedding repo (MIT-licensed code) also lists base and large versions.
• google/embeddinggemma-300m: a compact Gemma-based embedding model, supported by Text Embeddings Inference. Review the licence on the card before use.
• intfloat/multilingual-e5-large-instruct: a multilingual model that takes a task instruction on the query side.
• BAAI/bge-m3: multilingual (100+ languages), inputs up to 8,192 tokens, and dense, sparse and multi-vector retrieval in one model. Described in the BGE M3 paper.
• Qwen/Qwen3-Embedding-0.6B: 0.6B parameters, 32K context, 1,024-dimensional vectors, instruction-aware, with Matryoshka (MRL) support for shorter vectors. The Qwen3-Embedding repo reports 64.33 mean task score on MTEB Multilingual.
• Qwen/Qwen3-Embedding-4B and Qwen/Qwen3-Embedding-8B: larger members of the family with 2,560 and 4,096 dimensions; the repo reports 69.45 and 70.58 on MTEB Multilingual. See the Qwen3 Embedding paper.

Compare models on the MTEB leaderboard, filtering by language, task type (retrieval, clustering, STS) and model size. You can also browse sentence-transformers models on the Hub sorted by downloads.

How do you build it, step by step?

1. Chunk the documents. Split them into passages that fit the model's input length and still make sense alone, for example by heading or paragraph with a small overlap. Keep the source ID and position with each chunk.
2. Pick a model and embed the corpus. Encode all chunks in batches on a GPU if you can. Use the model's document prompt (if any) here and the query prompt at search time.
3. Store the vectors. For tens of thousands of chunks, an in-memory matrix is enough. For millions, use a vector index or database with approximate nearest-neighbour search, and store metadata for filtering.
4. Search. Embed the query, retrieve the top 20-100 nearest chunks by cosine similarity, and apply metadata filters (tenant, date, access rights).
5. Optionally rerank. Pass the candidates through a cross-encoder reranker to fix the order before showing results or sending them to an LLM. Our reranker guide covers this step.
6. Evaluate. Measure recall@k and nDCG@10 on a set of real queries with known relevant chunks, then tune chunking, model and k.

The core of the search step with sentence-transformers:

from sentence_transformers import SentenceTransformer

# Follows the Qwen3-Embedding README (sentence-transformers usage);
# the README requires transformers>=4.51.0 and sentence-transformers>=2.7.0
model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")

queries = ["How do I rotate an API key?"]
documents = [
    "API keys can be rotated from Settings > Security. Old keys stop working after 24 hours.",
    "Our office is closed on public holidays.",
    "To invite a teammate, open Workspace > Members and click Invite.",
]

# Queries use the built-in "query" prompt; documents are encoded without one
query_embeddings = model.encode(queries, prompt_name="query")
document_embeddings = model.encode(documents)

scores = model.similarity(query_embeddings, document_embeddings)  # cosine by default
best = scores[0].argmax().item()
print(documents[best], float(scores[0][best]))

For a full in-memory search loop over a larger corpus, the sentence-transformers semantic search example shows batched top-k retrieval. The same encode and similarity calls work for bge-m3, all-mpnet-base-v2 and most other models; only the prompt handling differs, so read each model card.

How do you serve and deploy it?

Embedding has two workloads: a one-off or periodic bulk job over the corpus, and low-latency query encoding. Plan for both.

• Text Embeddings Inference (TEI). TEI is a dedicated server for embedding, reranking and sequence classification models, with Docker images for NVIDIA GPUs and CPUs (x86_64 and arm64). The README's quick start looks like this:

# Follows the text-embeddings-inference README (adjust the image tag to your GPU)
model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data

docker run --gpus all -p 8080:80 -v $volume:/data --pull always \
  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

curl 127.0.0.1:8080/embed \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

• Pick the right image. TEI publishes separate images per GPU generation and notes that GPUs below CUDA compute capability 7.5 (V100, GTX 1000 series) are not supported. See the TEI quick tour.
• Shrink the vectors. Models with Matryoshka training let you keep only the first dimensions with a modest quality loss; see the Matryoshka embeddings post. Binary or int8 quantization cuts storage further, as described in the embedding quantization post.
• CPU serving. Small models such as bge-small-en-v1.5 or all-mpnet-base-v2 handle query encoding on a CPU; sentence-transformers also offers ONNX and OpenVINO backends through optional extras.
• Keep the model fixed. Vectors from different models (or different versions of one model) are not comparable. Store the model ID with the index and re-embed everything when you switch.

How do you evaluate it?

• Retrieval metrics: recall@k (did the relevant chunk make the shortlist), MRR and nDCG@10 (is it near the top).
• Similarity metrics: Spearman or Pearson correlation between cosine scores and human similarity ratings, as in the STS benchmark.
• Clustering and classification: V-measure for clusters, accuracy or F1 for a classifier on top of the embeddings.

The MTEB library runs all of these across many datasets, and you can add your own. It is described in the MTEB paper and the multilingual extension in the MMTEB paper. A minimal run from its README:

import mteb

# Follows the MTEB README: evaluate any sentence-transformers model on chosen tasks
model = mteb.get_model("sentence-transformers/all-MiniLM-L6-v2")
tasks = mteb.get_tasks(tasks=["Banking77Classification.v2"])
results = mteb.evaluate(model, tasks=tasks)
print(results)

Leaderboard averages are a shortlist, not a decision. Build a small test set of 50-200 real queries from your own users and compare two or three models on it before you commit.

Pitfalls and tips

• Prompts and prefixes. Many retrieval models expect a query instruction or prefix. Leaving it out, or adding it to documents by mistake, quietly lowers quality.
• Chunk size. Long chunks dilute the vector; tiny chunks lose context. Test two or three sizes on your evaluation set.
• Exact terms. Dense vectors can miss product codes, names and error strings. Hybrid search (BM25 plus embeddings) or bge-m3's sparse output helps.
• Normalisation. If your index uses dot product, normalise vectors so the scores equal cosine similarity.
• Access control. Filter by permissions before results reach the user or the LLM; similarity search ignores who is allowed to see what.
• Licences and privacy. Licences vary between model families. Self-host when the corpus contains customer or internal data.

Where to get help

• MTEB GitHub Discussions for benchmark and leaderboard questions.
• The sentence-transformers, FlagEmbedding and Qwen3-Embedding repositories for library and model issues.
• The MTEB leaderboard Space to compare current models.

Frequently asked questions

What is the difference between feature extraction and sentence similarity?

Feature extraction produces the vectors. Sentence similarity compares them, usually with cosine similarity, to score how close two texts are in meaning. In practice the same embedding model does both.

Do I need a vector database?

Not at first. A few hundred thousand vectors fit in memory and can be searched with a matrix product. Move to a vector index when the corpus, update rate or filtering needs grow.

Should I use embeddings or a reranker?

Both, in most search and RAG systems. Embeddings retrieve a shortlist quickly; a cross-encoder reranker then orders that shortlist more precisely.

Can I fine-tune an embedding model on my own data?

Yes. sentence-transformers supports training with pairs or triplets of related texts. Fine-tuning helps most when your domain uses vocabulary the base model rarely saw.

Acta uses this kind of retrieval to answer questions across many meetings; read how we approached multi-meeting RAG or see asking questions across meetings.