Build RAG Over Meeting Transcripts and Documents: A Practical Architecture

Quick answer: Split transcripts on speaker turns, keep the meeting title, date, speakers and timestamps on every chunk, and index each chunk twice: full-text (BM25) and vectors. At query time, filter to the meetings the user may see, run both searches, fuse them with Reciprocal Rank Fusion, rerank with a cross-encoder, and make the model cite the meeting and timestamp for every claim.

Retrieval-augmented generation (RAG) means fetching the right passages first and asking a language model to answer from them, an idea formalised in Lewis et al.'s RAG paper. Over meetings, the plan is the same, but most of the work is in the data: what a chunk is, which metadata it carries and who is allowed to see it. This article sets out an architecture that holds up for questions like "What did we agree with the vendor about the SLA?" across months of calls and the documents shared around them.

If you are still deciding whether you need RAG at all, start with the companion piece, RAG vs. fine-tuning vs. prompting for meeting assistants. This one assumes you have decided and want to build it.

Why is RAG over meetings harder than RAG over documents?

Documents are written to be read. Meetings are spoken, and the transcript carries every side effect of that:

• Spoken language. People say "let's do the second option" or "yeah, that one" instead of naming the thing. The words that make a passage retrievable are often in a turn two minutes earlier.
• Crosstalk and ASR errors. Overlapping speech gets merged or dropped, and product names, customer names and acronyms are misheard. A query for "Kubernetes" may need to match "cube in eighties".
• Speaker attribution. "Who promised the pricing deck?" is only answerable if the transcript knows who said what. Diarization tools such as pyannote.audio separate voices; mapping them to real names needs meeting and calendar data.
• Long transcripts. An hour-long call is far more text than any single answer needs, and putting all of it in the prompt does not guarantee the model uses it: Lost in the Middle found that models use information at the start and end of long inputs better than information in the middle.
• Decisions change. A launch date agreed in March may be moved in April. Retrieval that returns the most similar passage, not the latest, gives a confident wrong answer.
• Sensitive content. Board meetings, 1:1s and hiring panels sit in the same corpus as stand-ups. Access control has to be part of retrieval, not a check after generation.

What are the core concepts?

• Chunk: the unit you retrieve. For transcripts, a window of consecutive speaker turns; for documents, a section under a heading. The chunking guide in this series covers window sizes and overlap in detail.
• Context header: a short line prepended to each chunk before indexing, such as meeting title, date and speakers, so that a chunk saying "we'll push it a week" still matches a query about "the Q3 launch".
• Lexical retrieval (BM25): exact-term matching, strong for names, ticket numbers, prices and acronyms. rank_bm25 is enough for prototypes; in production, Postgres full-text search or a search engine does the same job.
• Dense retrieval: embeddings that match meaning across paraphrase ("pushed back" vs. "delayed"). sentence-transformers produces them; pgvector or FAISS stores and searches them. Compare embedding models on the retrieval tasks of the MTEB leaderboard, then test on your own transcripts.
• Fusion: merging the two ranked lists. Reciprocal Rank Fusion (Cormack et al.) scores each item by summing 1/(k + rank) across lists, with k = 60 in the paper, so it needs no score normalisation.
• Reranker: a cross-encoder that reads the query and a candidate together and scores relevance. The retrieve and re-rank pattern uses a fast retriever for candidates and a slower, more accurate cross-encoder on the shortlist.
• Grounded, cited answer: the model answers only from the retrieved chunks and points to its sources. ALCE is the standard reference for measuring citation quality.

What architecture should you build, step by step?

1. Ingest and normalise. For each meeting, store the transcript as a list of turns (speaker, start and end time, text) plus metadata: title, date, attendees, series, and the access list. Parse shared documents (agenda, PRD, slides) into sections and link them to the meeting they belong to.
2. Chunk on speaker turns. Group consecutive turns into windows (for example 6-12 turns) with a small overlap, never cutting inside a turn. Keep the first and last timestamps. For documents, chunk by heading.
3. Add context headers and index twice. Prepend "Meeting title | date | speakers" to the chunk text, then write it to a full-text index and a vector index. Store the meeting ID, timestamps and access list as filterable columns.
4. Parse the query. Pull out explicit filters ("last week", "with Acme", "in the pricing reviews") and turn them into metadata filters. Keep the remaining text as the search query.
5. Filter, then retrieve. Restrict candidates to meetings the user can see and to the date or series filters, then run BM25 and vector search (top 50 each) and fuse with RRF.
6. Rerank and assemble context. Rerank the fused list with a cross-encoder, keep the top 5-10, merge neighbouring chunks from the same meeting, and order them by meeting date so later decisions appear after earlier ones.
7. Generate with citations. Give each chunk an ID such as [M3 14:05], instruct the model to cite an ID after every claim, to prefer the most recent statement when sources conflict and to say so, and to answer "not found in these meetings" when the context does not support an answer.
8. Check citations before showing the answer. Verify that every cited ID was in the context, and optionally that a quoted phrase actually appears in that chunk. Log query, chunk IDs and answer for evaluation.

Here is what one transcript chunk might look like in storage. The meeting, people and content are an invented example.

{
  "chunk_id": "mtg_0412_c07",
  "meeting_id": "mtg_0412",
  "meeting_title": "Vendor review: Northwind Logistics",
  "date": "2026-04-12",
  "series": "vendor-reviews",
  "start": "00:14:05",
  "end": "00:16:40",
  "speakers": ["Priya Shah", "Tom Becker"],
  "allowed_groups": ["ops-leads", "procurement"],
  "text": "Vendor review: Northwind Logistics | 2026-04-12 | Priya Shah, Tom Becker\nPriya Shah: If they miss the 48-hour SLA again we move the EU lane to the backup carrier.\nTom Becker: Agreed. I'll send them the written notice on Friday.",
  "source_type": "transcript"
}

And a compact version of the retrieval path, using the APIs shown in the rank_bm25 and sentence-transformers READMEs. It assumes chunks has already been filtered to meetings the user can see.

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder

texts = [c["text"] for c in chunks]  # chunks: permission-filtered list of dicts
bm25 = BM25Okapi([t.lower().split(" ") for t in texts])
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
doc_emb = embedder.encode(texts)
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, idx in enumerate(ranking):
            scores[idx] = scores.get(idx, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

def retrieve(query, n=50, top=8):
    lexical = bm25.get_scores(query.lower().split(" ")).argsort()[::-1][:n].tolist()
    sims = embedder.similarity(embedder.encode([query]), doc_emb)[0]
    dense = sims.argsort(descending=True)[:n].tolist()
    fused = rrf([lexical, dense])[:n]
    ranked = reranker.rank(query, [texts[i] for i in fused], top_k=top)
    return [chunks[fused[r["corpus_id"]]] for r in ranked]

In production you would move both indexes into one store. The pgvector README suggests using it together with Postgres full-text search for hybrid search, combined with Reciprocal Rank Fusion or a cross-encoder. Its hnsw index with vector_cosine_ops handles the vector side; a generated tsvector column handles the lexical side.

What are the trade-offs and pitfalls?

• Filtering after approximate search loses results. The pgvector README notes that with approximate indexes, filtering happens after the index scan, so a selective WHERE clause can return fewer rows than you asked for. It recommends iterative index scans. For per-user permissions, check that you still get k results.
• Chunks that are too small lose the referent. "Yes, let's do that" retrieves nothing useful on its own. Windows of several turns plus a context header fix most of this.
• Chunks that are too large dilute the embedding and waste context. Test two or three window sizes against your evaluation set rather than guessing.
• Similarity is not recency. For decision questions, sort the final context by date and tell the model which statement is latest. A dedicated decision store, covered in the multi-meeting article of this series, is more reliable than hoping retrieval finds the reversal.
• ASR errors hurt BM25 more than embeddings, and embeddings miss rare names more than BM25. That asymmetry is the main argument for hybrid retrieval. A glossary of names and products fed to transcription or used for query expansion helps both.
• Rerankers cost latency. A cross-encoder scores every candidate pair, so cap the shortlist (around 50). For multilingual meetings, try a multilingual reranker such as BAAI/bge-reranker-v2-m3.
• Documents and transcripts compete. A polished PRD often outranks the messy call where the plan changed. Tag source_type and, for "what did we decide" questions, boost transcripts from the latest meetings.
• Citations can be invented. Models sometimes cite a chunk that does not support the claim. Check IDs mechanically and spot-check support; the citations guide goes deeper.

How do you evaluate it?

Build a test set of 50-200 real question types before tuning anything: single-meeting facts ("What budget did Tom quote?"), cross-meeting questions ("How did the launch date change?"), people questions ("What does Priya owe us?") and unanswerable questions. For each, record the meeting and timestamp that hold the answer.

• Retrieval: recall@k (is the gold chunk in the top k?), MRR or nDCG for ranking quality. Measure BM25, dense and hybrid separately so you can see what each adds.
• Answers: Ragas defines reference-free metrics; its metric list includes faithfulness, context precision, context recall and response relevancy. The library is on GitHub.
• Citations: citation precision (does each cited chunk support the claim?) and recall (is each claim cited?), in the style of ALCE.
• Abstention: the share of unanswerable questions where the system correctly says it did not find the answer.
• Latency and cost per query, split by retrieval, reranking and generation.

Slice every metric by question type. Averages hide the fact that cross-meeting and "who said" questions usually fail first. The evaluation article in this series walks through building the test set.

How does Acta approach it?

Ask Acta answers questions across the meetings and files you choose using retrieval-augmented generation over their transcripts, and its answers cite the meeting. Transcripts carry speaker names, taken from the meeting and the calendar invite, and timestamps, which is the metadata the architecture above depends on. The decision log tracks decisions across meetings and keeps the history when a later meeting changes or reverses one, with the quote and the moment it was said. Transcription happens after the meeting, not live. When you connect an AI assistant to Acta, it only sees meetings you can see. More on the product side: ask questions across all your meetings.

Frequently asked questions

Do I need a vector database, or is Postgres enough?

For most meeting corpora, Postgres with pgvector and built-in full-text search is enough, and it keeps permissions, metadata and both indexes in one place. FAISS is a good choice when you need an in-process index or very large-scale similarity search.

Should I embed whole meetings or individual turns?

Neither. Single turns are too short to carry meaning ("sounds good"), and whole meetings blur many topics into one vector. Windows of several consecutive turns with a context header work better; you can store a meeting-level summary as an extra chunk for broad questions.

How many chunks should go into the prompt?

Start with 5-10 reranked chunks and tune against your test set. More context is not automatically better, because models use the middle of long inputs less reliably.

How do I handle meetings and documents together?

Index both in the same store with a source_type field and a link from each document to its meeting. Let the reranker choose, but tell the model which sources are transcripts and which are documents, and prefer the most recent transcript when they disagree.

Where can I learn more about rerankers?

See the earlier guide build a reranker for search and RAG, and the sentence-transformers documentation on retrieve and re-rank.