Evaluate a Meeting RAG System: Retrieval Quality, Answer Accuracy, and Hallucinations

Quick answer: Evaluate a meeting RAG system in three layers. Score retrieval with recall@k and nDCG against hand-labelled transcript chunks, score answers against reference answers for correctness, and score grounding with claim-level faithfulness and citation checks. Build the test set from real meeting questions, including changed decisions, speaker questions and unanswerable ones, and rerun it on every pipeline change.

A RAG demo over meeting transcripts can look right for a week and then fail the first time someone asks "what did we decide about pricing?" after pricing changed twice. The only reliable defence is a test set and a set of numbers you track every time you change the chunker, the embedding model, the prompt or the LLM.

This article covers how to measure a meeting RAG system. Its companion, Reducing Hallucinations in AI Meeting Summaries and Q&A, covers how to fix the failures these measurements find. For the system being measured, see Build RAG Over Meeting Transcripts and Documents.

Why is evaluating meeting RAG harder than evaluating document RAG?

Most RAG evaluation advice assumes clean documents with stable facts. Meetings break those assumptions in ways that change what you have to measure:

• Spoken language. People say "let's go with the second one" or "yeah, that works for me". The relevant chunk often contains none of the words in the question, so lexical overlap between question and gold chunk is low and naive retrieval scores look worse than they are, or better than they should.
• ASR errors and crosstalk. A product name transcribed wrongly ("Kubernetes" as "cube in eighties") breaks keyword search, and overlapping speech can merge two people's sentences. Your test set has to include questions that touch these noisy passages, not only clean ones.
• Speaker attribution. "Who agreed to send the contract?" is only answerable if the transcript says who spoke. A correct answer with the wrong owner is a wrong answer, and generic metrics will not catch it unless you test for it.
• Long transcripts. A one-hour call can run to many thousands of words. Research on long contexts found models use information at the start or end of the input better than information in the middle (Lost in the Middle, Liu et al.), so where the answer sits in the context matters.
• Decisions change across meetings. The launch date set on 4 March can be moved on 11 March. A system that retrieves the older chunk and answers confidently is wrong even though its answer is "faithful" to what it retrieved.
• Sensitive content. HR, board and deal calls must never leak into another user's answers. Access control is part of correctness, so it belongs in the test set.

What are the core metrics, and what does each one tell you?

Separate the pipeline into the parts you can fix independently, and give each part its own number.

• Retrieval: recall@k. Of the chunks that contain the answer, how many appear in the top k? If recall@10 is low, no prompt will save the answer.
• Retrieval: MRR and nDCG@k. How high the first relevant chunk ranks, and how well the whole ranking follows graded relevance. These matter because of the position effect above and because you pay for every token you pass to the LLM.
• Answer correctness. Does the answer match a reference answer? For short facts (a date, an owner) use exact or normalised match. For longer answers, use claim-level comparison such as the factual correctness metric in Ragas, or a calibrated LLM judge.
• Faithfulness (groundedness). Is every claim in the answer supported by the retrieved context? This is the main automated hallucination signal. It does not tell you whether the context was the right one, which is why you also need correctness.
• Citation quality. Does each cited meeting actually support the sentence it is attached to, and is every factual sentence cited? The ALCE benchmark paper splits this into citation recall and citation precision, a useful framing for meeting answers too.
• Abstention. On questions the meetings do not answer, does the system say so instead of guessing?
• Permission leakage. Count of answers or citations that draw on a meeting the asking user cannot see. The target is zero, not "low".

The Ragas paper and ARES both frame RAG evaluation along similar lines: context relevance, faithfulness and answer relevance. RAGChecker goes further with claim-level diagnostics for the retriever and the generator separately.

How do you build a test set from meetings?

Public meeting datasets help you sanity-check a pipeline before you have your own data. QMSum pairs queries with summaries of relevant meeting spans, and MeetingBank aligns city council transcripts with segment-level minutes. Neither looks like your company's standups or sales calls, so the test set that matters is the one you write from your own (consented, access-controlled) meetings.

Aim for a mix of question types, because each one exercises a different failure:

• Single-meeting facts: "What budget did finance approve in the Q3 planning call?" (example)
• Speaker questions: "Who said they would draft the security questionnaire?" (example)
• Changed decisions: "What is the current launch date?" where two meetings disagree.
• Cross-meeting synthesis: "What objections has Northwind raised across all calls?" (example)
• Noisy passages: questions whose answer sits in crosstalk or a misheard term.
• Unanswerable: plausible questions the meetings never cover.
• Permission traps: questions whose best evidence is in a meeting the asking user is not allowed to see.

Label at the chunk level, with graded relevance, and record the date the question is asked "as of". Without an as-of date, a changed-decision question has no single correct answer.

How do you run the evaluation, step by step?

1. Collect questions. Pull real questions from logs or interviews, then add the hard types above until each has at least a handful of examples.
2. Label gold evidence. For each question, mark the transcript chunks (meeting id plus speaker turn) that answer it, with grades 2 (answers it) and 1 (useful context), and write a short reference answer.
3. Freeze the corpus. Snapshot transcripts and chunk ids so scores stay comparable between runs. If you change the chunker, re-map the gold labels by time span, not by chunk id.
4. Score retrieval. Run every question through the retriever as the asking user, then compute recall@k, MRR and nDCG@k against the gold chunks.
5. Score answers and grounding. Generate answers, then compute correctness against the reference, faithfulness against the retrieved context, citation precision and recall, and abstention on unanswerable questions.
6. Check permissions. Fail the run if any answer or citation uses a meeting outside the asking user's access list.
7. Gate releases. Store results per question type, compare with the last run, and block a change that improves the average but regresses changed-decision or permission questions.

A gold item for a changed-decision question might look like this (example data). The graded gold_chunks feed the retrieval metrics, and the must_cite and must_not_cite fields feed the citation and permission checks.

{
  "id": "q-017",
  "type": "changed_decision",
  "asked_by": "user_dana",
  "question": "What is the launch date for the billing revamp?",
  "as_of": "2026-03-20",
  "gold_chunks": {
    "mtg_0311_planning#t042": 2,
    "mtg_0304_planning#t015": 1
  },
  "reference_answer": "15 April. It was moved from 1 April in the 11 March planning call because the tax API was late.",
  "must_cite": ["mtg_0311_planning"],
  "must_not_cite": ["mtg_0218_exec_offsite"],
  "answerable": true
}

For retrieval metrics, ranx takes relevance labels (qrels) and retriever scores (a run) as plain dictionaries, following its README quickstart:

from ranx import Qrels, Run, evaluate

# Graded labels: 2 = answers the question, 1 = useful context
qrels = Qrels({
    "q-017": {"mtg_0311_planning#t042": 2, "mtg_0304_planning#t015": 1},
    "q-018": {"mtg_0225_design#t088": 2},
})

# Scores your retriever returned for each chunk id
run = Run({
    "q-017": {"mtg_0304_planning#t015": 0.82, "mtg_0311_planning#t042": 0.79,
              "mtg_0128_standup#t003": 0.41},
    "q-018": {"mtg_0225_design#t091": 0.66, "mtg_0225_design#t088": 0.61},
})

print(evaluate(qrels, run, ["recall@10", "mrr", "ndcg@10"]))

In this example, q-017 ranks the outdated 4 March chunk above the 11 March chunk that holds the current answer. Recall@10 is perfect and MRR stays at 1.0, because the older chunk is labelled as useful context (grade 1). Only graded nDCG drops, which is why graded labels matter for changed decisions. For q-018, an unlabelled neighbouring turn outranks the answer, so MRR falls as well.

For answer-level metrics, Ragas scores context recall, faithfulness and factual correctness with a judge LLM. This snippet follows the EvaluationDataset and evaluate() pattern in the Ragas 0.3 docs. Newer releases move metrics to a collections API, so check the current Ragas guide for your installed version.

from ragas import EvaluationDataset, evaluate
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness

rows = [{
    "user_input": "What is the launch date for the billing revamp?",
    "retrieved_contexts": [
        "[Planning, 11 Mar, 00:21:40] Dana: Tax API slipped, so we move launch to 15 April.",
        "[Planning, 4 Mar, 00:06:12] Luis: We are still on track for 1 April.",
    ],
    "response": "The launch moved to 15 April (Planning, 11 March).",
    "reference": "15 April, moved from 1 April on 11 March because the tax API was late.",
}]

dataset = EvaluationDataset.from_list(rows)
result = evaluate(
    dataset=dataset,
    metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],
    llm=evaluator_llm,  # your judge model, wrapped as the Ragas docs show
)
print(result)

What are the trade-offs and pitfalls?

• LLM judges have biases. The LLM-as-a-judge study by Zheng et al. documents position, verbosity and self-enhancement bias. Calibrate any judge on a few dozen human-labelled meeting answers before you trust it, and avoid judging a model with itself.
• Faithful is not the same as correct. An answer that faithfully repeats the superseded 4 March date scores well on faithfulness. Always pair faithfulness with correctness against an as-of reference.
• Averages hide the failures that matter. Ten easy fact questions can drown out two wrong owners. Report per question type.
• Public leaderboards are a starting point. The MTEB leaderboard is useful for shortlisting embedding models (use its retrieval view), but its datasets are not meeting transcripts, and the MTEB best-practices guide warns that small score gaps may not be significant. Rerun the shortlist on your own gold set.
• Test sets go stale. When the product starts handling new meeting types (board meetings, interviews), add questions for them.
• Labels contain sensitive data. A gold set built from real meetings needs the same access controls as the meetings themselves.

How do you know the evaluation itself is trustworthy?

Evaluate the evaluator. Have two people label a sample of the same questions and look at where they disagree: usually the question is ambiguous or the as-of date is missing. For the judge, compute agreement with human labels on faithfulness and correctness, and re-check it whenever you change the judge model or prompt. ARES uses a small human-labelled set to correct judge errors statistically, an idea worth borrowing if your test set grows past what you can read by hand.

Track a few production signals alongside offline scores: how often users open a citation, how often they rephrase the same question, and how often answers say "not discussed". A spike in rephrasing after a release is often the first sign of a retrieval regression that the offline set did not cover. When offline scores catch a hallucination, the fixes are in the companion article on reducing hallucinations, and citation-specific checks are covered in Give AI Answers Verifiable Citations from Meeting Transcripts.

How does Acta approach this?

The behaviours this test set probes are the ones Acta is built around. Ask Acta answers questions across the meetings and files you choose using retrieval-augmented generation over their transcripts, and its answers cite the meeting, so a reader can check the source. Transcripts carry speaker names, taken from the meeting and the calendar invite, plus timestamps, which is what speaker questions depend on. The decision log keeps the history when a later meeting changes or reverses a decision, with the quote and the moment it was said, which is the changed-decision case above. AI assistants connected to Acta only see meetings the user can see. Transcription happens after the meeting, not live.

Frequently asked questions

How many test questions do I need?

Enough to see each question type fail. Start with a few dozen covering every type in the list above, read every failure by hand, and grow the set when a production bug shows a gap. A small set you read is worth more than a large set you only average.

Can I evaluate without reference answers?

Partly. Faithfulness and context relevance can be judged from the question, context and answer alone, which is what Ragas was designed for. Correctness on changed decisions needs a reference with an as-of date, because the retrieved context itself may be outdated.

Should I use recall@5 or recall@20?

Use the k that matches how many chunks you actually pass to the LLM after reranking. If you pass eight chunks, recall@8 is the number that predicts answer quality.

What is the difference between faithfulness and hallucination rate?

Faithfulness is the share of answer claims supported by the retrieved context. A hallucination rate usually counts answers with at least one unsupported claim. Both are useful, and RAGChecker reports a separate hallucination metric alongside faithfulness, listed in its GitHub README.