Reducing Hallucinations in AI Meeting Summaries and Q&A

Quick answer: Reduce hallucinations in meeting summaries and Q&A by grounding every claim in a quoted, timestamped transcript span. Clean the audio with voice activity detection, get speaker names right, extract decisions and action items with evidence quotes before writing prose, verify each claim with exact quote matching plus an NLI-style checker, and let the model say "not discussed".

A meeting summary that invents a decision is worse than no summary: someone acts on it. The errors are rarely dramatic. They look like a plausible owner who never volunteered, a deadline nobody set, a "we agreed" that was really "let's think about it", or last month's price quoted as current.

This article covers where those errors come from and how to design them out. Its companion, Evaluate a Meeting RAG System, covers how to measure them, so you can tell whether a fix worked.

What counts as a hallucination in a meeting summary?

The summarization literature separates faithfulness (is the output supported by the source?) from factuality (is it true in the world?). The study by Maynez et al. on faithfulness and factuality in abstractive summarization found hallucinated content in the output of every summarization model its annotators reviewed. For meetings, faithfulness to the transcript is the main target, but a few failure types are specific to conversations:

• Invented commitments. "Priya will send the deck by Friday" when Priya only said the deck needed updating (example).
• Wrong attribution. The right action item assigned to the wrong person because two speakers were merged.
• Hedges turned into decisions. "Maybe we should drop the free tier" summarised as "The team decided to drop the free tier."
• Stale facts. A Q&A answer that quotes a decision a later meeting reversed.
• Negation and number flips. "We are not renewing" losing its "not", or "fifteen" becoming "fifty" through a transcription error.
• Answering the unanswerable. A confident reply to a question the meetings never covered.

Why do meetings make hallucinations more likely?

Errors stack up across the pipeline before the LLM writes a word.

• Speech recognition can invent text. The Careless Whisper study found that about 1% of the Whisper transcriptions it examined contained whole hallucinated phrases or sentences not present in the audio, more often for speakers with longer pauses. Silence and background noise are common in meetings.
• Crosstalk and diarization errors. Overlapping speech produces merged or misattributed turns, and the summarizer then repeats the wrong name with full confidence.
• Spoken language is vague. "Sure, I can look at that" may or may not be a commitment. Models tend to resolve ambiguity towards something tidy and actionable.
• Long transcripts. Models use information in the middle of a long input less reliably than at the start or end (Lost in the Middle), so details from minute 35 of a one-hour call are the ones that get blurred.
• Decisions change. Across meetings, the same topic has several "true" answers depending on the date.
• Retrieval misses. In Q&A, if the right chunk is not retrieved, the model fills the gap from its priors. A Hugging Face forum thread on RAG hallucination makes the same point: poor or outdated context is often the root cause, not the model.

What are the core ideas behind grounded meeting output?

Three ideas do most of the work. First, evidence before prose: extract structured items (decisions, action items, open questions), each with the exact quote and timestamp that supports it, and only then write the summary from those items. Second, verification as a separate step: a different, cheaper model or rule checks each claim against its evidence, so the generator is not grading itself. Third, abstention is a valid output: "No owner was named" and "This was not discussed in the selected meetings" are correct answers, and the prompt and the UI must allow them.

Prompt-only self-checking helps too. Chain-of-Verification drafts an answer, asks verification questions, answers them independently and revises. SelfCheckGPT flags sentences that disagree across several sampled outputs. Both cost extra LLM calls, and neither replaces checking against the transcript itself.

How do you reduce hallucinations, step by step?

1. Clean the audio input. Run voice activity detection so silence never reaches the ASR model, and transcribe with word timestamps. Spot-check transcripts of silent or noisy sections.
2. Get speakers right. Run diarization (for example pyannote.audio) and map speaker labels to real names from the meeting platform and calendar invite. Mark low-confidence turns instead of guessing a name.
3. Extract with evidence. Ask the model for structured JSON: each decision and action item carries an evidence quote, a timestamp and the speaker. Allow null for missing owners and dates.
4. Write from the extracted items. Generate the readable summary only from validated items, not from the raw transcript, so prose cannot introduce new facts.
5. Verify every claim. Reject items whose quote does not appear in the transcript, then score each claim against its quoted window with an NLI-style checker. Drop, regenerate or flag the failures.
6. Handle time across meetings. For Q&A, retrieve with meeting dates, prefer the latest decision on a topic, and state when an earlier one was changed.
7. Publish with citations. Link every decision and answer to its meeting and timestamp so a reader can check it in seconds.

Step 1 with faster-whisper, following its README options for VAD filtering and word timestamps. The README notes that the default VAD settings only remove silence longer than two seconds, so this example shortens the threshold. Whisper is one option here; the same idea applies to any ASR model.

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

segments, info = model.transcribe(
    "design_review.wav",
    vad_filter=True,                      # drop silence before decoding
    vad_parameters=dict(min_silence_duration_ms=500),
    condition_on_previous_text=False,     # stop repeated text carrying over
    word_timestamps=True,
)

for seg in segments:
    print(f"[{seg.start:.1f}-{seg.end:.1f}] {seg.text}")

Step 3 output for one planning call might look like this (example data). Note the null due date: nobody named one, so the model must not invent one.

{
  "meeting_id": "mtg_0311_planning",
  "decisions": [
    {
      "text": "Move the billing launch to 15 April",
      "decided_by": "Dana Ruiz",
      "evidence_quote": "Tax API slipped, so we move launch to 15 April.",
      "timestamp": "00:21:40",
      "replaces": "Launch on 1 April (mtg_0304_planning)"
    }
  ],
  "action_items": [
    {
      "text": "Email the pilot accounts about the new date",
      "owner": "Luis Ortega",
      "due": null,
      "evidence_quote": "I'll email the pilot accounts today or tomorrow.",
      "timestamp": "00:23:05"
    }
  ]
}

Step 5 combines an exact quote check with Vectara HHEM-2.1-Open, which scores (premise, hypothesis) pairs between 0 (not supported) and 1 (supported) through the predict() method shown on its model card. MiniCheck is a comparable open option, scoring document and claim pairs with small fact-checking models described in the MiniCheck paper.

import re
from transformers import AutoModelForSequenceClassification

hhem = AutoModelForSequenceClassification.from_pretrained(
    "vectara/hallucination_evaluation_model", trust_remote_code=True)

def norm(s):
    return re.sub(r"\W+", " ", s.lower()).strip()

def check_item(item, transcript_window):
    # 1) The quote must exist in the transcript (cheap, exact)
    if norm(item["evidence_quote"]) not in norm(transcript_window):
        return "reject: quote not found"
    # 2) The claim must follow from the quoted passage (NLI-style model)
    score = float(hhem.predict([(transcript_window, item["text"])])[0])
    return "ok" if score >= 0.5 else f"review: support score {score:.2f}"

The 0.5 threshold is a starting point. Tune it on your own labelled meeting items, and route borderline scores to human review rather than silently dropping them.

What are the trade-offs and pitfalls?

• Recall versus precision. Strict verification removes invented items but also drops real ones phrased indirectly. Track both, or you will "fix" hallucinations by producing empty summaries.
• Quotes fail on paraphrase. Models paraphrase when asked to quote. Normalise text before matching, or ask for timestamps and pull the quote yourself from the transcript.
• Checkers have limits. NLI-style models judge one claim against one passage. A claim that needs two turns ("Dana proposed it, Luis agreed") can fail unless the window covers both.
• Latency and cost. Per-claim checks add model calls. Small checkers are cheap enough to run on every item; LLM-based self-verification usually is not.
• Diarization errors pass every check. If the transcript says the wrong person spoke, the quote matches and the checker agrees. Speaker accuracy has to be fixed upstream, covered in Speaker Diarization for Meeting AI.
• Transcription quirks. Practitioners in a Hugging Face forum thread on Whisper hallucinations compare settings such as timestamps; test these against your own audio rather than assuming a fix carries over.
• Sensitive content. Verification logs contain quotes from private meetings. Store them under the same access rules as the transcripts.

How do you measure whether it worked?

Use a labelled set of meetings and score: the share of summary claims supported by the transcript (a FActScore-style atomic fact precision, with the transcript as the knowledge source), owner and date accuracy for action items, decision recall, the rate of invented owners or dates, and abstention accuracy on unanswerable questions. For Q&A, add retrieval recall and faithfulness, as described in the companion evaluation guide.

Public benchmarks help you choose components. The Vectara hallucination leaderboard ranks LLMs by hallucination rate on document summaries, and SelfCheckGPT's GitHub repository includes ready-to-use checkers. Neither uses meeting transcripts, so treat them as a shortlist and confirm on your own calls.

How does Acta approach this?

Acta transcribes after the meeting (not live) and produces transcripts with speaker names, taken from the meeting and the calendar invite, plus timestamps. Summaries include decisions and action items with owners. The decision log records who decided, links action items, and keeps the history when a later meeting changes or reverses a decision, with the quote and the moment it was said, which addresses the stale-fact problem directly. Ask Acta answers questions across meetings with retrieval-augmented generation over their transcripts and cites the meeting, and AI assistants connected to Acta only see meetings the user can see. For the citation side in more depth, see Give AI Answers Verifiable Citations from Meeting Transcripts.

Frequently asked questions

Can a better prompt alone stop meeting hallucinations?

No. Prompts that ask for quotes and allow "not discussed" help, but they cannot fix invented ASR text, merged speakers or a missed retrieval. You need checks outside the generating model.

Should the summary be generated from the full transcript or from retrieved chunks?

For a single meeting that fits the context window, extract from the full transcript in sections so nothing in the middle is skipped. For questions across many meetings, retrieval is required, and the verification step matters more.

Is a low hallucination-leaderboard score enough to pick an LLM?

It is a useful filter, but those leaderboards summarise written documents. Meeting transcripts are noisier and longer, so rerun your own labelled set before switching models.

What should the system do when the checker flags a claim?

Drop it from the summary if it is minor, regenerate it from the evidence if it is a decision or action item, and show it for human review when the score is borderline. Never publish it unmarked.