From Live Transcription to Searchable Meeting Intelligence: A System Design Guide

Quick answer: Treat live transcription and the meeting record as two different products. Stream audio through a low-latency ASR for captions if you need them, but build the searchable record from a full-context pass after the meeting: ASR with word timestamps, diarization and speaker names, LLM extraction of decisions and action items, then speaker-turn chunks indexed for hybrid search with permissions and citations.

Most teams start with "transcribe the call" and end up needing much more: who said it, what was decided, what changed since last week, and an answer to "what did we promise the customer about SSO?" six months later. This guide lays out a system design that gets from raw audio to searchable meeting intelligence, and explains where live pipelines fit and where they do not. For the speaker attribution step in detail, read the companion article on speaker diarization for meeting AI.

Why is meeting audio hard to turn into searchable knowledge?

Consider an example (invented) quarterly planning call: eight people, 70 minutes, three of them sharing a room microphone, one joining from a train. Each stage of the pipeline meets a different problem.

• Spoken language is not text. False starts, filler words and half sentences ("so we, I mean, the plan is, yeah, Q3") make both ASR and retrieval harder than on documents.
• ASR errors compound. A misheard product name ("Atlas" as "at last") breaks keyword search and confuses the LLM that extracts decisions.
• Attribution drives meaning. "I'll take that" is an action item only if you know who said it. Overlap and shared room microphones make that hard.
• Transcripts are long. A 70-minute meeting produces thousands of words, and a question usually concerns one or two minutes of it.
• Facts change across meetings. The launch date set in March is moved in April and cancelled in May. Search has to return the latest state and its history, not the first match.
• Content is sensitive. Board, HR and customer calls cannot leak into another user's answers.

What are the core building blocks?

• Capture: a meeting bot, a desktop or browser recorder, or file upload. Whatever the source, write the raw audio to durable storage before anything else.
• Streaming ASR: decodes audio in small chunks and emits partial text that may be revised. Policies such as LocalAgreement decide when text is stable enough to show.
• Batch ASR: transcribes the full recording with full context, usually with VAD segmentation and batching for throughput.
• Diarization and naming: anonymous speaker turns, then real names from participant data.
• Extraction: an LLM turns the transcript into a summary, decisions and action items, each linked back to the transcript span it came from.
• Indexing and retrieval: chunks, keyword and vector indexes, metadata filters, and a retrieval-augmented generation layer that answers with citations.

What are your options for live transcription?

If your product shows captions or live notes during the call, you need a streaming path. Open options include:

• Whisper-Streaming: turns Whisper into a streaming system using the LocalAgreement policy. Its paper reports 3.3 seconds latency on unsegmented long-form speech. The authors now point to SimulStreaming, which uses the AlignAtt policy and which its README describes as about five times faster.
• WhisperLiveKit: a self-hosted server with a WebSocket endpoint that combines SimulStreaming (or LocalAgreement), VAD and Sortformer-based live diarization.
• NVIDIA's Nemotron Speech streaming model: an English cache-aware FastConformer RNNT model with configurable chunk sizes from 80 ms to 1120 ms, so you can trade latency for accuracy. For live speaker labels, NVIDIA's Streaming Sortformer post describes a matching diarizer.
• sherpa-onnx: streaming and non-streaming ASR, VAD and speaker diarization that run locally without an internet connection, including on mobile and embedded devices.

Live text is a draft. Streaming models see little future context, revise their own output and have to commit to speaker labels early. That is acceptable for captions; it is a weak foundation for the permanent record that action items, decisions and search depend on.

How do you design the pipeline, step by step?

1. Capture and persist. Write raw audio (per-participant tracks if available) and meeting metadata (title, attendees from the calendar invite, owner, access list) to storage. Emit a meeting.recorded event. Everything downstream can be re-run from this.
2. Optional live path. If you show live captions, fork the audio stream to a streaming ASR service. Store its output as draft, never as the record.
3. Batch transcription after the meeting. A worker picks up the event and runs full-context ASR with word timestamps, for example faster-whisper with VAD filtering or WhisperX. Compare candidate models on the Open ASR Leaderboard, then on your own audio.
4. Diarize and name speakers. Run pyannote.audio or an offline Sortformer over the full recording, assign words to speaker turns and resolve names from participant tracks and the attendee list.
5. Extract structured outputs. Prompt an LLM for summary, decisions and action items, and require each item to reference turn IDs. Reject items whose quoted span does not exist in the transcript.
6. Chunk and index. Split on speaker turns (a few turns per chunk with timestamps), embed with a model such as one from sentence-transformers, and store vectors in Postgres with pgvector next to a keyword index. Copy the meeting's access list onto every chunk.
7. Answer with citations. At query time, filter by what the user may see, run hybrid retrieval, rerank, and generate an answer that cites meeting, speaker and timestamp.
8. Version and reprocess. Keep model versions on every artifact. When you upgrade ASR or diarization, re-run from stored audio and re-index affected meetings.

The record every later stage points to is the speaker turn. An example chunk as stored in the index:

{
  "chunk_id": "mtg_planning_example:turns_118-121",
  "meeting_id": "mtg_planning_example",
  "meeting_title": "Quarterly planning (example)",
  "speakers": ["Ana Ruiz", "Tom Becker"],
  "start": 2491.2,
  "end": 2563.8,
  "text": "Ana Ruiz: We move the Atlas launch to June...\nTom Becker: Then I'll update the pricing page by Friday.",
  "allowed_principals": ["user:ana", "user:tom", "group:product"],
  "asr_model": "faster-whisper large-v3",
  "transcript_version": 2
}

Hybrid retrieval combines keyword matches (good for names and product terms ASR got right) with embeddings (good for paraphrases). This sketch uses the APIs from the rank_bm25 and sentence-transformers READMEs and merges the two rankings with reciprocal rank fusion. chunks is a list of records like the one above, already filtered to what the user may see:

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def hybrid_search(query, chunks, k=5, rrf_k=60):
    texts = [c["text"] for c in chunks]
    bm25 = BM25Okapi([t.lower().split() for t in texts])
    kw_scores = bm25.get_scores(query.lower().split())
    emb = model.encode(texts)
    vec_scores = model.similarity(model.encode([query]), emb)[0].tolist()

    fused = [0.0] * len(chunks)
    for scores in (kw_scores, vec_scores):
        ranked = sorted(range(len(chunks)), key=lambda i: -scores[i])
        for rank, i in enumerate(ranked):
            fused[i] += 1.0 / (rrf_k + rank + 1)
    top = sorted(range(len(chunks)), key=lambda i: -fused[i])[:k]
    return [chunks[i] for i in top]

In production you would precompute embeddings at indexing time and let pgvector's HNSW index and Postgres do both searches, but the fusion logic is the same.

What are the main trade-offs and pitfalls?

• Latency versus accuracy. Smaller streaming chunks mean faster captions and more revisions. If users do not need text during the call, skipping the live path removes a whole class of infrastructure and failure modes.
• Two transcripts, one truth. If you run both paths, never mix them in the index. Users who see a live caption and a different final transcript need to know which one is authoritative.
• Cost of GPUs. Batch jobs can be queued and packed onto GPUs; live streams hold capacity for the whole meeting whether anyone speaks or not. VAD reduces wasted work in both.
• Stale facts. Retrieval that ignores dates will happily return the launch date that was later changed. Store decisions as entities with history and rank recent meetings higher for "current state" questions; see multi-meeting RAG for decisions and commitments.
• Permissions drift. Copy access lists onto chunks at index time and update them when sharing changes, or filter against the source of truth at query time. Never filter after generation. See permission-aware RAG for sensitive meetings.
• Deletion. When a recording is deleted, its transcript, extracted items, embeddings and caches must go too. Key everything by meeting ID so this is one job, not an archaeology project.

How do you evaluate the whole system?

• Transcription: word error rate on your own meetings, per audio condition; the Open ASR Leaderboard paper and its evaluation code show how to report WER and speed (RTFx) reproducibly.
• Live path: time from speech to first partial text, time to stable text, and how often displayed text is revised.
• Attribution: DER and cpWER, as described in the diarization guide linked above.
• Extraction: precision and recall of decisions and action items against human-labelled meetings, plus owner accuracy.
• Retrieval and answers: recall@k for questions with known source turns, answer correctness, and citation accuracy (does the cited turn support the claim?).
• Operations: time from meeting end to searchable, job failure rate, and reprocessing time per hour of audio.

Questions about specific tools are best taken to their communities, for example the pyannote.audio Discussions.

How does Acta approach this?

Acta does not transcribe live. It records Zoom, Google Meet and Microsoft Teams meetings through a desktop app, a Chrome extension, a notetaker that joins from your calendar, or mobile apps and uploads, and transcribes after the meeting. Each transcript has speaker names, taken from the meeting and the calendar invite, and timestamps, and Acta writes a summary, decisions and action items with owners. Ask Acta answers questions across the meetings you choose using retrieval-augmented generation over their transcripts, and its answers cite the meeting. AI assistants connected through Acta's MCP connector only see meetings the user can see; see AI assistants.

Frequently asked questions

Do I need live transcription at all?

Only if people need text during the call, for example captions or accessibility. Search, summaries, decisions and action items are all built after the meeting, and a full-context pass gives a more stable record than streaming output.

Should streaming and batch use the same model?

Not necessarily. Streaming needs models built for small chunks, such as cache-aware streaming models or Whisper with a streaming policy. Batch can use the most accurate model you can afford, because latency matters less.

Where should vectors live?

If your meeting metadata and permissions are already in Postgres, pgvector keeps vectors, filters and joins in one place. A separate vector database makes sense at larger scale, at the cost of keeping access metadata in sync.

How long after the meeting should results be ready?

Set a target and measure it end to end, from meeting end to searchable. Batch transcription, diarization and extraction run in sequence, so parallelise per meeting and keep queues short at peak hours.

Where do I start if I already have transcripts?

Start at indexing and retrieval. The RAG architecture guide covers chunking, hybrid search and citations over existing transcripts.