How to Build Speech Recognition With Voice Activity Detection (2026 Guide)

Quick answer: Automatic speech recognition (ASR) converts spoken audio into text; voice activity detection (VAD) finds where speech starts and stops. The shortest path: run Silero VAD or pyannote to cut audio into speech segments, then transcribe them with Parakeet TDT via NVIDIA NeMo, Qwen3-ASR or Granite Speech, and pick a model using the Open ASR Leaderboard.

Speech-to-text sits behind meeting notes, captions, voice search, call-centre analytics and voice agents. Open models now match or beat many paid APIs on public benchmarks, and they let you keep audio on your own servers.

VAD is the quiet partner in every production ASR system. It skips silence and music, splits long recordings into segments the model can handle, and tells a voice agent when the user has finished speaking. This guide covers both tasks together.

What are speech recognition and voice activity detection?

The Hugging Face ASR task page defines ASR as transcribing audio into text, also called speech-to-text. Input is a waveform, usually 16 kHz mono; output is text, optionally with punctuation, word or segment timestamps, and a detected language.

Current open ASR models fall into three groups:

• CTC and transducer (TDT/RNN-T) models such as Parakeet: a Conformer-style encoder with a light decoder. They are very fast and give reliable word timestamps.
• Encoder-decoder models such as Whisper: an audio encoder and an autoregressive text decoder, trained on many languages.
• Speech-LLMs such as Canary-Qwen, Granite Speech and Qwen3-ASR: an audio encoder connected to an LLM decoder. They currently top accuracy rankings but are slower than CTC or TDT models.

VAD is a frame-level classifier: for each short frame of audio it predicts speech or non-speech, and post-processing merges frames into segments with minimum durations. The voice activity detection model list includes pyannote segmentation models, FunASR's FSMN VAD and turn-detection models for voice agents. Some segmentation models also detect overlapping speakers, which leads into speaker diarization.

What can you build with it?

• Meeting transcription and notes: transcribe calls, attribute speakers and pass the text to a summarisation model.
• Captions and subtitles for video, lectures and live streams, using segment timestamps.
• Voice agents: streaming VAD decides when the user stopped talking, then ASR feeds an LLM and a TTS model replies.
• Call-centre analytics: transcribe recorded calls for search, QA scoring and compliance checks.
• Voice notes and dictation in mobile and desktop apps, including offline use on device.
• Media search: index podcasts and video archives by what was said.

Which models should you use?

ASR models:

• nvidia/parakeet-tdt-0.6b-v3: 600M FastConformer-TDT model, 25 European languages with automatic language detection, punctuation and word timestamps, CC-BY-4.0. Very high throughput; a strong default for batch transcription.
• nvidia/canary-qwen-2.5b: 2.5B English speech-LLM (FastConformer encoder with a Qwen3-1.7B-based decoder), CC-BY-4.0 and marked ready for commercial use. Trained on clips up to 40 seconds, so pair it with VAD for long audio.
• ibm-granite/granite-speech-4.1-2b: Apache 2.0 speech-language model for recognition and translation in English, French, German, Spanish, Portuguese and Japanese. Listed with a 5.33 average WER on the Open ASR Leaderboard; works with transformers and vLLM.
• Qwen/Qwen3-ASR-1.7B and Qwen3-ASR-0.6B-hf: Apache 2.0, language identification and recognition for 52 languages and dialects, with transformers and vLLM backends and a companion forced aligner. The 0.6B model targets low latency.
• openai/whisper-large-v3-turbo: 809M multilingual model, a pruned Whisper large-v3 with 4 decoder layers instead of 32. Very widely supported by tools such as transformers and many community runtimes.

VAD and segmentation:

• Silero VAD: small, fast VAD distributed as a pip package, supports 8 and 16 kHz audio. A common default for both batch and streaming. Check the licence file in the repo.
• pyannote/segmentation-3.0: MIT-licensed, gated model that labels 10-second windows as non-speech, single speakers or overlapping speakers; usable as VAD through pyannote's VoiceActivityDetection pipeline.
• pyannote/speaker-diarization-community-1: full diarization pipeline (who spoke when) for when you also need speaker labels.

Compare ASR models on the Open ASR Leaderboard, which ranks models by average WER and reports speed as RTFx across English short-form, long-form and multilingual sets. The method is described in the Open ASR Leaderboard paper.

How do you build it, step by step?

1. Ingest and resample. Decode any input format with ffmpeg and convert it to 16 kHz mono, which nearly all ASR and VAD models expect.
2. Detect speech. Run VAD to get speech regions. Merge regions separated by short pauses and cap segment length (for example 20-30 seconds) so it fits the ASR model.
3. Transcribe. Batch the segments through the ASR model on GPU. Keep each segment's start offset so you can map timestamps back to the original file.
4. Add speakers and timestamps. Use word or segment timestamps from the model, and optionally align them with a diarization pipeline to label speakers.
5. Post-process. Apply punctuation if the model lacks it, format numbers, redact sensitive data and store the transcript with its segments.
6. Evaluate. Compute WER on a labelled sample of your own audio, broken down by accent, noise level and audio source.

Step 2 with Silero VAD, following the silero-vad README:

# pip install silero-vad   (follows the snakers4/silero-vad README)
from silero_vad import load_silero_vad, read_audio, get_speech_timestamps

model = load_silero_vad()
wav = read_audio("meeting.wav")  # resampled to 16 kHz mono

speech_timestamps = get_speech_timestamps(
    wav,
    model,
    return_seconds=True,  # seconds instead of samples
)
for seg in speech_timestamps:
    print(seg["start"], seg["end"])  # send each region to the ASR model

Step 3 with Parakeet through NVIDIA NeMo, following the model card. Setting timestamps=True returns word and segment timestamps:

# pip install -U "nemo_toolkit[asr]"  (follows the parakeet-tdt-0.6b-v3 model card)
import nemo.collections.asr as nemo_asr

asr_model = nemo_asr.models.ASRModel.from_pretrained(
    model_name="nvidia/parakeet-tdt-0.6b-v3"
)

output = asr_model.transcribe(["meeting.wav"], timestamps=True)
print(output[0].text)

for stamp in output[0].timestamp["segment"]:
    print(f"{stamp['start']}s - {stamp['end']}s : {stamp['segment']}")

If you prefer the transformers pipeline (see the Hugging Face audio course ASR chapter), the same pattern works for Whisper-style models:

import torch
from transformers import pipeline

# Simplified from the openai/whisper-large-v3-turbo model card pipeline example
pipe = pipeline(
    "automatic-speech-recognition",
    model="openai/whisper-large-v3-turbo",
    torch_dtype=torch.float16,
    device="cuda:0",
)
result = pipe("meeting.wav", return_timestamps=True)
print(result["text"])

How do you serve and deploy it?

• Batch on GPU. For recorded audio, a queue of jobs and a GPU worker running a TDT or CTC model gives the best cost per hour of audio. Parakeet and similar models process audio far faster than real time.
• Speech-LLMs with vLLM. Granite Speech and Qwen3-ASR document vLLM backends, which help with throughput when you serve many concurrent requests.
• Streaming. For live captions and voice agents, run VAD on 20-30 ms frames, send each finished utterance to ASR, and show partial results. Turn-detection models can complement plain VAD.
• CPU and edge. Community ONNX, Core ML and GGUF conversions exist for several of these models, which makes on-device transcription practical. Verify accuracy after conversion.
• API layer. A FastAPI service that accepts an upload and returns JSON with text, segments and speakers covers most apps; a Gradio app on a Space is enough for internal demos.

How do you evaluate it?

• Word error rate (WER): substitutions plus deletions plus insertions, divided by reference words. Normalise text (case, punctuation, numbers) the same way for every model; the Open ASR Leaderboard repo includes the normaliser it uses.
• Character error rate (CER) for languages without spaces between words.
• RTFx: seconds of audio processed per second of compute, the leaderboard's speed metric.
• VAD metrics: precision, recall and F1 on speech frames, plus missed speech, which turns directly into missing words.
• Diarization error rate (DER) if you label speakers.

Public benchmarks such as LibriSpeech, GigaSpeech and earnings-call sets are useful, but your own audio matters most: phone calls, accents and domain vocabulary can move WER a lot.

Pitfalls and tips

• Hallucination on silence. Autoregressive decoders can invent text on silence or music. VAD before ASR reduces this sharply.
• Long audio. Many models are trained on short clips; Canary-Qwen's card mentions clips up to 40 seconds. Segment long recordings rather than passing an hour of audio in one call.
• Licences. Qwen3-ASR and Granite Speech are Apache 2.0, while the NVIDIA models above are CC-BY-4.0, which requires attribution. pyannote models are gated behind user conditions and an access token.
• Domain vocabulary. Product names and jargon cause most visible errors. Fine-tune on your data, or post-correct with a custom vocabulary or an LLM pass.
• Privacy and consent. Recordings contain personal data. Self-hosting helps, but you still need consent and retention rules; see our article on recording meetings with AI and consent.

Where to get help

• Open ASR Leaderboard Space and its repo for benchmark and evaluation questions.
• Silero VAD GitHub Discussions for VAD thresholds, streaming and sample rates.
• pyannote-audio on GitHub for segmentation and diarization.
• Model discussion tabs, for example training and fine-tuning Canary-Qwen.
• The NeMo repository for Parakeet and Canary training and inference issues.

Frequently asked questions

Do I need voice activity detection if my ASR model handles long audio?

Usually yes. VAD skips silence and music, which saves compute and reduces hallucinated text, and it gives natural segment boundaries for batching, captions and speaker alignment.

Which open ASR model is most accurate?

Rankings change often. At the time of writing, speech-LLMs such as Granite Speech 4.1 and Canary-Qwen are near the top of the Open ASR Leaderboard for English, while Parakeet TDT models are much faster. Check the leaderboard and test on your own audio.

Can I run speech recognition on CPU?

Yes for smaller models and offline jobs, especially with ONNX or GGUF conversions. For real-time transcription of many streams, a GPU is far more cost-effective.

How do I add speaker labels?

Run a diarization pipeline such as pyannote's speaker-diarization-community-1 alongside ASR, then assign each transcribed word or segment to the speaker whose turn overlaps its timestamps.

Acta uses this kind of pipeline (speech detection, transcription and speaker labels) to turn meetings into transcripts, summaries and action items, including with the bot-free AI note taker. Next in this series: build a summarization app.