Fill-Mask Models Explained and How to Use Them (2026 Developer Guide)

Quick answer: Fill-mask is the task of predicting a hidden word in a sentence, such as "Paris is the [MASK] of France". It is how encoder models like BERT, RoBERTa and ModernBERT are pre-trained. To use it, load ModernBERT, mmBERT or an Ettin encoder with the Hugging Face transformers fill-mask pipeline, then fine-tune the same encoder for your real task.

Few products ship "fill in the blank" as a feature, yet masked language models sit underneath a large share of production NLP: classifiers, entity extractors, embedding models, rerankers and on-device text features. They are small, fast and cheap to run compared with generative LLMs.

This guide explains what fill-mask models do, which current open encoders to choose, how to run and adapt them with transformers, and how to evaluate and deploy them.

What is fill-mask?

The Hugging Face fill-mask task page describes it as masked language modeling: the input is text where one or more tokens are replaced by a special mask token, and the output is a ranked list of candidate tokens with probabilities. For "The [MASK] barked at me", a model returns candidates such as "dog" or "wolf", each with a score.

Fill-mask models are encoders. Unlike decoder-only LLMs, which read left to right and predict the next token, an encoder sees the whole sentence at once, so each token's representation uses context from both sides. During pre-training a fraction of tokens (15% in the transformers training guide) is masked at random and the model learns to recover them. That objective needs no labels, which is why it scales to trillions of tokens of raw text.

The useful product of pre-training is not the mask predictions themselves but the contextual token embeddings. You keep the encoder, swap the masked language modeling head for a small task head, and fine-tune it on labelled data.

One detail matters in practice: the mask token string differs by model. BERT-style tokenizers, including ModernBERT, use [MASK]; RoBERTa-style tokenizers use <mask>. Read it from tokenizer.mask_token instead of hard-coding it.

What can you build with it?

• Domain-adapted encoders. Continue masked language modeling on your contracts, clinical notes or support tickets, then fine-tune for classification or extraction. The task page notes that domain pre-training tends to give better downstream results.
• Classifiers and entity extractors. Fine-tune the encoder for intent detection, sentiment, routing, PII detection or named entity recognition.
• Embedding and retrieval models. Encoders are the usual base for sentence embeddings, sparse retrievers and rerankers, for example with sentence-transformers.
• Text completion helpers. Suggest a missing word in forms, spell-correction candidates or autocomplete for structured fields.
• Data augmentation. Mask a word in a training sentence and substitute top predictions to create paraphrased variants.
• Probing and auditing. Inspect what a model has learned, such as factual associations or social biases, by checking its mask predictions on templates.

Which models should you use?

Most new work starts from encoders released since late 2024, which support much longer inputs than the 512-token limit of classic BERT. Older checkpoints are still widely fine-tuned and well supported.

• answerdotai/ModernBERT-base and answerdotai/ModernBERT-large: English and code encoders with 149M and 395M parameters, trained on about 2 trillion tokens with an 8,192-token context, rotary embeddings and alternating local-global attention. The ModernBERT repository is Apache 2.0; check each model card for the weights licence. ModernBERT-large is the example model on the task page.
• jhu-clsp/mmBERT-base and jhu-clsp/mmBERT-small: multilingual ModernBERT-style encoders trained on text in over 1,800 languages, 307M and 140M total parameters, 8,192-token context. Use them when your data is not only English. The mmBERT repository shows an MIT licence badge.
• jhu-clsp/ettin-encoder-150m: part of the Ettin suite, paired encoders and decoders trained on the same data in six sizes from 17M to 1B parameters. The 17M encoder is a good fit for CPU and edge deployments. The Ettin repository is MIT-licensed.
• FacebookAI/roberta-base: the long-standing English baseline with a 512-token limit and a huge ecosystem of fine-tuned variants. Its mask token is <mask>.
• microsoft/deberta-v3-base: a strong English encoder for classification and NLI fine-tuning, frequently used as a comparison point in encoder papers.

There is no single public leaderboard for fill-mask itself, because raw mask accuracy says little about downstream quality. Encoder papers compare models on GLUE-style classification, retrieval and code benchmarks; see the ModernBERT paper, the mmBERT paper and the Ettin paper, and the ModernBERT announcement post for a readable summary.

How do you build it, step by step?

1. Pick an encoder. English or code: ModernBERT. Many languages: mmBERT. Tight latency or memory budget: a small Ettin encoder. Existing fine-tuned ecosystem you depend on: RoBERTa or DeBERTa.
2. Check the tokenizer. Print tokenizer.mask_token and test a few sentences from your domain to confirm the vocabulary handles your jargon without splitting every term into fragments.
3. Try the pipeline. Run the fill-mask pipeline on domain sentences. Weak predictions on your own text are a sign that domain adaptation will help.
4. Adapt to your domain (optional). Continue masked language modeling on unlabelled in-domain text, following the transformers masked language modeling guide.
5. Fine-tune for the real task. Load the adapted checkpoint with a classification, token classification or embedding head and train on labelled data.
6. Evaluate and ship. Track perplexity for the adaptation step and task metrics such as F1 for the final model, then export and serve.

Inference with the fill-mask pipeline, as in the transformers ModernBERT documentation:

# pip install -U transformers torch  (follows the transformers ModernBERT model doc)
from transformers import pipeline

fill = pipeline(
    task="fill-mask",
    model="answerdotai/ModernBERT-base",
    device=0,  # use -1 or omit for CPU
)
mask = fill.tokenizer.mask_token  # "[MASK]" here, "<mask>" for RoBERTa
for pred in fill(f"Plants create {mask} through a process known as photosynthesis."):
    print(pred["token_str"], round(pred["score"], 3))

Domain adaptation follows the transformers masked language modeling task guide: tokenize your text, group it into fixed-size blocks, and let the data collator mask tokens on the fly.

# Simplified from the transformers masked_language_modeling task guide
from transformers import (AutoModelForMaskedLM, AutoTokenizer, Trainer,
                          TrainingArguments, DataCollatorForLanguageModeling)

model_id = "answerdotai/ModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

# lm_dataset: a datasets.DatasetDict of tokenized, grouped blocks of your text
collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)
args = TrainingArguments(output_dir="domain-mlm", learning_rate=2e-5,
                         num_train_epochs=3, weight_decay=0.01, eval_strategy="epoch")
trainer = Trainer(model=model, args=args, train_dataset=lm_dataset["train"],
                  eval_dataset=lm_dataset["test"], data_collator=collator)
trainer.train()

import math
print("Perplexity:", math.exp(trainer.evaluate()["eval_loss"]))

The guide sets mlm_probability=0.15, so 15% of tokens are masked each time the data is iterated. The Hugging Face course chapter on fine-tuning a masked language model walks through the same process with more explanation.

How do you serve and deploy it?

• CPU first. Encoders in the 17M to 150M range often run fast enough on CPU for synchronous API calls, especially with short inputs. Batch requests when you can.
• GPU for long inputs. ModernBERT and mmBERT accept up to 8,192 tokens. Attention cost grows with length, so long-document workloads benefit from a GPU and an efficient attention implementation; the transformers docs show attn_implementation="sdpa", and the ModernBERT code supports Flash Attention.
• Export. ONNX export and int8 quantization are common for encoder models in production. Re-check accuracy on your evaluation set after any conversion.
• Serve the fine-tuned model, not the MLM head. In most products the fill-mask head is only used during development. What you deploy is the classifier, tagger or embedding model built on the same encoder.
• Front ends. A FastAPI endpoint that returns labels or vectors as JSON covers most integrations; a Gradio demo on a Hugging Face Space is enough for internal review.

How do you evaluate it?

• Perplexity on held-out in-domain text, computed as the exponential of the evaluation loss. The task page lists cross-entropy and perplexity as its metrics. Lower is better; compare only between models with the same tokenizer.
• Top-k accuracy on a set of masked sentences you build from your domain: does the correct word appear in the top 1 or top 5 predictions?
• Downstream metrics, which matter most: accuracy and macro F1 for classification, entity-level F1 for NER, nDCG or recall@k for retrieval.
• Public benchmarks such as GLUE for English understanding and multilingual suites for mmBERT, as reported in each model's paper. Treat them as a shortlist filter, then measure on your data.

Pitfalls and tips

• Wrong mask token. Passing [MASK] to a RoBERTa model, or <mask> to a BERT-style model, silently gives poor predictions. Use tokenizer.mask_token.
• Single-token predictions. One mask predicts one subword token. Multi-word or rare-word answers need several masks or a generative model.
• Library versions. Newer architectures such as ModernBERT need a recent transformers release; if loading fails, upgrade transformers before anything else.
• Bias in predictions. Mask predictions reflect the pre-training data and can surface stereotypes. Do not expose raw predictions to end users without review.
• Licences. Repository licences and model licences can differ. Read each model card before commercial use.
• Do not reach for an LLM by default. For classification and extraction at volume, a fine-tuned encoder is usually far cheaper and faster than prompting a large generative model.

Where to get help

• The Hugging Face forum for transformers, training and Hub questions.
• The transformers repository on GitHub for bugs in the fill-mask pipeline, tokenizers or Trainer.
• The ModernBERT, mmBERT and Ettin repositories for model-specific training code and issues.
• The ModernBERT collection on the Hub for all official checkpoints in one place.

Frequently asked questions

Is fill-mask the same as text generation?

No. Fill-mask models are encoders that predict a hidden token using context on both sides. Text generation models are decoders that write new text left to right. Encoders are better suited to classification, extraction and embeddings; decoders to open-ended writing.

Which fill-mask model should I start with in 2026?

For English or code, ModernBERT-base. For multilingual text, mmBERT-base or mmBERT-small. For very small deployments, an Ettin encoder in the 17M to 68M range. Then fine-tune on your task and compare on your own data.

Can a fill-mask model predict more than one word?

Each mask token predicts one token, and many words split into several tokens. You can add several masks and decode them one at a time, but for phrases or sentences a generative model is the better tool.

Do I need domain adaptation?

Not always. If the base model already predicts sensible words on your text and downstream F1 is good, skip it. Continued masked language modeling helps most with specialised vocabulary such as medical, legal or internal product terms.

Encoders built this way power the next steps in this series: text classification, named entity recognition and semantic search with embeddings.