Evaluating NLP Models with the Hugging Face evaluate Library

The same prediction scores 0.0 and 0.96 depending on which metric you load. evaluate makes that swap one line — which is the danger, not the feature.
NLP
Hugging Face
Evaluation
Author

Ravi Kalia

Published

March 12, 2025

Evaluating NLP Models with the Hugging Face evaluate Library

Take one predicted sentence, "The cat sat.", and one reference, "A cat was sitting.". Score them with ROUGE-2 and you get 0.0 — a total failure. Score the same pair with BERTScore and you get 0.96 — near-perfect. Neither metric is broken. They are answering different questions, and only one of them is the question you meant.

The evaluate library makes swapping between them a single argument. That uniformity is genuinely useful and it is also the trap: evaluate.load("rouge") and evaluate.load("bertscore") look equally authoritative at the call site, cost nothing to interchange, and encode completely different theories of what “correct” means. This post is about reading that difference off the numbers.

Every metric is the same three calls

The library’s whole interface is load, then compute, with predictions and references. Classification is where this feels least surprising, because the notion of correct is unambiguous:

Code 
import warnings

warnings.filterwarnings("ignore")

import evaluate

accuracy = evaluate.load("accuracy")
print(accuracy.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 0]))
{'accuracy': 0.75}

Three of four match, so 0.75, and there is nothing to argue about — the prediction either equals the reference or it does not. Metrics for generated text have no such luxury, because a prediction can be right without being identical to the reference. Each one resolves that differently.

The metric is also an object rather than a function, which is what lets evaluate.combine run several over a single pass:

Code 
clf_metrics = evaluate.combine(["accuracy", "f1", "precision", "recall"])
print(clf_metrics.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 0]))
{'accuracy': 0.75, 'f1': 0.6666666666666666, 'precision': 0.5, 'recall': 1.0}

Accuracy 0.75 but F1 lower on the same four predictions — already a preview of the point. Accuracy counts every position equally; F1 only cares about the positive class.

The same pair, four verdicts

Here is the disagreement in full. One prediction, one reference, several metrics:

Code 
prediction = "The cat sat."
reference = "A cat was sitting."

rouge = evaluate.load("rouge")
bleu = evaluate.load("bleu")

rouge_scores = rouge.compute(predictions=[prediction], references=[reference])
bleu_scores = bleu.compute(predictions=[prediction], references=[[reference]])

print("ROUGE:", {k: round(float(v), 4) for k, v in rouge_scores.items()})
print("BLEU :", round(bleu_scores["bleu"], 4))
print("BLEU precisions by n-gram:", [round(p, 3) for p in bleu_scores["precisions"]])
ROUGE: {'rouge1': 0.2857, 'rouge2': 0.0, 'rougeL': 0.2857, 'rougeLsum': 0.2857}
BLEU : 0.0
BLEU precisions by n-gram: [0.5, 0.0, 0.0, 0.0]

ROUGE-1 finds some unigram overlap — cat and little else. ROUGE-2 is exactly 0.0, because the two sentences share no adjacent word pair. BLEU is 0.0 too, and its per-n-gram precisions show why: it matches half the unigrams and then nothing at all from bigrams upward, and BLEU’s geometric mean over n-gram precisions is zero the moment any one of them is.

All of these are counting string overlap. sat and sitting are different strings, so as far as these metrics are concerned the model did not say the same thing. Now change the theory of correctness:

Code 
bertscore = evaluate.load("bertscore")
bs = bertscore.compute(predictions=[prediction], references=[reference], lang="en")

print("BERTScore:", {k: round(v[0], 4) for k, v in bs.items() if isinstance(v, list)})
print("backbone :", bs["hashcode"])
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] RobertaModel LOAD REPORT from: roberta-large
Key                       | Status     | 
--------------------------+------------+-
lm_head.layer_norm.bias   | UNEXPECTED | 
lm_head.layer_norm.weight | UNEXPECTED | 
lm_head.bias              | UNEXPECTED | 
lm_head.dense.weight      | UNEXPECTED | 
lm_head.dense.bias        | UNEXPECTED | 
pooler.dense.weight       | MISSING    | 
pooler.dense.bias         | MISSING    | 

Notes:
- UNEXPECTED:   can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:  those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
BERTScore: {'precision': 0.9666, 'recall': 0.958, 'f1': 0.9623}
backbone : roberta-large_L17_no-idf_version=0.3.12(hug_trans=5.14.1)

0.96. BERTScore embeds both sentences with a pretrained transformer and matches tokens by cosine similarity, so sat and sitting land close together and the pair reads as near-equivalent.

Two metrics, one pair, verdicts of 0.0 and 0.96. The gap is not noise — it is the difference between asking “did the model produce these exact words” and “did the model mean this”. Summarization and translation papers report ROUGE and BLEU because they are cheap, deterministic and comparable across papers, not because anyone believes n-gram overlap is meaning.

Caveat: single-sentence scores are close to meaningless anyway

Every number above is computed on one example, which is fine for showing that the metrics disagree and useless for judging a model. BLEU in particular is designed as a corpus-level statistic: its brevity penalty and n-gram precisions are meant to be aggregated over a whole test set, and per-sentence BLEU is notoriously unstable — the same system can score 0.0 on a sentence it translated perfectly well with different word order.

The honest use of these calls is to accumulate over a corpus, then report once.

Caveat: BERTScore has a model inside it

bertscore is not a formula. The hashcode printed above names the backbone it silently downloaded — roughly 1.4 GB of roberta-large — and the score depends on that choice, on the layer used, and on the language flag. Two BERTScores from different backbones are not comparable, which is why the hashcode exists and why it belongs in anything you report.

It also means evaluate.load("bertscore") costs a model download and a forward pass per example, where evaluate.load("rouge") costs neither. The uniform interface hides a several-order-of-magnitude difference in what the call actually does.

Metrics carry dependencies the interface hides

evaluate.load fetches a small implementation module from the Hub, and that module may import a package you do not have. The failure then arrives at compute time rather than load time, which makes it look like a data problem rather than an install problem — an earlier version of this post shipped a commented-out BERTScore cell for exactly this reason, with a note that it was broken. It was not broken; bert_score simply was not installed.

ROUGE needs rouge_score, nltk and absl-py; BLEU variants may want sacrebleu; BERTScore needs bert_score. All are pinned in the requirements.txt beside this post, which is why the cell above runs.

Scoring real generated text

To evaluate a model rather than a hand-written string, generate something first. Note that pipeline("summarization") — which earlier versions of this post used — was removed in transformers 5, along with translation and text2text-generation. Sequence-to-sequence models are now driven directly:

Code 
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("google-t5/t5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")

article = (
    "The Hugging Face Transformers library provides thousands of pretrained models "
    "for natural language processing. It supports text classification, information "
    "extraction, question answering, summarization and translation, and it works "
    "with both PyTorch and TensorFlow."
)

enc = tok(f"summarize: {article}", return_tensors="pt").to(model.device)
summary = tok.decode(model.generate(**enc, max_new_tokens=40)[0], skip_special_tokens=True)
print("summary:", summary)

human = "Transformers offers many pretrained models for NLP tasks in PyTorch and TensorFlow."
print("\nROUGE:", {k: round(float(v), 4)
                   for k, v in rouge.compute(predictions=[summary], references=[human]).items()})
summary: the Hugging Face Transformers library provides thousands of pretrained models for natural language processing. it supports text classification, information extraction, question answering, summarization and translation.

ROUGE: {'rouge1': 0.2703, 'rouge2': 0.1143, 'rougeL': 0.2703, 'rougeLsum': 0.2703}

The summary is largely extractive — t5-small copies source phrasing rather than rewriting — and ROUGE rewards exactly that. A model that paraphrased more aggressively would score worse against this reference while arguably summarizing better. That bias belongs to the metric, not the model, and it is a large part of why summarization leaderboards reward extraction.

Writing your own

When no built-in metric encodes your notion of correct, a plain function will do, and the thing worth getting right is that it should express the asymmetry of your actual decision:

Code 
def expected_cost(predictions, references, false_negative_cost=5.0):
    """Mean cost per example when a missed positive hurts more than a false alarm."""
    incurred = sum(
        (false_negative_cost if ref == 1 else 1.0)
        for pred, ref in zip(predictions, references)
        if pred != ref
    )
    return {"expected_cost": round(incurred / len(references), 4)}


refs = [0, 1, 1, 0]
missed_positive = [0, 0, 1, 0]  # one false negative
false_alarm = [1, 1, 1, 0]  # one false positive

for name, preds in [("missed positive", missed_positive), ("false alarm", false_alarm)]:
    acc = accuracy.compute(predictions=preds, references=refs)["accuracy"]
    print(f"{name:16s} accuracy={acc:.2f}  {expected_cost(preds, refs)}")
missed positive  accuracy=0.75  {'expected_cost': 1.25}
false alarm      accuracy=0.75  {'expected_cost': 0.25}

This is the point of writing your own. Both prediction sets make exactly one mistake, so accuracy scores them identically at 0.75 and reports that the two models are equally good. They are not: one missed a positive and one raised a false alarm, and the cost metric separates them by the factor of five it was told to care about.

Which number you publish should follow from what a mistake costs in the world. A cancer screen and a spam filter have opposite asymmetries, and unweighted accuracy is blind to both.

To use a custom function inside Trainer, hand it back in the same {"name": value} shape that compute returns, which is what the built-ins produce and what compute_metrics expects.

What the uniform interface is worth

The claim at the top was that swapping metrics is one line, and that this is the danger. The evidence is 0.0 and 0.96 on an identical pair of sentences, produced by calls that differ only in a string.

What evaluate genuinely buys is that the plumbing stops being a variable: the same compute(predictions=…, references=…) shape works for every metric, combine runs a suite in one pass, and the implementation is versioned on the Hub rather than copy-pasted from a paper’s appendix. Reproducibility of the computation is real and worth having.

What it cannot do is choose for you, and the uniformity actively obscures how different the choices are — in what they measure, in what they cost, and in whether they quietly carry a 1.4 GB model and a hashcode you are then obliged to report. Pick the metric from the decision it informs, and only then let the library make it easy.