Efficient Few-Shot Learning with SetFit from Hugging Face

Sixteen labelled examples and about twenty seconds of CPU time will give you a working text classifier. That is SetFit’s claim and it holds up — but the number it produces depends far more on which task you point it at than on anything you configure, and the failure is quiet.

Below, identical code with an identical budget scores 0.71 on sentiment, twenty-one points clear of chance, and 0.59 on paraphrase detection, which is worse than always guessing the majority class. Neither run errors or warns. Understanding why is the difference between using SetFit well and being fooled by it.

SetFit is a frozen encoder plus a logistic regression

The method has two stages and no language-model fine-tuning at all. First, a sentence-transformer is adapted with contrastive learning: from your few labelled examples it generates many pairs, labelled “same class” or “different class”, and nudges the embedding so same-class sentences sit closer together. Then a plain logistic-regression head is fitted on the resulting embeddings.

That is where the efficiency comes from. Pair generation turns 16 examples into hundreds of training signals, and the classifier is fitting a few hundred parameters over a 384-dimensional vector rather than fine-tuning 100M weights. No prompts, no generation, no GPU required.

It also predicts exactly where the method will fail. If two classes are not already separable — or nearly so — in the encoder’s embedding space, a small contrastive nudge and a linear head will not put them there.

The data, and what is being asked of it

Both tasks come from GLUE (Wang et al., 2018), a benchmark assembled to measure general language understanding across nine tasks, distributed as nyu-mll/glue. Note the namespace: datasets 5.x requires namespace/name and the bare "glue" alias now raises HfUriError.

MRPC — the Microsoft Research Paraphrase Corpus, 3,668 training pairs of news sentences, each labelled by human annotators as paraphrases or not. It was built from a news crawl to study whether systems can recognise that two differently-worded sentences mean the same thing.
SST-2 — the Stanford Sentiment Treebank, movie-review sentences labelled positive or negative, collected to study compositional sentiment.

The question here is narrower than either benchmark’s: given only 8 labelled examples per class, can we build a usable classifier at all? That is the realistic situation when labelling is expensive — a domain expert can annotate a handful of examples, not thousands.

What it would cost to be wrong depends on the deployment. A paraphrase detector wrong in the permissive direction silently merges distinct claims, which in a deduplication or plagiarism setting is the expensive error. Neither model below is fit for either purpose; the point is the method, and the comparison is what makes the method legible.

Sentiment: the case that works

Load the corpus, take eight examples per class, train, evaluate. sample_dataset is SetFit’s helper for the stratified few-shot sample — taking a head slice instead would risk the single-class trap that ordered corpora set:

Code

import warnings

warnings.filterwarnings("ignore")

from collections import Counter
from datasets import load_dataset
from setfit import SetFitModel, Trainer, TrainingArguments, sample_dataset

ENCODER = "sentence-transformers/all-MiniLM-L6-v2"

sst2 = load_dataset("nyu-mll/glue", "sst2")
sst2_train = sample_dataset(sst2["train"], label_column="label", num_samples=8, seed=0)
sst2_eval = sst2["validation"].shuffle(seed=0).select(range(200))

counts = Counter(sst2_eval["label"])
sst2_majority = max(counts.values()) / sum(counts.values())
print(f"training examples: {sst2_train.num_rows}")
print(f"eval distribution: {dict(counts)}  majority baseline = {sst2_majority:.3f}")

training examples: 16
eval distribution: {1: 100, 0: 100}  majority baseline = 0.500

Sixteen training rows against a balanced 200-row evaluation set, so the trivial baseline is 0.5. Now train — one epoch, five contrastive iterations:

Code

sst2_model = SetFitModel.from_pretrained(ENCODER)
sst2_trainer = Trainer(
    model=sst2_model,
    args=TrainingArguments(batch_size=16, num_epochs=1, num_iterations=5, report_to=[]),
    train_dataset=sst2_train,
    eval_dataset=sst2_eval,
    metric="accuracy",
    column_mapping={"sentence": "text", "label": "label"},
)
sst2_trainer.train()

sst2_accuracy = sst2_trainer.evaluate()["accuracy"]
print(f"\naccuracy {sst2_accuracy:.3f}  vs majority baseline {sst2_majority:.3f}")

model_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.
Applying column mapping to the training dataset
Applying column mapping to the evaluation dataset

***** Running training *****
  Num unique pairs = 160
  Batch size = 16
  Num epochs = 1

[10/10 00:01, Epoch 1/1]

Step	Training Loss
1	0.368900

***** Running evaluation *****


accuracy 0.710  vs majority baseline 0.500

Around 0.71 against a 0.5 baseline, from sixteen examples in seconds. This is the result SetFit is advertised on, and it is real: sentiment is close to a direction in sentence-embedding space, because the encoder’s pretraining data already made positive and negative language look different.

column_mapping is the piece worth noting — SetFit expects columns named text and label, and this maps GLUE’s sentence onto that without copying the dataset.

Paraphrase: identical code, worse than guessing

MRPC is a sentence pair task, so the two sentences are joined into one string before encoding. Everything else is unchanged:

Code

mrpc = load_dataset("nyu-mll/glue", "mrpc")
mrpc = mrpc.map(lambda row: {"text": f"{row['sentence1']} [SEP] {row['sentence2']}"})

mrpc_train = sample_dataset(mrpc["train"], label_column="label", num_samples=8, seed=0)
mrpc_eval = mrpc["validation"].shuffle(seed=0).select(range(200))

counts = Counter(mrpc_eval["label"])
mrpc_majority = max(counts.values()) / sum(counts.values())
print(f"eval distribution: {dict(counts)}  majority baseline = {mrpc_majority:.3f}")

mrpc_model = SetFitModel.from_pretrained(ENCODER)
mrpc_trainer = Trainer(
    model=mrpc_model,
    args=TrainingArguments(batch_size=16, num_epochs=1, num_iterations=5, report_to=[]),
    train_dataset=mrpc_train,
    eval_dataset=mrpc_eval,
    metric="accuracy",
    column_mapping={"text": "text", "label": "label"},
)
mrpc_trainer.train()

mrpc_accuracy = mrpc_trainer.evaluate()["accuracy"]
print(f"\naccuracy {mrpc_accuracy:.3f}  vs majority baseline {mrpc_majority:.3f}")
print(f"beats baseline: {mrpc_accuracy > mrpc_majority}")

eval distribution: {1: 137, 0: 63}  majority baseline = 0.685

model_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.
Applying column mapping to the training dataset
Applying column mapping to the evaluation dataset

***** Running training *****
  Num unique pairs = 160
  Batch size = 16
  Num epochs = 1

[10/10 00:02, Epoch 1/1]

Step	Training Loss
1	0.531300

***** Running evaluation *****


accuracy 0.590  vs majority baseline 0.685
beats baseline: False

Roughly 0.59 against a 0.685 baseline. A model that ignored the input entirely and always answered “paraphrase” would have done better.

Two things caused this, and they compound. MRPC’s validation split is 68% positive, so accuracy on it is a soft target that a constant predictor already hits — the metric flatters the trivial solution. And more fundamentally, paraphrase-equivalence is not a direction in embedding space: two sentences about the same news event embed close together whether or not they are paraphrases, because sentence encoders are trained to place topically similar text nearby. The signal SetFit needs is close to orthogonal to the signal the encoder provides.

Caveat: this is not a verdict on SetFit or on MRPC

One seed, one sample of 8 per class, one evaluation subset. Few-shot results have large variance across seeds — the sample you happen to draw matters enormously at this size — so the honest reading is “this configuration failed”, not “SetFit cannot do MRPC”. With a few hundred labels per class, or a cross-encoder built for sentence pairs, MRPC is very learnable.

The transferable claim is narrower and safer: always compute the majority baseline. An accuracy of 0.59 looks like learning until you know that 0.685 was free.

Comparing the two runs

Side by side, with everything except the task held constant:

Code

print(f"{'task':<8} {'baseline':>9} {'SetFit':>8} {'gain':>8}")
for task, base, acc in [
    ("SST-2", sst2_majority, sst2_accuracy),
    ("MRPC", mrpc_majority, mrpc_accuracy),
]:
    print(f"{task:<8} {base:>9.3f} {acc:>8.3f} {acc - base:>+8.3f}")

task      baseline   SetFit     gain
SST-2        0.500    0.710   +0.210
MRPC         0.685    0.590   -0.095

Same encoder, same 16 examples, same hyperparameters, same seconds of compute. The only variable is whether the task’s decision boundary was already latent in the embedding, and it accounts for the entire difference between a useful classifier and a harmful one.

Predicting, and what the model actually is

A trained model predicts from raw strings, and is small enough to be worth looking at:

Code

examples = [
    "an absolute delight from start to finish",
    "a tedious, joyless slog",
]
predictions = sst2_model.predict(examples)

for text, label in zip(examples, predictions.tolist()):
    print(f"{['negative', 'positive'][label]:9s} {text}")

print(f"\nhead: {type(sst2_model.model_head).__name__}")
print(f"body: {type(sst2_model.model_body).__name__}")

positive  an absolute delight from start to finish
positive  a tedious, joyless slog

head: LogisticRegression
body: SentenceTransformer

The second one is wrong, and it is worth leaving in. “A tedious, joyless slog” is about as negative as film criticism gets, and the model calls it positive — which is what an accuracy of 0.71 means in practice. Roughly three in ten, and there is no reason the three should be the ambiguous ones. A number that reads as respectable in a table is still a model that will confidently misclassify plain cases, and sixteen training examples buy exactly that much.

The architecture explains how it got here: a LogisticRegression sitting on a SentenceTransformer. The head has seen sixteen points in 384 dimensions, so its decision boundary is determined by very little, and where the encoder’s geometry does not already separate the classes it has nothing else to go on. That also means the head is inspectable and swappable — SetFitModel.from_pretrained(..., use_differentiable_head=True) substitutes a torch head you can train jointly.

Where few-shot is the right tool

The claim at the top was that SetFit’s output depends more on the task than on the configuration. The evidence is two runs differing only in which dataset they loaded, landing 21 points above and 9 points below their respective baselines.

So the question to ask before reaching for it is not “do I have few labels” but “does a general-purpose sentence encoder already place my classes apart”. Topic, sentiment, intent and tone usually qualify — they are properties of what the sentence is about, which is what these encoders were built to represent. Relations between sentences, fine distinctions inside a specialist vocabulary, and anything requiring the model to reason rather than recognise usually do not.

When the answer is no, more few-shot examples will not rescue it, because the limitation is in the frozen encoder rather than in the sixteen labels. That is the point at which the cheap method has told you something useful: you need a different representation, not a bigger sample.