Transformers Library for NLP

One line of transformers classifies a sentence, and the brevity hides the interesting part. pipeline() is not a model. It is a tokenizer, a model, and a task-specific post-processor, wired together and handed to you as a single callable — and knowing that is the difference between using the library and being stuck the first time the default is wrong.

This post opens the call up. First by reproducing a pipeline’s output by hand from its own three parts, then by fine-tuning, which is where you have no choice but to assemble them yourself.

What one line actually does

Start with the one-liner. Always name the model explicitly: the task defaults have changed between versions, and an unnamed model is a silent dependency on whatever the library currently prefers.

Code

import warnings

warnings.filterwarnings("ignore")

from transformers import pipeline

SENTIMENT = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
classifier = pipeline("sentiment-analysis", model=SENTIMENT)

text = "I love this library!"
print(classifier(text))

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

[{'label': 'POSITIVE', 'score': 0.9998852014541626}]

POSITIVE with a confidence. Now take the same pipeline apart. The tokenizer and model it is holding are attributes, so we can run the computation manually and check that we land on the identical number:

Code

import torch

tokenizer, model = classifier.tokenizer, classifier.model
print(f"model device: {model.device}")

encoded = tokenizer(text, return_tensors="pt").to(model.device)
print(f"tokens      : {tokenizer.convert_ids_to_tokens(encoded['input_ids'][0])}")

with torch.no_grad():
    logits = model(**encoded).logits

probabilities = logits.softmax(-1)[0]
best = int(probabilities.argmax())

print(f"logits      : {logits.tolist()[0]}")
print(f"label map   : {model.config.id2label}")
print(f"manual      : {model.config.id2label[best]} {float(probabilities[best]):.7f}")
print(f"agrees      : {abs(float(probabilities[best]) - classifier(text)[0]['score']) < 1e-6}")

model device: mps:0
tokens      : ['[CLS]', 'i', 'love', 'this', 'library', '!', '[SEP]']
logits      : [-4.358635425567627, 4.713773727416992]
label map   : {0: 'NEGATIVE', 1: 'POSITIVE'}
manual      : POSITIVE 0.9998852
agrees      : True

Four steps — tokenize, forward, softmax, look up the label name — and the result matches the pipeline to seven decimal places. That is all pipeline was doing. The id2label mapping is worth noticing: POSITIVE is not something the network produces, it is a string in the model’s config that names column 1 of the logits.

Caveat: transformers 5 places models on the accelerator for you

The .to(model.device) above is not decoration. pipeline put the model on MPS (or CUDA), and feeding it CPU tensors fails — sometimes with RuntimeError: Placeholder storage has not been allocated on MPS device!, and sometimes, on this machine, by killing the interpreter outright with SIGBUS and no traceback at all.

If a hand-written forward pass dies inexplicably after migrating from transformers 4.x, this is the first thing to check. The pipeline never had the problem because it moves inputs for you; the moment you step outside it, device placement becomes yours to manage.

Different tasks, different third part

The tokenizer and model change with the checkpoint. What really varies between tasks is the post-processor — the logic turning raw logits into the structure you wanted. Token classification is the clearest case, because the useful output requires stitching word-pieces back into entities:

Code

ner = pipeline("ner", model="dslim/bert-base-NER", aggregation_strategy="simple")

for entity in ner("Hugging Face is based in New York City."):
    print(f"{entity['entity_group']:8s} {entity['word']:20s} {entity['score']:.3f}")

[transformers] BertForTokenClassification LOAD REPORT from: dslim/bert-base-NER
Key                      | Status     |  | 
-------------------------+------------+--+-
bert.pooler.dense.bias   | UNEXPECTED |  | 
bert.pooler.dense.weight | UNEXPECTED |  | 

Notes:
- UNEXPECTED:   can be ignored when loading from different task/architecture; not ok if you expect identical arch.

ORG      Hugging Face         0.766
LOC      New York City        0.999

aggregation_strategy="simple" is what merges New, York, City into one LOC span. Without it you get per-token labels in BIO scheme and reassembly is your problem. This argument used to be called grouped_entities; it was removed in transformers 5, and passing it now raises rather than warning.

Generation keeps the same shape, with decoding rules as the third part:

Code

generator = pipeline("text-generation", model="openai-community/gpt2")
generated = generator(
    "The future of AI is", max_new_tokens=20, do_sample=False, truncation=True
)
print(generated[0]["generated_text"])

[transformers] Passing `generation_config` together with generation-related arguments=({'do_sample', 'max_new_tokens'}) is deprecated and will be removed in future versions. Please pass either a `generation_config` object OR all generation parameters explicitly, but not both.
[transformers] Both `max_new_tokens` (=20) and `max_length`(=50) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer GPT2Tokenizer. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.

The future of AI is uncertain. The future of AI is uncertain.

The future of AI is uncertain. The future

Note max_new_tokens rather than max_length. max_length counts the prompt too, so it silently produces shorter and shorter continuations as prompts grow — one of those defaults that is not wrong so much as rarely what anyone means.

Caveat: `transformers` 5 deleted the sequence-to-sequence tasks

Translation and summarization used to be pipelines. They are not any more, and this is the sharpest illustration of the post’s point — the third part was removed, leaving the other two:

Code

for task in ("translation", "summarization", "text2text-generation"):
    try:
        pipeline(task, model="google-t5/t5-small")
        print(f"{task}: available")
    except KeyError:
        print(f"{task}: KeyError — no longer a registered task")

translation: KeyError — no longer a registered task
summarization: KeyError — no longer a registered task
text2text-generation: KeyError — no longer a registered task

pipeline("summarization") and pipeline("translation_en_to_fr"), both of which appeared in the original version of this post, now raise KeyError rather than warn. The replacement is to drive the encoder-decoder yourself, which for T5 means writing the task prefix the model was trained on:

Code

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

t5_tok = AutoTokenizer.from_pretrained("google-t5/t5-small")
t5 = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")


def run_t5(prompt: str, max_new_tokens: int = 40) -> str:
    """Encode, generate, decode — the three parts, by hand."""
    enc = t5_tok(prompt, return_tensors="pt").to(t5.device)
    out = t5.generate(**enc, max_new_tokens=max_new_tokens)
    return t5_tok.decode(out[0], skip_special_tokens=True)


print("translation:", run_t5("translate English to French: I like pizza."))

translation: Je veux la pizza.

The prefix translate English to French: is not API surface, it is part of the input the model was trained to condition on — which is precisely the knowledge the deleted pipeline used to hold for you. (The output here is imperfect; t5-small is 61M parameters and renders “I like” as “Je veux”. Small models translate badly, and the pipeline never protected you from that either.)

Zero-shot classification is the outlier worth seeing, because no part of it was trained on your labels:

Code

zero_shot = pipeline("zero-shot-classification", model="typeform/distilbert-base-uncased-mnli")
result = zero_shot(
    "I want to book a flight.", candidate_labels=["travel", "finance", "education"]
)
print(dict(zip(result["labels"], [round(s, 3) for s in result["scores"]])))

{'travel': 0.979, 'education': 0.014, 'finance': 0.007}

This is an entailment model being asked, once per candidate label, whether “This example is travel.” follows from the input. The post-processor turns those entailment probabilities into a ranking. It costs one forward pass per label, which is why zero-shot is convenient and not cheap.

Fine-tuning forces the parts apart

Everything so far consumed a model someone else trained. Fine-tuning is where the pipeline abstraction stops helping: you must load a tokenizer and a model separately, because the tokenizer has to be applied to a dataset before training and the model needs a freshly initialised head.

About the data. The task is sentiment on the IMDB corpus — Maas et al. (2011), 25,000 labelled training reviews scraped from IMDB and deliberately restricted to polarised ratings, discussed in more detail in the companion post on datasets. We are asking whether a general-purpose encoder can be adapted to predict review polarity from text, and using a small subset because the point here is the mechanics, not the score. A model built this way and pointed at real review traffic would face the distribution problem that corpus’s construction implies — no ambivalent reviews exist in it — so nothing below should be read as a usable sentiment classifier.

Caveat: IMDB is sorted by label, so `train[:1%]` is one class

The obvious way to take a small subset is the wrong one, and it fails silently:

Code

from collections import Counter
from datasets import load_dataset

first_slice = load_dataset("stanfordnlp/imdb", split="train[:2%]")
shuffled = load_dataset("stanfordnlp/imdb", split="train").shuffle(seed=0).select(range(500))

print(f"train[:2%]     : {Counter(first_slice['label'])}")
print(f"shuffled 500   : {Counter(shuffled['label'])}")

train[:2%]     : Counter({0: 500})
shuffled 500   : Counter({0: 254, 1: 246})

The corpus stores all 12,500 negative reviews before all 12,500 positive ones, so a head slice is entirely one class. Train a binary classifier on it and evaluation reports 100% accuracy for a model that has learned to answer “negative” unconditionally. Nothing errors, and the number looks like success. Shuffle before selecting.

Assembling the training run

With a balanced subset, the pieces go together explicitly — tokenize the dataset, load a model with an untrained classification head, define the metric, and hand all of it to Trainer:

Code

import numpy as np
import evaluate
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer,
)

BASE = "distilbert/distilbert-base-uncased"

data = shuffled.train_test_split(test_size=0.2, seed=0)
tok = AutoTokenizer.from_pretrained(BASE)
tokenized = data.map(
    lambda batch: tok(batch["text"], truncation=True, max_length=256), batched=True
)

model = AutoModelForSequenceClassification.from_pretrained(BASE, num_labels=2)
accuracy = evaluate.load("accuracy")


def compute_metrics(eval_pred):
    """Accuracy from raw logits."""
    return accuracy.compute(
        predictions=np.argmax(eval_pred.predictions, axis=1),
        references=eval_pred.label_ids,
    )


args = TrainingArguments(
    output_dir="./results",
    eval_strategy="epoch",
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    num_train_epochs=1,
    logging_steps=10,
    report_to=[],
    disable_tqdm=True,
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["test"],
    processing_class=tok,
    compute_metrics=compute_metrics,
)

trainer.train()
print(trainer.evaluate())

[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert/distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_transform.weight  | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
pre_classifier.bias     | MISSING    | 
pre_classifier.weight   | MISSING    | 
classifier.weight       | MISSING    | 
classifier.bias         | MISSING    | 

Notes:
- UNEXPECTED:   can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:  those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

{'loss': '0.6908', 'grad_norm': '3.073', 'learning_rate': '3.2e-05', 'epoch': '0.4'}
{'loss': '0.6409', 'grad_norm': '2.015', 'learning_rate': '1.2e-05', 'epoch': '0.8'}

{'eval_loss': '0.5859', 'eval_accuracy': '0.82', 'eval_runtime': '0.7805', 'eval_samples_per_second': '128.1', 'eval_steps_per_second': '5.125', 'epoch': '1'}
{'train_runtime': '17.06', 'train_samples_per_second': '23.44', 'train_steps_per_second': '1.465', 'train_loss': '0.6501', 'epoch': '1'}
{'eval_loss': '0.5859', 'eval_accuracy': '0.82', 'eval_runtime': '0.7058', 'eval_samples_per_second': '141.7', 'eval_steps_per_second': '5.667', 'epoch': '1'}
{'eval_loss': 0.5858583450317383, 'eval_accuracy': 0.82, 'eval_runtime': 0.7058, 'eval_samples_per_second': 141.683, 'eval_steps_per_second': 5.667, 'epoch': 1.0}

An accuracy meaningfully above 0.5 on a balanced held-out set, from 400 training examples and one epoch — modest, and unlike the 100% above, real.

Two argument names in that cell are the migration from transformers 4.x, and both are hard errors rather than warnings:

4.x	5.x	Symptom if unchanged
`TrainingArguments(evaluation_strategy=…)`	`eval_strategy`	`TypeError: unexpected keyword argument`
`Trainer(tokenizer=…)`	`processing_class`	Deprecated then removed; `tokenizer` no longer accepted

processing_class is the more interesting rename. Trainer needs whatever turns raw inputs into tensors, and for vision or audio models that is an image or feature processor rather than a tokenizer — the new name stopped pretending every modality is text.

What the three parts buy

The claim was that pipeline() is a tokenizer, a model, and a post-processor. The evidence is that we reconstructed its output by hand from its own attributes and matched it to seven decimals, and that fine-tuning — where no post-processor exists yet — requires touching exactly the other two.

The practical consequence is diagnostic. When a pipeline gives a wrong answer, the parts fail in distinguishable ways: garbled sub-word tokens are the tokenizer, a confident wrong label with sensible tokens is the model, and entities split across word-pieces or truncated generations are the post-processor and its arguments. Each has its own fix, and knowing which one you are looking at is most of the work.

Where the decomposition stops being enough is when the failure is in none of them. The two most expensive bugs in this post — a head slice that was secretly one class, and inputs on the wrong device — were invisible to every one of the three parts, and both reported success rather than raising. pipeline will not protect you from those, and neither will taking it apart.

What one line actually does

Caveat: transformers 5 places models on the accelerator for you

Different tasks, different third part

Caveat: transformers 5 deleted the sequence-to-sequence tasks

Fine-tuning forces the parts apart

Caveat: IMDB is sorted by label, so train[:1%] is one class

Assembling the training run

What the three parts buy

Caveat: `transformers` 5 deleted the sequence-to-sequence tasks

Caveat: IMDB is sorted by label, so `train[:1%]` is one class