Classifying PDF Documents with RAG and LLMs

Classifying PDF Documents with RAG and LLMs

A document classifier built on embeddings has a units problem. Embedding models take a few hundred tokens; documents are longer than that, so you split them into chunks and embed those. Now your classifier predicts labels for chunks, and you needed a label for a document.

Something has to collapse many chunk predictions into one document prediction, and that step is usually written as a single line — a mean, a majority vote — and never examined. It deserves examining, because averaging the labels good and bad produces neutral, and a furious review of a product with one redeeming feature is not a lukewarm one.

This post builds the pipeline end to end, then shows it getting exactly that wrong.

About the documents

The corpus here is synthetic: a handful of product-review documents written for this post, rendered to PDF with matplotlib, and labelled bad, neutral or good by construction. Nothing is scraped and no real customer text is involved.

Synthesising is the right choice for this particular question. The post is about the mechanics of chunk-to-document aggregation, and the failure it demonstrates needs a document whose chunks genuinely disagree — a polarised review with a glowing paragraph and a furious one. Finding that reliably in a real corpus means labelling a lot of data first; constructing it takes six lines and makes the demonstration exact. It also keeps the post reproducible from its own requirements.txt, with no data licence in the way.

What it costs is realism. Nothing here tells you the pipeline works on real documents: real reviews are longer, messier, full of boilerplate and OCR noise, and their class balance is not three-way even. This corpus stands in for the shape of the problem, not its difficulty. Were this a real deployment — routing support tickets, triaging complaints — the cost of a wrong label falls on a customer whose complaint is filed as neutral and never escalated, which is precisely the failure this post ends on.

Building documents to classify

matplotlib writes PDFs and pypdf reads them, so the whole corpus can be manufactured in-process. The one setting that matters is pdf.fonttype = 42, which embeds TrueType fonts and keeps the text extractable — the default renders glyphs as vector paths, and extraction then silently returns nothing.

import os
import tempfile
import warnings

warnings.filterwarnings("ignore")

import matplotlib

matplotlib.use("Agg")
matplotlib.rcParams["pdf.fonttype"] = 42
import matplotlib.pyplot as plt
from pypdf import PdfReader

TRAINING = {
    "good": [
        "The battery lasts all day and the screen is bright and clear. Setup took two "
        "minutes and everything worked the first time. Support answered within the hour.",
        "Excellent build quality and the metal chassis feels solid. Delivery arrived "
        "early and the packaging was immaculate. I would buy from this seller again.",
    ],
    "neutral": [
        "The device functions as described in the listing. Battery life is around six "
        "hours, which matches the specification. Packaging was standard cardboard.",
        "Delivery took four working days, as stated. The manual covers basic setup in "
        "six languages. The colour matches the product photograph reasonably closely.",
    ],
    "bad": [
        "The screen developed dead pixels within a week and support never replied. The "
        "battery drains overnight even when powered off. The returns process was slow.",
        "Arrived with a cracked case and no power cable. Three emails to support went "
        "unanswered over two weeks. The refund is still not processed after a month.",
    ],
}

workspace = tempfile.mkdtemp()


def write_pdf(text: str, path: str) -> str:
    """Render text to a single-page PDF whose contents can be extracted again."""
    figure = plt.figure(figsize=(8.5, 11))
    figure.text(0.08, 0.9, text, wrap=True, fontsize=11, va="top")
    figure.savefig(path, format="pdf")
    plt.close(figure)
    return path


def read_pdf(path: str) -> str:
    """Extract all text from a PDF."""
    return "\n".join(page.extract_text() or "" for page in PdfReader(path).pages)


paths, labels = [], []
for label, documents in TRAINING.items():
    for index, text in enumerate(documents):
        paths.append(write_pdf(text, os.path.join(workspace, f"{label}_{index}.pdf")))
        labels.append(label)

print(f"{len(paths)} PDFs written")
print(f"round-trip check: {read_pdf(paths[0])[:60]}...")

6 PDFs written
round-trip check: The battery lasts all day and the screen is bright and clear...

Six documents, and the text survives the round trip. Had that check returned an empty string, everything downstream would have trained on nothing and still reported a confident answer — worth asserting rather than assuming.

Chunk, embed, classify

Now the pipeline proper. Note the import path: RecursiveCharacterTextSplitter moved out of the top-level langchain package, which no longer holds it in the 1.x layout — that and the related deletions are covered in the companion post on LangChain.

import numpy as np
from langchain_text_splitters import RecursiveCharacterTextSplitter
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

LABEL_TO_ID = {"bad": 0, "neutral": 1, "good": 2}
ID_TO_LABEL = {value: key for key, value in LABEL_TO_ID.items()}

splitter = RecursiveCharacterTextSplitter(chunk_size=120, chunk_overlap=20)
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")


def chunks_of(path: str) -> list[str]:
    """Split one PDF into embeddable pieces."""
    return splitter.split_text(read_pdf(path))


features, targets = [], []
for path, label in zip(paths, labels):
    pieces = chunks_of(path)
    features.extend(embedder.encode(pieces))
    targets.extend([LABEL_TO_ID[label]] * len(pieces))

classifier = LogisticRegression(max_iter=1000).fit(features, targets)

print(f"{len(paths)} documents -> {len(features)} chunks")
print(f"embedding dimension: {np.shape(features)[1]}")

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

6 documents -> 12 chunks
embedding dimension: 384

Each document became several chunks, and every chunk inherited its document’s label. That inheritance is the first assumption worth noticing: it asserts that every part of a bad document is itself bad, which is how the training data comes to disagree with the world before any prediction happens.

Prediction on unseen documents

Two held-out documents, one clearly positive and one clearly negative:

HELD_OUT = {
    "good": "Battery easily lasts a full day and the display is crisp and even. "
            "Everything worked immediately out of the box and setup was painless.",
    "bad": "The unit stopped charging after ten days and nobody from support has "
           "replied to any of my messages. The packaging arrived crushed as well.",
}


def predict_chunks(text: str) -> np.ndarray:
    """Per-chunk label ids for one document."""
    path = write_pdf(text, os.path.join(workspace, "held_out.pdf"))
    return classifier.predict(embedder.encode(chunks_of(path)))


for true_label, text in HELD_OUT.items():
    predicted = predict_chunks(text)
    document_label = ID_TO_LABEL[int(round(float(predicted.mean())))]
    print(f"true={true_label:8s} chunks={[ID_TO_LABEL[p] for p in predicted]}  ->  {document_label}")

true=good     chunks=['good', 'good']  ->  good
true=bad      chunks=['bad', 'neutral']  ->  bad

On documents this uniform the aggregation never had to make a decision — the chunks all vote together, so mean, median and majority agree. That is exactly the case that makes the next one surprising.

The aggregation is the model

Now a document that is genuinely mixed: a reviewer who loves the hardware and cannot get support to answer.

from collections import Counter

MIXED = (
    "The screen is genuinely beautiful and the battery easily lasts a full day. "
    "Build quality is superb and it feels worth every penny. "
    "However support has ignored four emails about the charging fault. "
    "The returns process is a disgrace and I still have no refund."
)

predicted = predict_chunks(MIXED)

mean_label = ID_TO_LABEL[int(round(float(predicted.mean())))]
majority_label = ID_TO_LABEL[Counter(predicted.tolist()).most_common(1)[0][0]]

print(f"chunk predictions : {[ID_TO_LABEL[p] for p in predicted]}")
print(f"mean, then round  : {mean_label}")
print(f"majority vote     : {majority_label}")
print(f"disagreement (sd) : {predicted.std():.2f}")

chunk predictions : ['good', 'bad', 'bad']
mean, then round  : neutral
majority vote     : bad
disagreement (sd) : 0.94

Mean-and-round is the rule the original version of this post used, and its pathology is on display. good is 2 and bad is 0, so mixing them lands in the middle: here the negative chunks outnumber the positive one two to one, and the document is still reported as neutral. Majority vote, looking at the same predictions, says bad.

That is not a rounding artefact, it is a category error. The label set {bad, neutral, good} is ordinal: the values are ordered, but the gaps between them are not quantities you may add. Averaging them asserts that one glowing paragraph plus one furious paragraph equals two indifferent ones. It does not. A polarised document and a lukewarm document are different things a reader would want distinguished, and mean aggregation makes them identical.

Majority vote has the opposite failure. It discards the minority entirely, so a document that is 60% praise and 40% “the device caught fire” reports as good.

The honest answer is that neither rule is right, because the question is wrong. Once chunks disagree, “what is this document’s label” has no good answer, and the useful output is the disagreement itself — the standard deviation above, or the full distribution. A routing system should escalate a mixed document to a human rather than average it into blandness.

Caveat: six documents is not an evaluation

Everything here is fitted on six synthetic documents and demonstrated on three. There is no held-out measurement worth the name, no cross-validation, and the classifier has seen so few examples that its decision boundary is essentially inherited from the embedding geometry.

The aggregation argument does not depend on the classifier being good — it is a claim about ordinal labels that holds for any chunk-level predictor. But nothing here says how well this pipeline would perform, and no accuracy from this toy corpus should be reported as though it did.

Where an LLM fits instead

The original version of this post called openai.ChatCompletion.create to label chunks. That method was removed in openai 1.0, and the modern equivalent needs an API key and bills per call. A local zero-shot model does the same job with neither:

from transformers import pipeline

zero_shot = pipeline(
    "zero-shot-classification", model="typeform/distilbert-base-uncased-mnli"
)

result = zero_shot(
    "Support has ignored four emails about the charging fault.",
    candidate_labels=["positive review", "neutral review", "negative review"],
)
print({label: round(score, 3) for label, score in zip(result["labels"], result["scores"])})

{'negative review': 0.464, 'positive review': 0.343, 'neutral review': 0.193}

It ranks negative review highest, with no training data at all — though note how flat the distribution is. A 0.46/0.34 margin on an unambiguously negative sentence is not a confident model, and a 67M-parameter MNLI checkpoint is the reason. The trade against the fitted classifier is the usual one: zero-shot needs no labels but costs a forward pass per candidate label per chunk, and cannot learn your domain’s idiosyncrasies; the logistic regression is nearly free at inference but required labelled documents to exist first.

Neither choice touches the aggregation problem. Whichever model labels the chunks, something still has to decide what the document is.

Serving it

For completeness, the API surface — not executed here, since it would bind a port:

from fastapi import FastAPI, UploadFile, File

app = FastAPI()


@app.post("/predict")
async def predict(pdf: UploadFile = File(...)):
    """Return the per-chunk distribution, not just a single label."""
    import shutil
    from tempfile import NamedTemporaryFile

    with NamedTemporaryFile(delete=False, suffix=".pdf") as handle:
        shutil.copyfileobj(pdf.file, handle)
        chunk_labels = classifier.predict(embedder.encode(chunks_of(handle.name)))

    counts = Counter(ID_TO_LABEL[label] for label in chunk_labels)
    return {
        "distribution": dict(counts),
        "majority": counts.most_common(1)[0][0],
        "mixed": len(counts) > 1,
    }

Note what it returns. The original endpoint returned {"predicted_class": ...} — a single string, with the disagreement already destroyed server-side. Returning the distribution and a mixed flag costs nothing and leaves the escalation decision to the caller, who is the only party positioned to make it.

What the units mismatch costs

The claim was that the chunk-to-document step is where the modelling decision lives. The demonstration is a document whose chunks split, and which mean-aggregation therefore labelled with the one value no part of it deserved.

Everything upstream of that line is mechanical: extraction, splitting, embedding and fitting are well-defined operations with sensible defaults. The aggregation has no default that is right, because it answers a question the data does not support. If your documents are uniform in sentiment, any rule works and the choice does not matter. If they are not, no rule works, and the correct move is to stop collapsing and report the spread.

That is worth settling before the pipeline reaches production, because by then the single label is a database column and the disagreement that produced it is gone.