Model Evaluation Metrics

Model Evaluation Metrics

Two models, trained on the same diagnostic data, score identical accuracy to three decimal places and identical AUC. One of them misses twice as many cancers as the other.

Accuracy is not broken. It is answering the question it was asked — what fraction of predictions were right — and that question is indifferent to which kind of wrong you were. Every metric encodes a position on that indifference, and the position is the whole content of the choice. Picking a metric is not a reporting decision made after the modelling; it is a modelling decision, and this post is about seeing the difference in numbers rather than in prose.

The data, and what a mistake costs

Everything below uses the Wisconsin Diagnostic Breast Cancer dataset, collected by Dr William Wolberg and colleagues at the University of Wisconsin and distributed with scikit-learn. Each of its 569 rows describes a fine-needle aspirate of a breast mass: a clinician took a cell sample, the image was digitised, and thirty features — radius, texture, concavity and so on, each as a mean, a standard error and a worst value — were computed from the nuclei visible in it. The label is the biopsy-confirmed diagnosis, malignant or benign.

It was collected to answer whether cytological features visible in an aspirate could predict malignancy well enough to inform whether a patient needs surgery. That is what makes it the right example here, because the two errors are not comparable. A false positive sends someone for a biopsy they did not need: unpleasant, expensive, not dangerous. A false negative tells a woman with cancer to go home. Any metric that treats those as interchangeable asserts something clinically false, and accuracy treats them as interchangeable by construction.

The question asked of the data in this post is deliberately not “can we diagnose cancer” — the models below are minimal and nothing here is clinically usable. It is “when two models differ, which metrics notice”.

import warnings

warnings.filterwarnings("ignore")

import numpy as np
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
# scikit-learn encodes malignant as 0; flip so the dangerous class is positive.
X, y = data.data, (data.target == 0).astype(int)

print(f"samples         : {len(y)}")
print(f"features        : {X.shape[1]}")
print(f"malignant (pos) : {y.sum()} ({y.mean():.1%})")
print(f"benign    (neg) : {(1 - y).sum()}")

samples         : 569
features        : 30
malignant (pos) : 212 (37.3%)
benign    (neg) : 357

Note the balance: 37% positive. Not catastrophically skewed, but skewed enough that a model can do nothing at all and still look respectable — which is the first thing worth demonstrating.

Accuracy alone cannot see a useless model

Three models. One always predicts benign; one is ordinary logistic regression; one is the same logistic regression, reported only when it is more than 90% sure.

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=0
)

always_benign = DummyClassifier(strategy="constant", constant=0).fit(X_train, y_train)
logistic = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)).fit(
    X_train, y_train
)

scores = logistic.predict_proba(X_test)[:, 1]
predictions = {
    "always benign": always_benign.predict(X_test),
    "logistic": logistic.predict(X_test),
    "cautious (p>0.9)": (scores > 0.9).astype(int),
}

Now score them. The last column is the one that matters clinically — malignancies sent home.

from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)

header = f"{'model':18s} {'acc':>6} {'prec':>6} {'recall':>7} {'f1':>6} {'missed cancers':>15}"
print(header)
print("-" * len(header))
for name, predicted in predictions.items():
    _, _, false_negatives, _ = confusion_matrix(y_test, predicted).ravel()
    print(
        f"{name:18s} "
        f"{accuracy_score(y_test, predicted):6.3f} "
        f"{precision_score(y_test, predicted, zero_division=0):6.3f} "
        f"{recall_score(y_test, predicted):7.3f} "
        f"{f1_score(y_test, predicted, zero_division=0):6.3f} "
        f"{false_negatives:15d}"
    )

model                 acc   prec  recall     f1  missed cancers
---------------------------------------------------------------
always benign       0.626  0.000   0.000  0.000              64
logistic            0.953  0.938   0.938  0.938               4
cautious (p>0.9)    0.953  1.000   0.875  0.933               8

Three things in that table, in increasing order of how much trouble they cause.

The do-nothing model scores 0.626. It has never identified a single malignancy and never will, and accuracy rewards it purely for the class balance. An accuracy figure without the majority-class baseline printed beside it carries almost no information.

The two real models tie on accuracy, to three decimals. Accuracy has declared them equivalent.

They are not equivalent. The cautious model misses twice as many cancers. Precision prefers it — refusing to say malignant unless nearly certain makes you right whenever you do say it — while recall correctly prefers the ordinary one. Two metrics, opposite rankings, the same pair of models.

Caveat: AUC also calls them identical, for a different reason

print(f"AUC (both models): {roc_auc_score(y_test, scores):.4f}")

AUC (both models): 0.9917

That is not a flaw in the demonstration. The two models emit the same probabilities and differ only in where the threshold sits, and AUC is threshold-independent by construction, integrating over all thresholds at once.

This makes AUC excellent for “is this score any good at ranking patients” and useless for “should I deploy this configuration”. A model with a superb AUC still has to be operated at one specific threshold, and every clinical consequence lives in that choice. AUC is silent about it.

Choosing a threshold is choosing a trade

Since the threshold is what differs, look at the whole sweep:

for threshold in [0.1, 0.3, 0.5, 0.7, 0.9]:
    predicted = (scores > threshold).astype(int)
    _, false_positives, false_negatives, _ = confusion_matrix(y_test, predicted).ravel()
    print(
        f"threshold {threshold:.1f}: "
        f"recall={recall_score(y_test, predicted):.3f}  "
        f"precision={precision_score(y_test, predicted, zero_division=0):.3f}  "
        f"unnecessary biopsies={false_positives:2d}  missed cancers={false_negatives:2d}"
    )

threshold 0.1: recall=0.984  precision=0.818  unnecessary biopsies=14  missed cancers= 1
threshold 0.3: recall=0.953  precision=0.859  unnecessary biopsies=10  missed cancers= 3
threshold 0.5: recall=0.938  precision=0.938  unnecessary biopsies= 4  missed cancers= 4
threshold 0.7: recall=0.922  precision=1.000  unnecessary biopsies= 0  missed cancers= 5
threshold 0.9: recall=0.875  precision=1.000  unnecessary biopsies= 0  missed cancers= 8

The trade is explicit and monotone: lowering the threshold buys fewer missed cancers at the price of more unnecessary biopsies. No setting minimises both, and no statistical procedure picks for you — the choice follows from what the two errors cost, which is a clinical and ethical question rather than a numerical one.

That is the honest content of “choose the right metric”. You are choosing where on this curve to sit.

Regression makes the same choice about outliers

The pattern is not specific to classification. In regression the analogous question is how much you care about rare large errors, and the metrics disagree just as sharply. Two synthetic predictors — one uniformly a little wrong, one nearly perfect except for three badly missed points:

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

rng = np.random.default_rng(0)
truth = rng.normal(50, 10, 200)

uniformly_close = truth + rng.normal(0, 2, 200)
mostly_perfect = truth + rng.normal(0, 1, 200)
mostly_perfect[:3] += 40  # three large misses

print(f"{'predictor':30s} {'RMSE':>6} {'MAE':>6} {'R2':>7}")
for name, predicted in [
    ("uniformly close", uniformly_close),
    ("near-perfect, 3 big misses", mostly_perfect),
]:
    print(
        f"{name:30s} "
        f"{mean_squared_error(truth, predicted) ** 0.5:6.2f} "
        f"{mean_absolute_error(truth, predicted):6.2f} "
        f"{r2_score(truth, predicted):7.3f}"
    )

predictor                        RMSE    MAE      R2
uniformly close                  2.06   1.63   0.954
near-perfect, 3 big misses       4.95   1.37   0.735

MAE prefers the second predictor. RMSE rejects it. Both are right, because they measure different things: MAE averages absolute errors, so one miss of 40 counts as forty misses of 1; RMSE squares first, so it counts as very much worse than that.

Which you want depends on whether large errors are proportionally worse in your setting. Delivery-time estimates: probably MAE, since being an hour late is roughly ten times as annoying as six minutes late. Structural load prediction: certainly RMSE, because the rare large underestimate is the one that kills people. R², a rescaling of MSE, inherits RMSE’s position and adds a comparison against predicting the mean — useful for “is this model worth anything at all”, not for choosing between these two.

Caveat: MAPE has a hole in it

Mean absolute percentage error is popular because percentages feel interpretable, but it divides by the true value:

truth_with_zero = np.array([10.0, 5.0, 0.0, 8.0])
predicted = np.array([11.0, 5.5, 0.5, 8.0])
with np.errstate(divide="ignore", invalid="ignore"):
    mape = np.mean(np.abs((truth_with_zero - predicted) / truth_with_zero)) * 100
print(f"MAPE with a zero in the targets: {mape}")

MAPE with a zero in the targets: inf

Undefined — and in the merely near-zero case, enormous: a target of 0.01 predicted as 0.02 contributes 100% error on its own. It is also asymmetric, penalising over-prediction more heavily than under-prediction. Use it only when the target is strictly positive and comfortably away from zero.

What a metric actually is

The claim was that a metric encodes which errors you will tolerate. The evidence is two models that accuracy and AUC both scored identically while one missed twice as many cancers, and two regressors that MAE and RMSE ranked in opposite orders.

So the practical procedure is the reverse of the usual one. Do not fit a model and then look for a number to report. Start from the decision the model informs, work out what each kind of mistake costs the person on the other end, and pick the metric that reflects it — then threshold and optimise against that. When the costs are genuinely asymmetric and no standard metric matches, write your own; the evaluate library accepts a plain function and treats it like any built-in.

And report more than one. Every metric here was individually defensible and individually misleading, and the missed cancers were only visible because the confusion matrix was printed alongside. No single number survives contact with a real decision.