The Two Cultures, Part 2: Rashomon, or Many Equally Good Models

Five-variable regressions on 1976 ozone, trees on hepatitis, a network from 100 starts, and what bagging does to an unstable choice

Breiman’s Rashomon effect: many different models fit about equally well and tell different stories. All 142,506 five-variable regressions on the 1976 LA ozone record, pruned trees refitted after removing 3% of the hepatitis data, a small network restarted 100 times, and best-subset logistic regression single and bagged, with an explorer for the subset fits.
Statistics
Machine Learning
Reading
Author

Ravi Kalia

Published

November 18, 2026

The Two Cultures, Part 2: Rashomon, or Many Equally Good Models

A fitted model that explains the data well is usually one of many that explain it about as well, and those models can disagree about which inputs matter.

Anyone who has run stepwise regression twice, on last year’s data and then this year’s, has seen this. The fit barely changes and the list of “significant” variables does. The usual reading is that the second run found the truth or the first one did. Breiman’s reading, in section 8 of The Two Cultures, is that neither did: the data never picked one model, and the procedure picked one at random from a crowd.

Part 1 set out Breiman’s two cultures and re-ran his ozone forecast. This part tests his first lesson from machine learning on data close to his own. Part 3 takes the second and third.

This part covers two ideas:

1 The name

Rashomon is Akira Kurosawa’s 1950 film, built from two stories by Ryūnosuke Akutagawa, “In a Grove” and “Rashōmon”. In Breiman’s summary, four witnesses to a death in a forest report the same facts and tell very different stories about what happened. His statistical version (p. 206): in a class of functions there is often a multitude of different descriptions, all with about the same minimum error.

He gives three examples:

  1. Subsets. With 30 variables there are about 140,000 five-variable regressions, and generally many have residual sum of squares within 1% of the lowest. He prints three on disjoint variables, each telling a different story (p. 206).
  2. Trees. Removing a random 2–3% of the training set can give a quite different tree with almost the same test error (p. 206).
  3. Networks. A small net run 100 times on simple three-dimensional data from small random starting weights found 32 distinct minima, each a different picture, with about equal test error (p. 206).

The three equations he prints are illustrations, and the data behind all three examples are not given. Each is re-run here on public data close to his.

2 Data

  • 1976 Los Angeles ozone (mlbench::Ozone), as in Part 1: next-day maximum ozone from days \(t\), \(t-1\) and \(t-2\) of eight meteorological readings and ozone, plus season and a weekend flag. That makes 30 candidate predictors, the count in Breiman’s subset example. Forecasts made January–August train (189 day pairs), those made September–December test (93). Ozone is a forecasting problem of the kind he worked on, and its predictors are correlated the way weather is, which is what makes many subsets compete.
  • Hepatitis (UCI, contributed by Gail Gong): 155 patients, 19 covariates, 32 deaths. Breiman’s own example in section 11; Part 4 reproduces his analysis. Its small size and correlated laboratory values are the conditions under which a tree or a variable list is unstable, and its users wanted a short list of prognostic variables, which is the output that instability corrupts.

Sources, licences and checksums are in the series’ data/README.md.

3 Five-variable regressions

3.1 Every subset

There are \(\binom{30}{5} = 142{,}506\) five-variable regressions. Fitting them all takes a fraction of a second: each fit needs only a \(5 \times 5\) block of the Gram matrix \(X^\top X\), and the blocks are solved as one batch.

Code
names, fits = rashomon_exp.ozone_subsets()
ratio = fits["rss"] / fits["rss"].min()
test = fits["test_mse"]
counts = {t: int((ratio <= 1 + t).sum()) for t in (0.01, 0.02, 0.03, 0.05, 0.10)}
oz_in = names.index("ozone_lag0")
w5 = ratio <= 1.05
share_oz_5 = float(np.isin(fits["idx"][w5], [oz_in]).any(1).mean())
best = int(np.argmin(ratio))
best_test = int(np.argmin(test))
w3 = ratio <= 1.03
t3_lo, t3_hi = float(test[w3].min()), float(test[w3].max())
inc3 = np.array([np.isin(fits["idx"][w3], [j]).any(1).mean() for j in range(len(names))])
always3 = [names[j] for j in np.flatnonzero(inc3 == 1)]
rank_of_best_on_test = int((test < test[best]).sum()) + 1
  • Within 1% of the lowest training RSS: 4 models. Breiman says there are generally many; on 189 training days there are few.
  • Within 2%, 3%, 5% and 10%: 15, 42, 175 and 2,886 models.
  • Today’s ozone is in 100% of the models within 5%. The disagreement is about the meteorology.
  • On the test months, the models within 3% have test MSE from 24.5 to 29.2. The model with the lowest training RSS ranks 12,533 of 142,506 on test error.
Code
fig, axes = plt.subplots(1, 2, figsize=(7.5, 3.0))
ax = axes[0]
ax.hist(np.minimum(ratio, 2), bins=100, color=PURPLE, log=True)
ax.set_xlabel("training RSS ÷ best")
ax.set_ylabel("subsets")
ax = axes[1]
w10 = ratio <= 1.10
ax.scatter(ratio[w10], test[w10], s=4, color=TEAL, alpha=0.5, lw=0)
ax.scatter([ratio[best]], [test[best]], s=30, color=AMBER, zorder=3, label="lowest training RSS")
for t in (0.01, 0.03, 0.05):
    ax.axvline(1 + t, color=MUTED, ls="--", lw=0.7)
ax.set_xlabel("training RSS ÷ best")
ax.set_ylabel("test MSE (Sep–Dec)")
ax.legend(loc="upper left")
fig.tight_layout()
plt.show()
Left, a right-skewed histogram with a thin left tail of near-best models. Right, a scatter in which models within a few percent of the best training RSS spread widely in test error, and the model with the lowest training RSS sits above many others on test error.
Figure 1: All 142,506 five-variable regressions of next-day ozone. Left: training RSS as a multiple of the best, log-scale counts, ratios above 2 pooled at 2. Right: training RSS ratio against test MSE for the 2,886 models within 10% of the best; the dashed lines mark 1%, 3% and 5%.

3.2 Three pictures

Breiman’s three pictures share no variables. The three here are picked from the models within 5% so that each overlaps the others as little as possible. They cannot be disjoint, because all of them keep today’s ozone.

Code
order = np.where(w5)[0][np.argsort(ratio[w5])]
picked = []
for _ in range(3):
    best_i, best_ov = None, 99
    for i in order:
        if i in picked:
            continue
        ov = sum(len(set(fits["idx"][i]) & set(fits["idx"][j])) for j in picked)
        if ov < best_ov:
            best_i, best_ov = i, ov
    picked.append(best_i)


def equation(i):
    terms = [f"{'+' if b >= 0 else '−'} {abs(b):.2g}·{nice(names[j])}" for j, b in zip(fits["idx"][i], fits["coef"][i])]
    return f"{fits['intercept'][i]:.1f} " + " ".join(terms)


pics = pd.DataFrame([{"Picture": k + 1, "RSS ÷ best": f"{ratio[i]:.3f}",
                      "Test MSE": f"{test[i]:.1f}", "Equation": equation(i)}
                     for k, i in enumerate(picked)])
pic_vars = [set(nice(names[j]) for j in fits["idx"][i]) for i in picked]
md(pics)
Table 1: Three five-variable regressions within 5% of the best training RSS, chosen to overlap as little as possible. Coefficients are on the raw scales of the variables, to two significant figures; inversion base height is in feet, so its coefficient is small.
Picture RSS ÷ best Test MSE Equation
1 1.000 27.3 -13.6 + 0.53·ozone lag 0 + 0.0012·ibh lag 0 + 0.4·ibt lag 0 + 0.071·dpg lag 1 − 0.16·temp lag 2
2 1.018 29.2 -95.6 − 3.9·season_cos + 0.57·ozone lag 0 + 0.019·vh lag 0 − 0.032·dpg lag 0 − 0.13·ibt lag 1
3 1.032 28.9 -0.1 + 0.58·ozone lag 0 + 0.23·temp lag 0 − 0.2·ozone lag 1 + 0.31·ozone lag 2 − 0.16·ibt lag 2
  • Picture 1 forecasts from the inversion layer: its height and temperature at LAX today, plus a pressure gradient and an older temperature.
  • Picture 2 forecasts from the season, the upper air (500 mb height) and today’s pressure gradient, with no inversion height at all.
  • Picture 3 forecasts mostly from ozone itself, three days of it, plus today’s temperature and an inversion temperature from two days back.

A meteorologist reading one picture would learn something different from one reading another. The data do not choose between them.

3.3 Explorer

  1. At the starting tolerance of 1.0%, the tiles read 4 models using 9 different variables.
  2. Drag the tolerance to 3.0%. There are 42 models, and between them they use all 30 variables. Their test MSE runs from 24.5 to 29.2.
  3. Drag it to 10%. The inclusion bars flatten: apart from today’s ozone, no variable is in more than half the qualifying models.

4 Trees

4.1 Design

  • Data. Hepatitis, with a fixed random 30% of patients held out for testing (46 patients, 10 deaths) and missing values filled with training medians.
  • Model. CART with cost-complexity pruning, size chosen by ten-fold cross-validation with the one-standard-error rule, as in Breiman et al. (1984).
  • Perturbation. Remove a random 3% of the 109 training patients (3 patients), refit, repeat 100 times.
Code
tree_rows, tree_info = (lambda o: (o["rows"], o["info"]))(cache.obj(
    "trees_v2", lambda: dict(zip(["rows", "info"], rashomon_exp.tree_perturbations(reps=100)))))
tr = pd.DataFrame([{k: v for k, v in r.items() if k != "structure"} for r in tree_rows])
pert = tr[tr["rep"] > 0]
root_counts = pert["root"].value_counts()
sig_counts = pert["signature"].value_counts()
base_err = float(tr.loc[tr["rep"] == 0, "test_error"].iloc[0])
err_lo, err_hi = float(pert["test_error"].min()), float(pert["test_error"].max())
n_sig = int(pert["signature"].nunique())
n_stump = int(root_counts.get("leaf", 0))
split = pert[pert["root"] != "leaf"]
base_rate = tree_info["test_deaths"] / tree_info["n_test"]
split_lo, split_hi = float(split["test_error"].min()), float(split["test_error"].max())

4.2 Result

  • No split at all. The tree fitted to the full training set splits first on ascites. In 45 of the 100 perturbed fits the one-standard-error rule prunes the tree back to its root: on about 106 patients, predicting that everyone lives is within one standard error of the best cross-validated tree.
  • Root split, when there is one. Ascites 39 times and albumin 14 times; the rest go to bilirubin or prothrombin time.
  • Structure. Counting the root-only tree, the top two levels take 9 different forms.
  • Test error. 17.4% for the unperturbed tree. The root-only trees all score 21.7%, the share of the 46 test patients who died; trees with a split range from 15.2% to 23.9%. On 46 test patients one misclassification is 2.2 points: the best tree with a split gets 3 more patients right than predicting survival for everyone, and the worst gets 1 fewer.
Code
top = sig_counts.index[:7]
fig, axes = plt.subplots(1, 2, figsize=(7.5, 3.0), gridspec_kw={"width_ratios": [1.3, 1]})
ax = axes[0]
labels = ["root only" if s == "leaf" else s.replace("SPLEEN_PALPABLE", "SPLEEN").replace(" / ", " │ ")
          for s in top]
ax.barh(range(len(top))[::-1], sig_counts[top].to_numpy(), color=PURPLE)
ax.set_yticks(range(len(top))[::-1], labels, fontsize=7.5)
ax.set_xlabel("perturbed fits")
ax = axes[1]
for k, s in enumerate(top):
    e = pert.loc[pert["signature"] == s, "test_error"].to_numpy()
    jitter = np.random.default_rng(k).uniform(-0.15, 0.15, len(e))
    ax.scatter(100 * e, len(top) - 1 - k + jitter, s=8, color=TEAL, alpha=0.6, lw=0)
ax.set_yticks([])
ax.set_xlabel("test error (%)")
fig.tight_layout()
plt.show()
Left, a bar chart in which the root-only tree with no split is most common, followed by a one-split tree on ascites and trees rooted on albumin. Right, dot plots: every root-only tree scores about 22 percent, and the trees with splits spread between about 15 and 24 percent.
Figure 2: Pruned CART on hepatitis after removing 3 of 109 training patients, 100 times. Left: the split variables of the top two levels (root first, then left and right child; ‘leaf’ means no further split, and ‘root only’ a tree with no split at all). Right: test error on the 46 held-out patients for each form.

Breiman’s point survives in a sharper form: removing three patients changes not just which variable a tree uses but whether it uses any. The swap between ascites and albumin is not noise in the tree algorithm. The two are each other’s strongest correlate in these data, and Breiman’s section 11 analysis finds them to be surrogates: either carries most of what the other does. A tree must pick one, and which one depends on three patients. Part 4 returns to the pair.

5 Networks

5.1 Design

  • Data. The ozone training months, three predictors: today’s temperature, inversion base height and pressure gradient. Three standardised inputs is the “simple three-dimensional data” of Breiman’s account.
  • Model. One hidden layer of 3 tanh units (16 weights), light weight decay (\(\alpha = 0.01\)), fitted to convergence by L-BFGS from scikit-learn’s default random starting weights.
  • Restarts. 100, identical in everything but the starting weights.
  • Distinct minima. Two restarts count as the same minimum when their fitted values on the 189 training days differ by less than 0.1 ozone units (root mean square). Comparing functions rather than weights ignores the trivial symmetries: the hidden units can be relabelled, and each unit’s signs flipped, without changing the function.
Code
net_rows = cache.obj("nets_v2", lambda: {"rows": rashomon_exp.net_restarts(reps=100)})["rows"]
labels = {t: rashomon_exp.distinct_minima(net_rows, t) for t in (0.05, 0.1, 0.2)}
nets = pd.DataFrame([{"rep": r["rep"], "train_mse": r["train_mse"], "test_mse": r["test_mse"]}
                     for r in net_rows])
nets["group"] = labels[0.1]
n_groups = {t: int(v.max() + 1) for t, v in labels.items()}
g = nets.groupby("group").agg(k=("rep", "size"), train=("train_mse", "mean"), test=("test_mse", "mean"))
big = g.sort_values("k", ascending=False)
q10, q90 = nets["test_mse"].quantile([0.1, 0.9])

5.2 Result

  • Distinct functions. 28 from 100 starts; the count is 28 at a tolerance of 0.05 and 28 at 0.2, so it does not hinge on the threshold. Breiman found 32 on his data; the count depends on the data, the width of the net and the weight decay, so the agreement in size means little.
  • Most common minimum. Reached from 31 starts; 14 minima are reached from one start only.
  • Training error ranges from 24.5 to 27.5.
  • Test error. 80% of restarts fall between 31.7 and 36.5; the worst is 49.1. “About equal test error” holds for most, not all.
Code
cols = [PURPLE, TEAL, AMBER, PINK, BLUE, INK]
top6 = list(big.index[:6])
fig, axes = plt.subplots(1, 2, figsize=(7.5, 3.0))
ax = axes[0]
for gi, c in zip([None] + top6, [RULE] + cols):
    m = ~nets["group"].isin(top6) if gi is None else nets["group"] == gi
    ax.scatter(nets.loc[m, "train_mse"], nets.loc[m, "test_mse"], s=14, color=c, alpha=0.85, lw=0)
ax.set_xlabel("training MSE")
ax.set_ylabel("test MSE (Sep–Dec)")
ax = axes[1]
xs = np.linspace(-2, 2, 21)
for gi, c in zip(top6, cols):
    r = net_rows[int(nets.loc[nets["group"] == gi, "rep"].iloc[0])]
    ax.plot(xs, r["pd_temp"], color=c, lw=1.6)
ax.set_xlabel("today's temperature (standardised)")
ax.set_ylabel("predicted next-day ozone")
fig.tight_layout()
plt.show()
Left, a scatter in which training errors span a narrow band while test errors mostly cluster between 31 and 37 with a few outliers. Right, six curves of predicted ozone against standardised temperature that agree near the middle and diverge at the extremes, some flattening and some rising steeply.
Figure 3: One 3-unit network, 100 random starts, ozone training months. Left: training against test MSE, one point per start, coloured by minimum (the six most common in colour, the rest grey). Right: the fitted effect of today’s temperature (partial dependence, other inputs at their observed values) for the six most common minima.

The minima agree where the data are dense and part company at the edges, which is where a forecaster most wants the model’s opinion: hot days.

6 Best-subset logistic regression, single and bagged

6.1 Breiman’s account

Breiman reports that cutting a logistic or Cox model from 15 variables to 4 makes it unstable: perturb the data and a different four-variable model, with a different conclusion, appears (p. 206). He cites Arena et al. (2000), who bagged logistic regressions on toxicity data with best-subset selection from 15 covariates to 4 and found the bagged model significantly more accurate on a test set, and more stable. That work was a conference presentation, and the data are not public.

6.2 Design

  • Data. Hepatitis, 19 covariates, as a stand-in for the toxicity data.
  • Selection rule. The best four covariates by the score statistic, the criterion behind SAS’s best-subset search for logistic regression. At the intercept-only fit it equals \(n R^2\) of a least-squares fit of the 0/1 outcome on the subset, so all \(\binom{19}{4} = 3{,}876\) subsets are scored in one batch.
  • Fit. Median imputation and standardisation inside the training data, then unpenalised logistic regression on the chosen four.
  • Bagged version. Repeat selection and fit on 50 bootstrap resamples of the training data and average the predicted probabilities.
  • Evaluation. Breiman’s protocol: 100 random 10% holdouts, both versions on the same splits.
  • Stability. In the spirit of Diaconis and Efron (1983): rerun the selection on 500 bootstrap resamples of all 155 patients and count which covariates it picks.
Code
bag = cache.frame("bagged_v1", lambda: rashomon_exp.bagged_logistic(reps=100))
diff = bag["single"] - bag["bagged"]
sel = cache.obj("selection_v1", lambda: dict(zip(["picks", "full", "names"],
                                                 rashomon_exp.selection_frequency(boots=500))))
picks, full, hep_names = sel["picks"], sel["full"], sel["names"]
freq = pd.Series(collections.Counter(i for p in picks for i in p)).reindex(range(19), fill_value=0) / len(picks)
n_sets = len({tuple(sorted(p)) for p in picks})
modal_set, modal_n = collections.Counter(tuple(sorted(p)) for p in picks).most_common(1)[0]
either = float(np.mean([(11 in p) or (16 in p) for p in picks]))
both = float(np.mean([(11 in p) and (16 in p) for p in picks]))

6.3 Result

  • Accuracy. The single best-subset model misclassifies 19.1% of held-out patients, the bagged one 16.6%. The paired difference is 2.6 percentage points, with a standard error of 0.6. Calling every patient a survivor scores 20.6%, so the single model barely beats the baseline.
  • Full-data choice. Spiders (11), Ascites (12), Bilirubin (14), Albumin (17), numbered as Breiman numbers covariates.
  • Bootstrap choices. 205 different four-variable sets in 500 resamples. The most common, Steroid, Spiders, Ascites, Bilirubin, appears 22 times.
  • The surrogate pair. Ascites (12) or albumin (17) is in 87% of the chosen sets; both are in only 11%. Neither is picked as often as the pair is, because a set rarely needs both.
Code
fig, ax = plt.subplots(figsize=(7.5, 2.8))
colors = [AMBER if i in (11, 16) else PURPLE for i in range(19)]
ax.bar(range(1, 20), freq.to_numpy(), color=colors)
ax.set_xticks(range(1, 20), [f"{i + 1}\n{hep_names[i][:5].title()}" for i in range(19)], fontsize=6.5)
ax.set_ylabel("share of resamples")
ax.set_ylim(0, 1)
fig.tight_layout()
plt.show()
Bar chart over the 19 covariates. Bilirubin, ascites and spiders are picked most often, albumin less often; no covariate is picked in more than about three quarters of resamples.
Figure 4: Share of 500 bootstrap resamples of the hepatitis data in which the best-subset rule picks each covariate, numbered as in Breiman’s paper. Ascites (12) and albumin (17) are highlighted.

Bagging helps here for the reason Breiman gives. The single model’s error includes the cost of picking the wrong four, and averaging over 50 picks lowers that cost. The price is the one he names: the bagged model has no four variables to report.

7 Hoadley’s Fat Scorecard

Hoadley’s comment (p. 222) reports the same effect from credit scoring. Fitting logistic scorecards on 36 candidate characteristics, he found that every model with anywhere from 27 to 36 characteristics had the same test performance, and he named it Rashomon. He kept the 27-characteristic model: 162 coefficients, one of them significant at 5%, which slightly beat a neural network with 2,000 weights. Part 3 returns to it as an argument about interpretability.

8 Since 2001

The idea has since been given a name and a formal treatment.

  • The Rashomon set. Semenova, Rudin and Parr (2022) define it as all models in a class within a tolerance of the best loss, which is what the subset explorer shows. They argue that when the set is large it often contains a simple model.
  • Model class reliance. Fisher, Rudin and Dominici (2019) measure a variable’s importance as a range over the Rashomon set rather than for one model. This post does not compute that range, but the subsets fix one end of it: a model that leaves a variable out does not rely on it, and within 3% of the best RSS 29 of the 30 candidates are left out of at least one qualifying model. The exception is today’s ozone, which every one of them uses.
  • Underspecification. D’Amour et al. (2022) find the same effect in deep learning: models trained with different seeds agree on the held-out test set and disagree under distribution shift. The networks in Section 5 diverge at the edge of the data.

9 Constraints

  • Sample sizes are small. 189 training days for the subsets and networks, 109 training patients for the trees, 155 patients for bagging. Breiman’s “many within 1%” presumably came from a larger sample; ours gives 4.
  • Test sets are small. 93 ozone days and 46 hepatitis patients. Differences in test error of a few percent between Rashomon models are within the noise of a single test set.
  • The selection rule is a stand-in. Arena et al. used best-subset selection from 15 to 4 on toxicity data; this is the same rule from 19 to 4 on hepatitis.
  • Minima counts depend on choices. Width, weight decay and the tolerance all move the count of distinct network minima.

10 Reproduction status

Table 2: Part 2’s rows of the series reproduction table.
Experiment Paper’s data Data used here Paper’s result Our result Notes
Subsets (§8) 30 variables, not given 30 ozone predictors, 1976 many within 1% of best RSS 4 within 1%, 175 within 5%; different stories all share today’s ozone
Trees (§8) not given hepatitis, 3% removed different tree, same test error no split 45, root ascites 39, albumin 14 of 100; 9 top-level forms; test error 15%–24% reproduced in kind
Networks (§8) simple 3-D data, not given 3 ozone predictors 32 distinct minima, equal test error 28 distinct functions; 80% of test MSE in 31.7–36.5 count depends on architecture
Bagged logistic (§8) toxicity data (Arena et al.) hepatitis, 19 → 4 bagged significantly more accurate, more stable 19.1% single vs 16.6% bagged; 205 distinct 4-sets in 500 bootstraps stand-in data

Same. Error. Different. Stories. Instability. Picks. One. Averaging. Keeps. Accuracy.

11 References