flowchart LR X["x: inputs"] --> N["nature"] --> Y["y: responses"]

Leo Breiman’s Statistical Modeling: The Two Cultures (2001) argued that statistics was checking its models with the wrong test, and this series re-runs as much of the paper’s evidence as public data allow.
Anyone who has fitted a regression knows the routine. Fit the model, read \(R^2\), glance at a residual plot, and if nothing looks alarming, read the coefficients as facts about the world: this variable matters, that one does not. Breiman’s complaint was that nothing in the routine checks the assumption the reading depends on, that the fitted equation works the way nature does. His alternative was blunt: hold some data back and see whether the model predicts it. A quarter of a century later the paper is still the usual citation for the split between statistics and machine learning, and its claims can now be tested with tools he did not have.
This is the first of five parts.
- Two ways to fill a black box (this part): the paper’s two goals and two cultures, the consulting projects behind them, and the question of how a model should be checked.
- Rashomon: many different models fit about equally well.
- Occam and Bellman: forests against single trees on Breiman’s ten benchmark sets, and whether more features help.
- Information from a black box: the hepatitis, liver and lymphoma analyses, and what variable importance can and cannot say.
- The pushback, and what prediction cannot tell you: Cox, Efron, Hoadley, Parzen, Breiman’s reply, causality, and the full reproduction table.
This part covers two ideas:
- Two cultures. A data model assumes the mechanism inside the box and estimates its parameters; an algorithmic model treats the mechanism as unknown and looks for a function that predicts.
- Two kinds of check. A goodness-of-fit test asks whether the data contradict the assumed model; a held-out test asks whether the model predicts new cases. The second section of experiments asks when these two checks disagree.
1 The paper
Statistical Science published the paper in August 2001 (volume 16, number 3, pages 199–231) with four invited comments and a rejoinder. All page numbers in this series are journal pages.
- Author. Breiman spent seven years as an academic probabilist, then thirteen as a freelance consultant, and joined the Berkeley statistics department in 1980 (p. 200). He co-wrote the CART book (1984) and introduced bagging (1996) and random forests (2001).
- Discussants. D. R. Cox, Brad Efron, Bruce Hoadley and Emanuel Parzen. Part 5 sets out each comment.
- Structure. Sections 3–7 explain how his view formed and what he saw wrong with data models. Sections 8–10 give three lessons from machine learning, named after a film, a friar and a mathematician: Rashomon, Occam and Bellman. Section 11 claims an algorithmic model can say more about the mechanism than a data model can.
2 Two goals, two cultures
Breiman starts from a picture. Inputs \(x\) go into a box, responses \(y\) come out, and nature does something inside (p. 199).
The analyst can want two things from the box (p. 199):
- Prediction: what \(y\) will be for future inputs.
- Information: how nature associates \(y\) with \(x\).
The two cultures differ in what they put inside the box.
| Data modeling culture | Algorithmic modeling culture | |
|---|---|---|
| Inside the box | an assumed stochastic model, e.g. \(y = f(x, \text{noise}, \text{parameters})\) | unknown and possibly unknowable |
| Typical fillings | linear regression, logistic regression, the Cox model | decision trees, neural networks, later forests and SVMs |
| Fitted object | parameter estimates, read as facts about nature | a function \(f(x)\) that predicts \(y\) |
| Validation | yes–no goodness-of-fit tests and residual checks | predictive accuracy on data not used to fit |
| Assumption in the theory | the model is true, up to parameters | the data are i.i.d. draws from some distribution |
| Breiman’s head count | 98% of all statisticians | 2% of statisticians, many people in other fields |
The head count drew the most direct responses. Cox called the paper’s picture of mainstream statistics partly “a caricature” (p. 216). Efron said the prediction culture at Stanford alone was larger than 2% (p. 219). Breiman replied that the dispute was over the word “mainstream”: count the people running canned regression packages, not the people writing journal papers, and his version wins by at least 100 to 1 (p. 228).
3 Where the view came from
Breiman says his views formed on consulting projects (p. 200). He describes two in detail.
3.1 Ozone, Los Angeles, mid-1970s
- Problem. Smog alerts were issued in the morning, too late to change commuting; the EPA funded an effort to forecast ozone 12 hours ahead (p. 200).
- Data. Over 450 meteorological variables, hourly and daily, from stations as far away as Oregon and Arizona, for seven years, with predictors reaching several days back (p. 201).
- Validation. First five years to train, last two to test (p. 201).
- Method. Large linear regressions with variable selection, then quadratic terms and interactions of the retained variables, then selection again (p. 201).
- Outcome. A failure: the false-alarm rate was too high. Breiman regretted not being able to revisit it with modern tools (p. 201).
3.2 Chlorine from mass spectra
- Problem. Predict whether a compound contains chlorine from its mass spectrum, to triage the EPA’s toxicity screening (p. 201).
- Data. Known structures and spectra for 30,000 compounds, split 25,000 / 5,000 at random (p. 201).
- Method. Linear and quadratic discriminant analysis struggled because a spectrum’s length varies with molecular weight. A decision tree over 1,500 yes–no questions, designed with chemists’ knowledge of chlorine signatures, reached 95% accuracy on both classes (p. 201).
The chlorine data were never released, so that project is described here and not reproduced.
3.3 What he took away
Breiman lists five lessons (p. 201):
- Focus on finding a good solution.
- Live with the data before modeling.
- Search for a model that solves the problem, algorithmic or data.
- Predictive accuracy on test sets is the criterion of a good model.
- Computers are indispensable.
Item 4 is the one the rest of the paper defends.
4 Data: the 1976 Los Angeles ozone record
The project’s seven-year, 450-variable database is not public. The closest public record is one year of the same kind of data from the same basin, which Breiman and Friedman (1985) used to introduce their ACE transformation method.
- Source.
mlbench::Ozone(366 days of 1976, missing values kept) andfaraway::ozone(330 complete days), both copied from the R packages’ data files. Download dates, licences and checksums are in the series’data/README.md. - Variables. The daily maximum one-hour-average ozone reading, and eight meteorological readings: 500 millibar pressure height at Vandenberg AFB, wind speed and humidity at LAX, temperature at Sandburg, inversion base height and temperature at LAX, the LAX–Daggett pressure gradient, and visibility at LAX.
mlbenchalso has a second temperature, at El Monte, missing on 139 days and left out here. Neither package documents the ozone units or station clearly (farawaysays ppm, which cannot be right for readings from 1 to 38), so ozone is quoted in the data’s own units. - Who collected it and why. Routine air-quality and weather monitoring in the Basin, collected for the smog alerts and weather forecasting the ozone project served.
- What this post asks of it. Whether a forest predicts next-day ozone better than the selected linear regressions of the 1970s project, under a time-ordered split.
- Cost of a wrong answer. A missed alert leaves children on playgrounds on a smog day; a false alarm keeps government workers home for nothing (p. 200). Breiman’s project failed on false alarms.
- Why these methods. They are the two the paper compares: the project’s own procedure, and the forest he wished he had had. Section 5.1 adds three the paper does not run: the lasso, gradient boosting and a pretrained forecasting model.
- What it is not. One year against seven, nine variables against 450, one station against dozens. It is a stand-in.
Two quirks decided which copy to use. faraway::ozone labels each row with a doy field, but it repeats at month boundaries (two rows each for days 61, 121, 181, …) and runs to 390, so it cannot pair a day with the next one. mlbench::Ozone carries month and day of month, so the next-day design uses it, keeping only rows where all the needed days are present.
Code
df = ozone.design()
Xtr, ytr, Xte, yte = ozone.split(df)
n_pairs, n_tr, n_te = len(df), len(ytr), len(yte)
alert_tr, alert_te = int((ytr >= ozone.ALERT).sum()), int((yte >= ozone.ALERT).sum())5 The ozone project, re-run
The design follows the project’s description as closely as nine variables allow.
- Target. Day \(t+1\)’s maximum ozone.
- Alert. A forecast of 20 or more counts as an alert. The training months reach 20 on 21% of days. Because the units are undocumented this is a relative threshold, not the legal alert level.
- Predictors. Days \(t\), \(t-1\) and \(t-2\) of the eight meteorological readings and of ozone itself (27 columns), plus season (sine and cosine of day of year) and whether day \(t+1\) is a weekend: 30 candidates. The project included pollutant readings and days back, so ozone and its lags are fair inputs.
- Split. By the day the forecast is made: train on pairs made January–August (189 day pairs, the last forecasting 1 September), test on pairs made September–December (93). The project’s split was also by time, but its five training years covered every season; here the test months are autumn and winter, which the training months only partly cover.
- Linear route. Forward selection on the 30 columns; then squares and pairwise products of the retained columns, and forward selection again. Each step’s size is chosen by five-fold time-ordered cross-validation inside the training months.
- Forest route. A random forest of 500 regression trees, one third of the columns tried at each split (Breiman’s default for regression), no tuning, five seeds.
- Baseline. Persistence: tomorrow’s ozone equals today’s.
Code
lin = ozone.linear_route(Xtr, ytr, Xte)
rf_preds = [ozone.forest_route(Xtr, ytr, Xte, seed=s)[0] for s in range(5)]
persist = Xte["ozone_lag0"].to_numpy()
rows = [("Persistence (today's ozone)", ozone.scores(persist, yte)),
("Linear, selected", ozone.scores(lin["stage1_pred"], yte)),
("Linear, selected + squares and products", ozone.scores(lin["stage2_pred"], yte))]
rf_scores = [ozone.scores(p, yte) for p in rf_preds]
rf_rmse = [s["rmse"] for s in rf_scores]
rows.append(("Random forest, seed 0", rf_scores[0]))
tab = pd.DataFrame([{"Model": name, "Test RMSE": f"{s['rmse']:.2f}",
"Test R²": f"{s['r2']:.2f}", "Alerts issued": s["alerts_issued"],
"Hits": s["hits"], "False alarms": s["false_alarms"]}
for name, s in rows])
rmse = dict(zip(["persist", "linear", "quad"], [s["rmse"] for _, s in rows[:3]]))
n_terms1, n_terms2 = len(lin["stage1_terms"]), len(lin["stage2_terms"])
rf_lo, rf_hi = min(rf_rmse), max(rf_rmse)
md(tab)| Model | Test RMSE | Test R² | Alerts issued | Hits | False alarms |
|---|---|---|---|---|---|
| Persistence (today’s ozone) | 6.06 | 0.20 | 8 | 2 | 6 |
| Linear, selected | 5.28 | 0.39 | 4 | 1 | 3 |
| Linear, selected + squares and products | 5.14 | 0.42 | 6 | 0 | 6 |
| Random forest, seed 0 | 4.94 | 0.47 | 9 | 3 | 6 |
- The first selection kept 12 of the 30 columns; the second kept 13 terms, most of them products.
- The forest’s test RMSE is 4.94 to 5.00 over five seeds, against 5.14 for the selected quadratic regression and 5.28 for the selected linear one. The forest is better on these days, by less than the gap between either and persistence (6.06). Section 5.1 finds the margin over the quadratic regression inside the noise of 93 test days.
- The alert columns rest on 7 alert days, too few to rank the methods. Every method issued false alarms, which is how the project failed.
Code
# Rows are indexed by day t; the target is day t + 1 (day of year d is 1 Jan + d - 1).
days = pd.to_datetime("1976-01-01") + pd.to_timedelta(Xte.index.to_numpy(), unit="D")
fig, ax = plt.subplots(figsize=(7.5, 3.2))
ax.plot(days, yte, color=INK, lw=1.4, label="observed")
ax.plot(days, rf_preds[0], color=PURPLE, lw=1.2, label="random forest")
ax.plot(days, lin["stage2_pred"], color=AMBER, lw=1.2, label="selected regression")
ax.axhline(ozone.ALERT, color=MUTED, ls="--", lw=0.8)
ax.set_ylabel("next-day max ozone")
ax.legend(loc="lower center", bbox_to_anchor=(0.5, 1.0), ncol=3)
ax.xaxis.set_major_locator(mdates.MonthLocator())
ax.xaxis.set_major_formatter(mdates.DateFormatter("%b"))
fig.tight_layout()
plt.show()
Both forecasts track the autumn decline and both fall short of the highest peaks in September and October. A forest is no cure for a short, single-year record.
5.1 Models the paper does not run
In 2001 the forest was the newest tool in the comparison. Three kinds of model that the paper does not run are now standard first tries for a forecast like this one. The lasso and gradient boosting were published within five years of the paper, Chronos-2 in 2025.
- Lasso (Tibshirani 1996): a linear regression on all 30 candidates whose coefficients are shrunk toward zero, many of them to zero. It selects variables in one fit, where the 1970s route searched.
- Gradient boosting (Friedman 2001): a sum of small trees, each fitted to the errors left by the trees before it. The version here is scikit-learn’s
HistGradientBoostingRegressor, which bins every input before searching for splits, the method LightGBM made standard (Ke et al. 2017). - Chronos-2 (Ansari et al. 2025): a transformer pretrained on a large collection of time series, which forecasts a new series from its own history with no training on it. It runs twice: on the ozone record alone, and with the eight meteorological series as covariates and the next day’s weekend flag as a known future input. A forecast made on day \(t\) sees every day from 1 January to day \(t\), gaps included; the trained models choose their inputs from days \(t\), \(t-1\) and \(t-2\).
Tuning needs data that the test months do not touch, so the training months are split again by forecast day.
Code
from twocultures import modern
tr_m, dv_m, _ = modern.masks(df)
n_trm, n_dev = int(tr_m.sum()), int(dv_m.sum())- Train. Pairs made January–June (151 day pairs).
- Dev. Pairs made July–August (38). The lasso’s penalty, the forest’s split and leaf settings, and boosting’s learning rate, tree size, leaf size, penalty and number of trees are each chosen by dev RMSE from a fixed grid (
twocultures/modern.py). - Test. September–December, as before. Every tuned model is refit on January–August before it is scored.
- Chronos-2 is not tuned or trained, so its dev score is a report, not a choice.
- Uncertainty. The forest, the tuned forest and boosting run with five seeds. The interval on each model’s test RMSE minus the untuned forest’s is a 95% circular moving-block bootstrap: 4,000 resamples of runs of seven consecutive test forecasts, a precaution in case errors persist from one day to the next. Days with missing data are already out of the design, so a run can span more than a week.
Code
mod = pd.concat([cache.frame(modern.CACHE_TAG, modern.run), modern.chronos()],
ignore_index=True)
mod["y"] = df["target"].loc[mod["doy"]].to_numpy()
def pick(model, split, seed=0):
return mod[(mod["model"] == model) & (mod["split"] == split) & (mod["seed"] == seed)]
# The forest and the selected regression here are the rows of the first table.
assert np.allclose(pick("forest", "test")["pred"], rf_preds[0])
assert np.allclose(pick("linear_quad", "test")["pred"], lin["stage2_pred"])
names = {"persistence": "Persistence (today's ozone)",
"linear_quad": "Linear, selected + squares and products",
"forest": "Random forest, untuned", "lasso": "Lasso",
"forest_tuned": "Random forest, tuned", "boosting": "Gradient boosting",
"chronos_ozone": "Chronos-2, ozone only",
"chronos_weather": "Chronos-2, ozone + meteorology"}
ref = pick("forest", "test")
dev_rmse, test_rng, ci, rows_m = {}, {}, {}, []
for key, label in names.items():
d, t = pick(key, "dev"), pick(key, "test")
# The bootstrap pairs rows by position, so every model must cover the same days in order.
assert (t["doy"].to_numpy() == ref["doy"].to_numpy()).all()
dev_rmse[key] = modern.rmse(d["pred"], d["y"])
seeds = sorted(mod.loc[mod["model"] == key, "seed"].unique())
r = [modern.rmse(g["pred"], g["y"]) for g in (pick(key, "test", s) for s in seeds)]
test_rng[key] = (min(r), max(r))
ci[key] = modern.block_bootstrap(t["pred"], ref["pred"], t["y"])
s = ozone.scores(t["pred"].to_numpy(), t["y"].to_numpy())
rows_m.append({
"Model": label, "Dev RMSE": f"{dev_rmse[key]:.2f}",
"Test RMSE": (f"{r[0]:.2f}" if len(r) == 1 else f"{min(r):.2f}–{max(r):.2f}"),
"Test − forest, 95% interval": ("—" if key == "forest"
else f"{ci[key][0]:+.2f} to {ci[key][1]:+.2f}"),
"Alerts issued": s["alerts_issued"], "Hits": s["hits"],
"False alarms": s["false_alarms"]})
dev_order = sorted(dev_rmse, key=dev_rmse.get)
trained = ["linear_quad", "forest", "lasso", "forest_tuned", "boosting"]
worse_than_forest = [k for k in names if k != "forest" and ci[k][0] > 0]
assert worse_than_forest == ["persistence"]
assert not any(ci[k][1] < 0 for k in names if k != "forest")
# The same verdict seed by seed: each seeded model against the forest of that seed.
for k in ["forest_tuned", "boosting"]:
for s in modern.SEEDS:
a, b = pick(k, "test", s), pick("forest", "test", s)
lo_s, hi_s = modern.block_bootstrap(a["pred"], b["pred"], a["y"])
assert lo_s < 0 < hi_s, (k, s, lo_s, hi_s)
assert dev_order[0] == "persistence" and set(dev_order[1:3]) == {"chronos_ozone", "chronos_weather"}
lo = {k: v[0] for k, v in test_rng.items()}
assert min(lo, key=lo.get) == "boosting" and max(lo, key=lo.get) == "persistence"
assert lo["forest"] < lo["lasso"] < lo["linear_quad"]
assert min(lo["chronos_ozone"], lo["chronos_weather"]) > max(lo[k] for k in trained)
tuned_mf = pick("forest_tuned", "test")["params"].iloc[0]
assert tuned_mf.startswith("max_features=1.0,")
assert all(r["False alarms"] > 0 for r in rows_m)
cover = {k: float(((pick(k, "test")["q10"] <= pick(k, "test")["y"])
& (pick(k, "test")["y"] <= pick(k, "test")["q90"])).mean())
for k in ["chronos_ozone", "chronos_weather"]}
# The equal average dropped from the table (see Constraints); seed 0 of each part.
avg = np.mean([pick(k, "test")["pred"].to_numpy()
for k in ["forest_tuned", "boosting", "chronos_weather"]], axis=0)
avg_rmse = modern.rmse(avg, ref["y"])
assert test_rng["boosting"][0] <= avg_rmse <= test_rng["forest"][1]
md(pd.DataFrame(rows_m))| Model | Dev RMSE | Test RMSE | Test − forest, 95% interval | Alerts issued | Hits | False alarms |
|---|---|---|---|---|---|---|
| Persistence (today’s ozone) | 5.61 | 6.06 | +0.45 to +1.76 | 8 | 2 | 6 |
| Linear, selected + squares and products | 6.44 | 5.14 | -0.15 to +0.56 | 6 | 0 | 6 |
| Random forest, untuned | 6.42 | 4.94–5.00 | — | 9 | 3 | 6 |
| Lasso | 6.37 | 5.07 | -0.15 to +0.45 | 6 | 2 | 4 |
| Random forest, tuned | 6.35 | 4.97–5.05 | -0.14 to +0.24 | 10 | 3 | 7 |
| Gradient boosting | 6.55 | 4.67–4.93 | -0.34 to +0.38 | 10 | 4 | 6 |
| Chronos-2, ozone only | 5.91 | 5.46 | -0.13 to +1.16 | 8 | 2 | 6 |
| Chronos-2, ozone + meteorology | 5.97 | 5.30 | -0.26 to +0.97 | 7 | 2 | 5 |
- No model separates from the untuned forest on the test months. Only persistence’s interval lies wholly above zero; every other interval includes zero, and for the tuned forest and boosting it does so on all five seeds. Boosting has the lowest error in the table, 4.67–4.93 over five seeds, against 4.94–5.00 for the forest.
- The lasso does as well as the 1970s search, in one fit. It scores 5.07, between the forest and the selected quadratic regression (5.14).
- Tuning bought nothing. The tuned forest settled on max_features=1.0, min_samples_leaf=1 and scored 4.97–5.05. With
max_features=1.0it tries every column at every split, which makes it bagged trees rather than a forest. - The dev months rank the models almost in reverse. On July–August persistence is best (5.61), the two Chronos-2 runs come next, and every trained model is worse than all three. On the test months persistence is last. A dev set drawn from summer is a poor sample of autumn, the same limit Hoadley raises against test sets in Section 8.
- Chronos-2, with no training on this record, lands between persistence and the trained models. It brings the error from 6.06 to 5.46 on ozone alone and to 5.30 with the meteorology. It is also the only model here that gives a forecast interval: its nominal 80% intervals held the next day’s ozone on 77% of test days on ozone alone and 67% with the meteorology.
- Alerts still cannot rank them. The test months hold 7 alert days, and every model issued false alarms.
On 93 autumn days the gap between a selected regression and a model that assumes nothing cannot be resolved, and the models now standard for the job do not resolve it either.
6 Conclusions are about the model
Section 5 of the paper states the objection to data models in two lines (p. 202):
- Conclusions drawn from a fitted model are about the model’s mechanism, not nature’s.
- If the model emulates nature poorly, the conclusions may be wrong.
Nobody disputes either line. The dispute is over whether the standard checks catch a poor emulation. Breiman argued they do not: goodness-of-fit tests and residual plots give a yes–no answer, they lose power as dimension grows, and they are invalid once the model has been tinkered with on the same data (p. 203). He cited William Cleveland, one of the inventors of residual analysis, as saying it cannot find lack of fit beyond four or five dimensions (p. 203).
7 Goodness-of-fit tests against held-out error
7.1 Breiman’s claim
Breiman reports one experiment (p. 203): a simulated regression in seven dimensions with a controlled amount of nonlinearity, in which standard goodness-of-fit tests did not reject linearity until the nonlinearity was extreme. He gives no sample size, no form of nonlinearity and no names of tests. He also cites Bickel, Ritov and Stoker, then unpublished (published 2006), for the theory: a test has little power unless the direction of the alternative is specified, so omnibus tests have little power.
7.2 Design
The experiment here asks a two-sided question: does the test reject before a flexible model gains anything out of sample, or only after?
- Predictors. Seven standardised meteorological columns of
faraway::ozone(330 days):vh,wind,humidity,temp,ibh,dpg,vis. They are real data, so their correlations are those of the basin’s weather, not of a convenient simulation. - Response (simulated). \(y = \sqrt{1-\eta}\,L + \sqrt{\eta}\,G + \varepsilon\), where \(L\) is a fixed linear combination of the seven columns, \(G\) a nonlinear term, both scaled to unit variance, and \(\varepsilon \sim N(0, 0.5)\). The share \(\eta\) of signal variance that is nonlinear runs from 0 to 0.8.
- Nonlinear term, fixed in advance. \(G = \max(0, \text{temp}) \cdot \max(0, -\text{ibh})\): ozone builds when it is hot and the inversion base is low.
- Tests on the OLS fit, at 5%.
- RESET (Ramsey): do squares and cubes of the fitted values improve the fit?
- Quadratic F test: do all 35 squares and pairwise products of the inputs improve it? This is the omnibus test.
- Rainbow (Utts): does a fit to the central half of the data predict the rest worse than chance allows?
- Held-out comparison. On a random two-thirds of the rows fit OLS and a random forest; score both on the remaining third. A ratio of forest to OLS test MSE below 1 means the forest predicts better.
- Replicates. 200 noise draws per level of \(\eta\).
Code
res = cache.frame("gof_v1", lambda: gof.run(reps=200))
res["ratio"] = res["mse_rf"] / res["mse_ols"]
summ = res.groupby(["shape", "eta"]).agg(
reset=("p_reset", lambda p: (p < 0.05).mean()),
quad=("p_quad", lambda p: (p < 0.05).mean()),
rainbow=("p_rainbow", lambda p: (p < 0.05).mean()),
ratio=("ratio", "median"),
).reset_index()
def at(shape, eta, col):
return float(summ[(summ["shape"] == shape) & np.isclose(summ["eta"], eta)][col].iloc[0])
def first_eta(shape, cond):
s = summ[summ["shape"] == shape]
hit = s[cond(s)]
return float(hit["eta"].min()) if len(hit) else float("nan")
reg_break_even = first_eta("regime", lambda s: s["ratio"] < 1)
wig_break_even = first_eta("wiggle", lambda s: s["ratio"] < 1)
wig_max_reject = float(summ[summ["shape"] == "wiggle"][["reset", "quad", "rainbow"]].max().max())7.3 Result for the design fixed in advance
The result runs against Breiman’s claim.
- At \(\eta = 0.1\), RESET rejects linearity in 76% of replicates and the quadratic F test in 72%. The forest’s test error is then 1.19 times OLS’s: it is worse.
- The forest first beats OLS at η = 0.3, where both directed tests reject in every replicate.
- The rainbow test never rises above its 5% false-alarm rate: its notion of misfit, curvature at the edges of the data, does not match this one.
On this design a statistician who ran RESET would learn of the nonlinearity well before a forest could use it.
7.4 A second shape, chosen afterwards
Bickel, Ritov and Stoker’s point suggests why: the product term lies close to the squares-and-products directions that RESET and the quadratic test look in. A second nonlinear term was chosen after seeing the first result, to look for the case Breiman describes: \(G = \sin(3 \cdot \text{temp})\), a wiggle that no low-order polynomial points at.
- No test rejects in more than 23% of replicates at any \(\eta\) up to 0.8.
- The forest beats OLS from η = 0.2, and at \(\eta = 0.5\) its test error is 0.68 times OLS’s.
This is Breiman’s case: a linear fit that passes every test the analyst ran, with a test error half as large again as that of a model that assumed nothing.
Code
fig, axes = plt.subplots(2, 2, figsize=(7.5, 4.8), sharex=True, sharey="row")
titles = {"regime": "hot × low inversion (fixed in advance)", "wiggle": "sin(3 × temp) (chosen afterwards)"}
for j, shape in enumerate(gof.SHAPES):
s = summ[summ["shape"] == shape]
ax = axes[0, j]
for col, c, lab in [("reset", PURPLE, "RESET"), ("quad", TEAL, "quadratic F"), ("rainbow", AMBER, "rainbow")]:
ax.plot(s["eta"], s[col], color=c, marker="o", ms=3, label=lab)
ax.axhline(0.05, color=MUTED, ls=":", lw=0.8)
ax.set_title(titles[shape])
ax.set_ylim(-0.02, 1.02)
ax = axes[1, j]
ax.plot(s["eta"], s["ratio"], color=BLUE, marker="o", ms=3)
ax.axhline(1, color=MUTED, ls="--", lw=0.8)
ax.set_xlabel("nonlinear share of signal, η")
axes[0, 0].set_ylabel("rejects linearity")
axes[1, 0].set_ylabel("forest MSE ÷ OLS MSE")
axes[0, 0].legend(loc="center right")
fig.tight_layout()
plt.show()
The widget shows one replicate’s residual plot next to the power curves.
- With the product term, drag \(\eta\) to 0.10. RESET’s tile reads 76% and the ratio tile 1.19: the test sees a nonlinearity that the forest cannot yet use.
- Switch to the sine term and drag \(\eta\) to 0.50. The ratio tile reads 0.68, and the three rejection tiles stay under 20%. The residuals against fitted values show no pattern.
- Drag \(\eta\) to 0.80 on the sine term. The OLS fit has lost most of its signal, the forest’s error is half of OLS’s, and the residual plot still looks like noise.
7.5 What this shows
- Whether a goodness-of-fit test notices a nonlinearity depends on whether it looks in that direction. That is Bickel, Ritov and Stoker’s result, not a defect of one test.
- An analyst does not know in advance which direction nature chose. A held-out comparison does not need to know.
- The sine term would show at once in a plot of residuals against
temp, one of seven such plots. A residual check works when the analyst knows where to look. With 450 variables, as in the ozone project, there are too many places to look.
8 Predictive accuracy as the test
Breiman’s remedy (p. 204) is to compare nature’s box with the model’s box directly: put the same \(x\) into both and see how close the outputs are, using a test set or cross-validation so the comparison is not flattered by fitting. He cites Stone (1974) and Mosteller and Tukey (1977) as early advocates, and notes how rarely JASA papers reported predictive accuracy.
Two limits matter for the rest of the series.
- Accuracy is measured on the population the test set came from. Hoadley’s comment (Part 5) points out that a test sample is a sample of the present, not the future (pp. 221–222).
- Accuracy says nothing about interventions. Cox’s comment asks what would happen if medical X-ray use fell by 10% (p. 216), and no held-out set contains the answer. Breiman agreed (p. 227). Part 5 takes this up.
9 Constraints
- The ozone stand-in is one year at one station. The test months differ in season from the training months, and the alert comparison rests on 7 alert days.
- The linear route is a re-enactment. Breiman describes the 1970s procedure in one paragraph; the selection criterion, the candidate sets and the stopping rule here are choices, made before seeing the test months.
- The dev months are 38 day pairs from one season. Settings chosen on them are a guess about autumn made from summer. The grids and the split were set once and not revised after the test scores were seen. One model was dropped after them: an equal average of the tuned forest, boosting and Chronos-2 with meteorology, which scores 4.92 on the test months, within the forest’s and boosting’s range.
- Chronos-2’s forecasts are read from a file.
src/chronos_forecast.pywrites them with its own pinned venv (src/requirements-chronos.txt) and a pinned model revision, because the series’ shared venv has no torch; the render does not run the model. Its pretraining corpus was not checked for this 1976 record. - LightGBM itself was not run. Its macOS wheel needs a system OpenMP library; scikit-learn’s histogram boosting uses the same binning idea.
- The goodness-of-fit design is ours. The paper does not describe its simulation. The product term was fixed before any result; the sine term was chosen after the first result was seen, which the text labels.
- The forest and OLS are compared on one random split per replicate. The median ratio over 200 replicates is stable; a single replicate’s ratio is not.
10 Reproduction status
| Experiment | Paper’s data | Data used here | Paper’s result | Our result | Notes |
|---|---|---|---|---|---|
| Ozone forecast (§3.1) | 7 years, 450+ variables, LA Basin | 1976 LA record (mlbench::Ozone), 30 candidate predictors |
selected regression; false-alarm rate too high | forest RMSE 4.94–5.00 vs selected regression 5.14; all methods issued false alarms; lasso, boosting and Chronos-2 within noise of the forest | stand-in; original data not public |
| Chlorine (§3.2) | 30,000 mass spectra | none | tree on 1,500 questions, 95% accuracy | not attempted | data not public; not simulated |
| Goodness of fit (§5.2) | 7-D simulation, setup not given | 7 real meteorological columns, simulated response | tests did not reject until nonlinearity extreme | product term: tests reject before the forest gains; sine term: tests miss what the forest gains | outcome depends on the shape of the nonlinearity |
Model. Inside. Or. Outside. Tests. See. Only. Where. They. Look.
11 References
- Ansari, A. F., Shchur, O., Küken, J., et al. (2025). Chronos-2: from univariate to universal forecasting. arXiv:2510.15821. Run here as
amazon/chronos-2, revision29ec376. - Bickel, P. J., Ritov, Y. & Stoker, T. M. (2006). Tailor-made tests for goodness of fit to semiparametric hypotheses. Annals of Statistics 34(2), 721–741. Cited by Breiman as a 2001 manuscript.
- Breiman, L. (2001). Statistical modeling: the two cultures (with comments and a rejoinder). Statistical Science 16(3), 199–231.
- Breiman, L. (2001). Random forests. Machine Learning 45, 5–32.
- Breiman, L. & Friedman, J. H. (1985). Estimating optimal transformations for multiple regression and correlation. Journal of the American Statistical Association 80(391), 580–598. The source of the 1976 ozone data.
- Breiman, L., Friedman, J. H., Olshen, R. A. & Stone, C. J. (1984). Classification and Regression Trees. Wadsworth.
- Faraway, J. faraway: Functions and Datasets for Books by Julian Faraway, R package 1.0.9 (
ozone). - Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of Statistics 29(5), 1189–1232.
- Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. & Liu, T.-Y. (2017). LightGBM: a highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems 30.
- Leisch, F. & Dimitriadou, E. mlbench: Machine Learning Benchmark Problems, R package 2.1-11 (
Ozone). - Mosteller, F. & Tukey, J. W. (1977). Data Analysis and Regression. Addison-Wesley.
- Ramsey, J. B. (1969). Tests for specification errors in classical linear least-squares regression analysis. Journal of the Royal Statistical Society B 31(2), 350–371.
- Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society B 36(2), 111–147.
- Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B 58(1), 267–288.
- Utts, J. M. (1982). The rainbow test for lack of fit in regression. Communications in Statistics: Theory and Methods 11(24), 2801–2815.
- The Two Cultures, Part 2: Rashomon — many equally good models.
- The Two Cultures, Part 5: The pushback — the discussants, causality, and the full reproduction table.