The Two Cultures, Part 5: The Pushback, and What Prediction Cannot Tell You

Cox, Efron, Hoadley and Parzen in their strongest form, Breiman’s reply, the question accuracy cannot answer, and the reproduction table for the series

The four published comments on Breiman’s Two Cultures and his rejoinder, set against what Parts 1–4 found. Then the gap Breiman conceded to Cox: a model that predicts well can give the wrong answer to an intervention question. A confounding widget, the Lalonde job-training data with its experimental benchmark, later perspectives, and the series’ full reproduction table.
Statistics
Machine Learning
Causality
Reading
Author

Ravi Kalia

Published

November 18, 2026

The Two Cultures, Part 5: The Pushback, and What Prediction Cannot Tell You

Breiman’s paper was published with four comments and a reply, and the comment he answered with “I agree.” raises the question predictive accuracy cannot settle: what happens if we change something.

Every predictive model invites the same misreading. A churn model ranks discount codes among its top features, a readmission model gives a drug a large importance score, and the next meeting asks what happens if the company sends more codes or the hospital prescribes more of the drug. The model was never asked that question. It learned which customers and which patients tend to have which outcomes, and the people who get discounts or drugs are not a random sample. Cox’s comment makes this point in 2001, and it is the point Breiman conceded.

Parts 1–4 re-ran the paper’s evidence. This part sets the discussants against those results and then takes up Cox’s question with a widget and with data whose causal answer is known from an experiment.

This part covers two ideas:

1 The discussion

Statistical Science printed the paper with four invited comments and Breiman’s reply (pp. 216–231). Breiman sorted them himself in the rejoinder’s first paragraph (p. 226).

Table 1: The four discussants (pp. 216, 218, 220, 224, 226).
Discussant In 2001 Breiman’s summary of the stance
D. R. Cox Nuffield College, Oxford downright disagreement
Brad Efron Stanford serious reservations
Bruce Hoadley Fair, Isaac (credit scoring) more or less in agreement
Emanuel Parzen Texas A&M more or less in agreement

Each subsection states the comment in its strongest form, then what the re-runs in Parts 1–4 do to it.

1.1 Cox: start from the question, and know what you are predicting

Cox grants more than his reputation for the opposing view suggests. He calls work without an explicit probability model a legitimate part of statistics, and he accepts that a purely empirical predictor sometimes wins; short-term economic forecasting and real-time flood forecasting are his examples (p. 216). His objections are four.

  1. The picture of mainstream practice is a caricature. His word; he adds that good caricatures expose real weaknesses (p. 216). Journal papers illustrate technique, while live analyses that reach conclusions are reported in subject-matter journals and reports, so the journals Breiman read are the wrong sample (p. 217).
  2. Data have a meaning before they have a model. What a variable measures, how the cases were ascertained, measurement error and why values are missing all bear on the answer, and he doubts these can be handled without some probabilistic model (p. 216).
  3. Much prediction worth doing is extrapolation. His examples: the course of the vCJD epidemic in the UK, and the effect on US cancer incidence of cutting medical X-ray use by 10% (p. 216). Both ask what happens under conditions the data never showed, and the second asks what an intervention would do. A predictor validated on a random test set from the same population says nothing about either.
  4. A model should carry subject-matter meaning. He lists Cox and Wermuth’s requirements: connect to background knowledge, point toward a generating process, have parameters with a clear interpretation, specify the noise well enough to state precision, and fit adequately. Fit, which predictive accuracy measures, is last and is not the basis of choice (p. 217). The aim is descriptions that are potentially causal (p. 218).

Two smaller points land on specific results. Cox warns that complete separation, on which Breiman’s SVM argument rests, is unstable, and that the bootstrap and cross-validation may give “a quite misleading illusion of stability” (p. 218). He also quotes an aphorism he attributes, tentatively, to Kelvin: a rough answer to the right question beats an exact answer to the wrong one (p. 218).

What the re-runs say:

  • Point 2 holds. The Bupa “class” in Part 4 is a train/test selector created by the people who collected the data; a forest predicted it with 25% error and no amount of accuracy could reveal what it was. The hepatitis file’s binary codes are documented only by the data themselves.
  • The separation warning holds. In Part 4’s forward-selection bootstrap, sex is chosen in 81% of resamples because no woman in the data died.
  • Point 3 is the one Breiman conceded, and the subject of Section 2.

1.2 Efron: where did the bias go?

Efron reads the paper as a call for theory, not against it.

  1. Twentieth-century statistics was “100 years of unbiasedness” (p. 218). Unbiased estimation needs a favourable ratio of sample size to parameters, and good design exists to secure it.
  2. Black boxes with many knobs must be biased estimators. Regularisation, tree pruning and SVM margins all trade variance for bias somewhere, and there is little theory for heavily biased estimators (p. 219).
  3. Two rules of thumb about comparisons (p. 219). New methods always look better than old ones, because honest simulation comparisons are hard and the old method rarely gets the same tuning effort. Complicated methods are harder to criticise than simple ones.
  4. Prediction is rarely enough. The post office needs only addresses read correctly; Gregory studied the hepatitis data partly to understand the disease, and most surveys aim at causal factors (p. 219).
  5. The hepatitis variables. From his fuller analysis with Gong (1983): variables 12 or 17 were selected in 60% of bootstrap samples, against 59% for variable 19 (p. 219). He asks in what sense a variable is important.

What the re-runs say:

  • Rule 1. Every single tree in Part 3 was pruned by cross-validation inside its training data, and the forest was not tuned. The forest still won on all ten sets.
  • Point 2 has an answer for one model family. In Part 3’s double-descent curve the bias hides in which interpolating fit least squares picks: the one of smallest norm.
  • Point 5. Part 4 found importance depends on the model, the measure and the correlation among inputs; Breiman’s rejoinder says the same.

1.3 Hoadley: an algorithmic culture inside the data-modeling toolbox

Hoadley wrote from Fair, Isaac, where credit scoring had been algorithmic since the 1960s (p. 220). He agrees with Breiman’s direction and disputes the details.

  1. “Primitive” is wrong. Fair, Isaac’s segmented scorecards were built by engineers: thousands of derived characteristics, a shallow tree of segments, and a binned additive score in each leaf, about 10,000 features from 24 raw inputs, judged only on test samples (pp. 220–221).
  2. Accuracy and interpretability can coexist. The Fat Scorecard, logistic regression on binned dummies, slightly beat a 2,000-weight neural network and held up on three later test samples (p. 222).
  3. The test sample is not the future. It samples the current population; scorecards are rebuilt every 18–24 months (pp. 221–222). Tuning that peeks at the test sample biases the estimate (p. 223).
  4. Error rate is not the objective. A fraud model is judged by dollars saved at a fixed review rate (p. 223).
  5. Importance is not a property of the variable. If an equally accurate model exists without \(x_1\), then \(x_1\) is unimportant to that model (p. 223).

What the re-runs say:

  • Point 2 reproduces. On Taiwan credit-card data in Part 3, a 78-coefficient scorecard matched a 500-tree forest in AUC on three splits.
  • Point 5 reproduces. The 175 ozone regressions within 5% of the best in Part 2 use every one of the 30 candidate variables between them.

1.4 Parzen: management and science

  1. Two goals, renamed. Prediction is “management” (short-run, profit, decisions); information is “science” (long-run truth and understanding) (p. 224).
  2. Wrong model, right arithmetic. Fitting a model that misdescribes nature is the right answer to the wrong question, the error of the third kind (p. 224).
  3. Two checks to make routine. Test whether inputs are dependent, since correlated inputs act as proxies and make equally good models disagree; and test for conditional clustering, where one class occupies two separate regions of input space (pp. 224–225).
  4. Many cultures, not two. He offers his quantile-based approach and “statistical methods mining” as a synthesis (p. 225).

What the re-runs say: point 3 names both of Part 4’s findings in advance. The first check is the ascites–albumin surrogate pair; the second is Breiman’s Bupa clusters.

1.5 Breiman’s rejoinder

  1. To Cox. Data models are useful when the mechanism is well understood, or when data are too few and a policy decision is due anyway. On extrapolation and intervention he quotes Cox’s v-CJD passage and replies “I agree.” (p. 227). On the caricature he says the dispute is over the word “mainstream”: counted by SAS licences and canned-package users, his version wins by at least 100 to 1 (p. 228).
  2. To Efron. Science moves from simple models to complex ones; the job is to extract information from the most accurate model available (p. 229). A model barely better than the baseline cannot be trusted to identify causal factors (p. 229). Random forests have one tuning parameter (p. 230).
  3. On importance. Importance has always been defined operationally, and his definition is predictive. It lacks a theoretical definition and depends on the dependence among inputs: “The problem begs for research.” (p. 230).
  4. To Hoadley. Models must be rebuilt with data from the new context, and Fair, Isaac is an exception on interpretability (pp. 230–231).
  5. To Parzen. Two cultures, because the line is clear: is the analyst modeling the inside of the box or not (p. 231).

2 Prediction and intervention

2.1 Three tasks

Hernán, Hsu and Healy (2019) sort data-science work into three tasks:

  1. Description: summarise the data (what fraction of patients were readmitted).
  2. Prediction: map inputs to outputs (which patients will be readmitted).
  3. Counterfactual prediction: what the outcome would be under a different action (how many would be readmitted if all got the drug).

Breiman’s validation, held-out predictive accuracy, checks task 2. Cox’s X-ray question is task 3. Pearl’s “ladder of causation” (Pearl and Mackenzie 2018) draws the same line between seeing and doing; an earlier post works through it, and another separates attributing a model’s output from explaining the world.

The two tasks agree only when the people who received the action are comparable to those who did not. When something drives both who gets the action and the outcome, a confounder, the association in the data mixes the action’s effect with the confounder’s.

2.2 A confounder, in miniature

The widget is synthetic. It stands in for a clinical setting in which sicker patients get more of a treatment.

  • Severity \(S \sim N(0, 1)\).
  • Dose \(T = aS + \sqrt{1 - a^2}\,U_T\): sicker patients get more.
  • Outcome \(Y = \tau T + bS + 0.6\,U_Y\): severity hurts the outcome (\(b < 0\)) and the dose helps (\(\tau > 0\)).

A model that predicts \(Y\) from \(T\) learns the slope \(\tau + ab\). Raising everyone’s dose by one unit changes \(Y\) by \(\tau\), whatever \(a\) and \(b\) are.

  1. At the starting setting (\(a = 0.9\), \(b = -1.6\), \(\tau = 0.5\)) the fit of outcome on dose has a slope of −0.96 and an \(R^2\) of 0.51: a respectable predictive model that says more dose means a worse outcome. The causal effect is +0.50. Adjusting for severity gives a slope of 0.49.
  2. Drag “Severity → dose” to 0. The two lines nearly coincide: with no confounding the predictive slope estimates the causal one, and the \(R^2\) drops, because dose no longer carries information about severity.
  3. Press New patients a few times at the starting setting. The predictive slope stays between −1.0 and −0.8 over the first five draws: the error is not noise, and more data will not remove it.

The predictive model is not wrong. It answers the question it was trained on: among patients like these, who will do badly. The doctor’s question is different.

2.3 Lalonde: a predictive model against an experiment

2.3.1 Data

  • Source. The National Supported Work (NSW) demonstration, a 1970s US job-training program that assigned applicants to training or control at random. LaLonde (1986) compared the experimental estimate of its effect on earnings with non-experimental estimates built from survey comparison groups. The version here is Dehejia and Wahba’s (1999): 185 trained men and 260 controls from the experiment, and 2,490 men from the Panel Study of Income Dynamics (PSID) as a comparison group. Files from Dehejia’s data page; provenance in the series’ data/README.md.
  • Variables. 1978 earnings (the outcome), training, age, education, race, marital status, whether a high-school degree, and 1974 and 1975 earnings.
  • Why this data. The experiment gives the causal answer, so every observational method can be graded against it.
  • Cost of a wrong answer. A policy funded or cut on the wrong sign.

The observational question replaces the experiment’s controls with the PSID men, who are older, better paid and more likely to be married than the program’s applicants.

Code
L = cache.obj("lalonde_v2", causal.lalonde)


def usd(v):
    """Dollars for Markdown: escaped sign (so MathJax leaves it alone), true minus."""
    return ("−" if v < 0 else "") + f"\\${abs(v):,.0f}"


def rng(key):
    v = L[key]
    return f"{usd(min(v))} to {usd(max(v))}"


rows = [
    ("Experiment: treated minus experimental controls", f"{usd(L['experimental'])} (SE {usd(L['experimental_se'])})", "the benchmark"),
    ("Difference in means, treated vs PSID", usd(L["naive"]), "the raw association"),
    ("Linear regression on training and covariates", usd(L["ols"]), "adjusts linearly"),
    ("Random forest, read as partial dependence", rng("s_learner"), "five seeds"),
    ("Double ML, partially linear model", rng("dml_plr"), f"five seeds; SE about {usd(np.mean(L['dml_plr_se']))}"),
    ("Doubly robust effect on the treated", rng("aipw_att"), f"five seeds; SE about {usd(np.mean(L['aipw_att_se']))}"),
]
md(pd.DataFrame(rows, columns=["Estimate", "Effect of training on 1978 earnings", "Notes"]))
Table 2: Estimates of the effect of NSW training on 1978 earnings. All rows except the first use the 185 trained men and the 2,490 PSID men; the forests use 500 trees (300 for the nuisance models), minimum leaf size 5, five-fold cross-fitting.
Estimate Effect of training on 1978 earnings Notes
Experiment: treated minus experimental controls $1,794 (SE $671) the benchmark
Difference in means, treated vs PSID −$15,205 the raw association
Linear regression on training and covariates $752 adjusts linearly
Random forest, read as partial dependence $1,340 to $1,416 five seeds
Double ML, partially linear model −$1,216 to −$921 five seeds; SE about $1,019
Doubly robust effect on the treated $1,317 to $2,006 five seeds; SE about $778
Code
est = [("Difference in means", [L["naive"]]), ("Linear regression", [L["ols"]]),
       ("Forest, partial dependence", L["s_learner"]), ("Double ML, partially linear", L["dml_plr"]),
       ("Doubly robust, effect on treated", L["aipw_att"])]
fig, axes = plt.subplots(1, 2, figsize=(7.5, 2.6), gridspec_kw={"width_ratios": [1, 3]}, sharey=True)
for ax in axes:
    ax.axvspan(L["experimental"] - 2 * L["experimental_se"], L["experimental"] + 2 * L["experimental_se"],
               color=RULE, alpha=0.6, lw=0)
    ax.axvline(L["experimental"], color=INK, ls="--", lw=1)
    for k, (name, vals) in enumerate(est):
        ax.scatter(vals, [len(est) - 1 - k] * len(vals), color=[AMBER, BLUE, PURPLE, PINK, TEAL][k], s=22, zorder=3)
axes[0].set_xlim(-16500, -13500)
axes[0].set_xticks([-15000], ["−15,000"])
axes[1].set_xlim(-2500, 4000)
axes[1].set_xticks([-2000, 0, 2000, 4000], ["−2,000", "0", "2,000", "4,000"])
axes[0].set_yticks(range(len(est))[::-1], [n for n, _ in est])
axes[0].spines["right"].set_visible(False)
axes[1].spines["left"].set_visible(False)
axes[1].tick_params(left=False)
fig.supxlabel("effect on 1978 earnings (dollars); left panel: the raw difference, on its own scale", fontsize=8.5)
fig.tight_layout()
plt.show()
Dot plot. The experimental benchmark is about 1,800 dollars. The raw difference in means sits far to the left at about minus 15,000. Linear regression, forest partial dependence and the doubly robust estimate cluster between about 750 and 2,000 dollars, inside or near the benchmark band; the partially linear double-ML estimates sit near minus 1,000.
Figure 1: Estimates of the training effect on 1978 earnings from the observational comparison (trained men against PSID men), one point per seed for the forest-based methods, against the experimental estimate (dashed line, shaded ±2 standard errors).
  • The association is not the effect. Trained men earned \\\\$15,205 less in 1978 than the PSID men. The program selected men with low earnings; that is the confounding.
  • The forest predicts earnings reasonably well: five-fold cross-validated \(R^2\), on shuffled folds, of 0.57.
  • Its partial dependence lands near the experiment. Setting training to 1 and to 0 for every trained man and averaging the difference gives \\\\$1,340 to \\\\$1,416. Because the forest sees the earnings history and demographics that drove selection, its comparison is between similar men.
  • The doubly robust estimate of the effect on the treated, \\\\$1,317 to \\\\$2,006 with a standard error near \\\\$778, is consistent with the experiment.
  • The partially linear double-ML estimate is negative, −\\\\$1,216 to −\\\\$921. It estimates a different quantity: an average of individual effects weighted by how uncertain each man’s treatment status is given his covariates, which here means the minority of PSID men who resemble the trainees. 69% of the PSID men have a fitted probability of training below 1%, so the two targets weight very different men, and with effects that vary between men they need not agree.

The lesson differs from the widget’s. Here the confounders were measured, and a flexible predictive model with all of them recovered roughly the right answer. But the estimates span a range of thousands of dollars, including both signs, depending on the target and the method. Without the experiment there would be no way to tell from the data which one to believe. Dehejia and Wahba (1999) made the same point with propensity scores: the estimates depend on whether the comparison group overlaps with the treated.

2.4 A proxy outcome

Obermeyer, Powers, Vogeli and Mullainathan (2019) studied a commercial algorithm used to refer US patients to care-management programs. It predicted future health-care costs well, and costs were used as a stand-in for health needs. Because less money was spent on Black patients than on white patients with the same needs, Black patients assigned the same risk score were sicker. The prediction was accurate; the question it answered, who will cost the most, was not the question the referral program asked, who needs help most. It is Parzen’s error of the third kind and Cox’s point about the meaning of the data, at national scale.

3 Since 2001

The paper is still argued over. A few later lines, briefly:

  • To explain or to predict (Shmueli 2010): explanatory and predictive modeling differ in goals, data, methods and validation, and conflating them misleads in both directions.
  • Prediction, estimation and attribution (Efron 2020): prediction algorithms succeed by not committing to a model, which is why their output is hard to attribute to individual inputs; the scientific uses of statistics need the other two.
  • The Rashomon set (Semenova, Rudin and Parr 2022; Fisher, Rudin and Dominici 2019): see Part 2.
  • Underspecification (D’Amour et al. 2022): equally accurate models diverge under distribution shift, Hoadley’s “the test sample is not the future” in deep learning.
  • Calibrated and conformal uncertainty. Modern neural networks can be accurate and badly calibrated (Guo et al. 2017). Conformal prediction wraps any black box in prediction sets with guaranteed coverage under exchangeability (Angelopoulos and Bates 2023), one answer to Cox’s demand for a statement of precision.
  • Twenty years on. Observational Studies 7(1) (2021) reprinted the paper with new commentary from more than two dozen authors or teams (introduction by Mitra). Many titles alone show the drift: “A Melting Pot” (Tibshirani and Hastie), “From Two Cultures to One” (Neufeld and Witten), “You Don’t Have to Choose Sides” (Miller, Foti and Fox), “Revisiting Rashomon” (D’Amour), and causal readings from Pearl and from Ogburn and Shpitser.

4 An analogy from the schools

This section is an analogy, not history: neither Ockham nor Aquinas had anything to say about regression.

  • Two views of universals. Ockham was a nominalist: only individual things exist, and general terms are names or concepts that sort them. Aquinas held a moderate realism: the common natures of things are real, existing in the individuals, and the intellect abstracts them. The data-modeling culture resembles the realist, treating a parameter as a feature of nature to be estimated. The algorithmic culture resembles the nominalist, treating a model as a useful way of sorting cases with no claim about what is inside the box. Breiman named his second lesson after Ockham for the razor; the nominalism fits him better.
  • Two kinds of knowing. Aquinas distinguishes demonstration propter quid, which reasons from a cause to its effect, from demonstration quia, which reasons from an effect, better known to us, back to the fact of its cause (Summa Theologiae I, q. 2, a. 2, following Aristotle’s distinction between knowing that and knowing why in Posterior Analytics I.13). A predictive model gives knowledge quia, that the outcome follows these inputs. The intervention question asks for knowledge propter quid, what the dose does. The analogy’s point is the old one: the first kind does not license the second.

5 What held up

  • Breiman was right that held-out accuracy should be reported. It is now routine, and his Table 2 reproduces closely on the same data 25 years later (Part 3).
  • He was right that equally good models tell different stories (Part 2), and that selecting one of them and reading its coefficients is unsafe.
  • He was partly right that an accurate black box gives better information. The forest predicted hepatitis deaths a little better than logistic regression (14.9% against 16.0% error), and best-subset logistic regression picked 205 different variable sets in 500 resamples (Part 2). But the forest’s own ranking was not steady either: over 30 bootstrap resamples albumin’s rank among the 19 covariates ran from 1 to 16 and ascites’ from 1 to 17. Importance under correlation is fragile, as he admitted (Part 4).
  • The critics were right that a goodness-of-fit test is not always blind. In Part 1 a directed test detected a nonlinearity well before a forest could use it; it failed when the nonlinearity lay where the test did not look.
  • Hoadley was right that accuracy does not force opacity, at least on credit data (Part 3).
  • Cox was right about the question. Accuracy on a random test set validates task 2. Policy, medicine and science usually ask task 3, and Breiman agreed.

6 Reproduction status

Code
status_rows = [
    ("1", "Ozone forecast (§3.1)", "7 years, 450+ variables", "1976 LA record", "selected regression failed on false alarms",
     f"forest RMSE {S1['rf_rmse_lo']:.2f}–{S1['rf_rmse_hi']:.2f} vs selected regression {S1['quad_rmse']:.2f}; "
     + ("false alarms for all" if S1["min_false_alarms"] > 0 else "some methods without false alarms"), "stand-in"),
    ("1", "Chlorine (§3.2)", "30,000 spectra", "—", "tree, 95% accuracy", "—", "not attempted: data not public"),
    ("1", "Goodness of fit (§5.2)", "7-D simulation", "7 real meteorological columns", "tests miss nonlinearity until extreme",
     f"product term: RESET rejects {S1['reset_regime_01']:.0%} at η = 0.1 while the forest is "
     f"{'worse' if S1['ratio_regime_01'] > 1 else 'better'}; sine term: tests miss, forest gains {S1['gain_wiggle_05']:.0%}",
     "depends on the shape"),
    ("2", "Subsets (§8)", "30 variables", "30 ozone predictors", "many within 1% of best", f"{S2['within_1']} within 1%, {S2['within_5']} within 5%", "partly"),
    ("2", "Perturbed trees (§8)", "—", "hepatitis", "different tree, same error",
     f"no split {S2['tree_stumps']}, root ascites {S2['tree_root_ascites']}, albumin {S2['tree_root_albumin']} of {S2['tree_reps']}", "reproduced in kind"),
    ("2", "Network restarts (§8)", "3-D data", "3 ozone predictors", "32 minima", f"{S2['net_minima']} distinct functions", "reproduced in kind"),
    ("2", "Bagged logistic (§8)", "toxicity data", "hepatitis", "bagged more accurate",
     f"{S2['bag_single']:.1%} single vs {S2['bag_bagged']:.1%} bagged", "stand-in"),
    ("3", "Table 2 (§9.2)", "10 benchmark sets", "same 10 sets", "forest beats tree on all 10",
     f"forest beats tree on {S3['forest_wins']} of {S3['n_sets']}; {S3['within_1_5']} of {S3['n_cells']} errors within 1.5 points", "reproduced"),
    ("3", "Statlog ranks (§9.3)", "18 classifiers", "—", "forest ranks 1, 1, 1, 1", "—", "not attempted"),
    ("3", "Fat Scorecard (Hoadley)", "proprietary", "Taiwan credit default", "scorecard ≈ neural net",
     f"scorecard AUC within {S3['credit_gap']:.3f} of forest", "stand-in"),
    ("3", "Shape forest (§10.2)", "NIST", "—", "0.7%", "—", "not attempted"),
    ("3", "SVM on ZIP code (§10.3)", "USPS", "USPS", "4.1% at degree 4", f"{S3['svm4_error']:.1%} at degree 4", "partly"),
    ("4", "Hepatitis errors (§11.1)", "UCI", "UCI", "LR 17.4%, RF 12.3%", f"LR {S4['lr_all']:.1%}, RF {S4['rf_all']:.1%}", "partly: smaller gap"),
    ("4", "Hepatitis importance (§11.1)", "UCI", "UCI", "RF singles out 12 and 17",
     f"17 ranks {S4['rf_rank_albumin']}, 12 ranks {S4['rf_rank_ascites']}; resamples move both", "partly"),
    ("4", "Bupa (§11.2)", "UCI", "UCI", "28%; tests 3 and 5, then 4; two clusters",
     f"{S4['bupa_lo']:.0%}–{S4['bupa_hi']:.0%}; {'same tiers' if S4['bupa_tiers'] else 'different tiers'} of importance; two clusters", "reproduced; label is a selector"),
    ("4", "Lymphoma (§11.3)", "81 × 4,682", "Breiman's file", "low error", f"OOB {S4['lymphoma_oob']:.1%}", "reproduced"),
    ("5", "Lalonde (extension)", "—", "NSW + PSID", "—",
     f"partial dependence {rng('s_learner')}; experiment {usd(L['experimental'])}", "extension"),
]
md(pd.DataFrame(status_rows, columns=["Part", "Experiment", "Paper's data", "Data used", "Paper's result", "Our result", "Status"]))
Table 3: The series’ reproduction table: every experiment Breiman reports, what data were used here, and how close the re-run came. Numbers come from the status values Parts 1–4 write when they execute (cache/status.json in each post).
Part Experiment Paper’s data Data used Paper’s result Our result Status
1 Ozone forecast (§3.1) 7 years, 450+ variables 1976 LA record selected regression failed on false alarms forest RMSE 4.94–5.00 vs selected regression 5.14; false alarms for all stand-in
1 Chlorine (§3.2) 30,000 spectra — tree, 95% accuracy — not attempted: data not public
1 Goodness of fit (§5.2) 7-D simulation 7 real meteorological columns tests miss nonlinearity until extreme product term: RESET rejects 76% at η = 0.1 while the forest is worse; sine term: tests miss, forest gains 32% depends on the shape
2 Subsets (§8) 30 variables 30 ozone predictors many within 1% of best 4 within 1%, 175 within 5% partly
2 Perturbed trees (§8) — hepatitis different tree, same error no split 45, root ascites 39, albumin 14 of 100 reproduced in kind
2 Network restarts (§8) 3-D data 3 ozone predictors 32 minima 28 distinct functions reproduced in kind
2 Bagged logistic (§8) toxicity data hepatitis bagged more accurate 19.1% single vs 16.6% bagged stand-in
3 Table 2 (§9.2) 10 benchmark sets same 10 sets forest beats tree on all 10 forest beats tree on 10 of 10; 19 of 20 errors within 1.5 points reproduced
3 Statlog ranks (§9.3) 18 classifiers — forest ranks 1, 1, 1, 1 — not attempted
3 Fat Scorecard (Hoadley) proprietary Taiwan credit default scorecard ≈ neural net scorecard AUC within 0.003 of forest stand-in
3 Shape forest (§10.2) NIST — 0.7% — not attempted
3 SVM on ZIP code (§10.3) USPS USPS 4.1% at degree 4 5.2% at degree 4 partly
4 Hepatitis errors (§11.1) UCI UCI LR 17.4%, RF 12.3% LR 16.0%, RF 14.9% partly: smaller gap
4 Hepatitis importance (§11.1) UCI UCI RF singles out 12 and 17 17 ranks 2, 12 ranks 4; resamples move both partly
4 Bupa (§11.2) UCI UCI 28%; tests 3 and 5, then 4; two clusters 25%–27%; same tiers of importance; two clusters reproduced; label is a selector
4 Lymphoma (§11.3) 81 × 4,682 Breiman’s file low error OOB 1.2% reproduced
5 Lalonde (extension) — NSW + PSID — partial dependence $1,340 to $1,416; experiment $1,794 extension

Every part has its own table with the details; the code for every number is in its folded cells and in posts/two-cultures-breiman-2001/src/.

7 Session

pkgs = ["numpy", "pandas", "scipy", "scikit-learn", "statsmodels", "matplotlib"]
print(f"Python {platform.python_version()} on {platform.system()} {platform.machine()}")
print(", ".join(f"{p} {version(p)}" for p in pkgs))
Python 3.12.13 on Darwin arm64
numpy 2.5.5, pandas 3.1.1, scipy 1.18.1, scikit-learn 1.9.1, statsmodels 0.15.0, matplotlib 3.11.2

The full pinned environment for the series is each post’s requirements.txt.

Forests. Predict. Experiments. Decide. Cox. Asked. The. Question. Breiman. Agreed.

8 References