References

Books, papers, library documentation and datasets used in the workshop, grouped by day. Every entry was opened and checked on the date shown.

Each entry links to a page we opened, and says what that page supports in the workshop. The tag Verified (date) means we read the page on that date. Not read means we could not open the text itself, and we say what we relied on instead. How we checked is explained at the end of the page.

Day 1 · Who are our customers?

Customer-base analysis and buy-till-you-die models (Modules 1–3)

  • Schmittlein, D. C., Morrison, D. G., & Colombo, R. (1987). Counting your customers: Who-are they and what will they do next? Management Science, 33(1), 1–24. https://doi.org/10.1287/mnsc.33.1.1
    Verified (2026-10-09). The Pareto/NBD model: from the number and timing of a customer’s past purchases it gives the probability the customer is still active and forecasts future purchases in a non-contractual setting. (Module 2.) The title is printed with the hyphen, as in the Crossref record.

  • Fader, P. S., Hardie, B. G. S., & Lee, K. L. (2005). “Counting your customers” the easy way: An alternative to the Pareto/NBD model. Marketing Science, 24(2), 275–284. https://doi.org/10.1287/mksc.1040.0098
    Verified (2026-10-09). The BG/NBD model forecasts almost as well as the Pareto/NBD and is far easier to estimate. (Module 2.)

  • Fader, P. S., Hardie, B. G. S., & Lee, K. L. (2005). Implementing the BG/NBD model for customer base analysis in Excel (note, June 2005). https://www.brucehardie.com/notes/004/
    Verified (2026-10-09; the PDF read again on 2026-10-10 for Module 13). Maximum-likelihood BG/NBD estimates for the 2,357 CDNOW customers over a 39-week calibration period (r = 0.243, α = 4.414, a = 0.793, b = 2.426). Module 2 uses these as a checkpoint: a correct RFM table reproduces them. Module 13 writes the BG/NBD likelihood in the note’s form, and its maximum-likelihood helper is tested against these estimates.

  • Fader, P. S., Hardie, B. G. S., & Shang, J. (2010). Customer-base analysis in a discrete-time noncontractual setting. Marketing Science, 29(6), 1086–1108. https://doi.org/10.1287/mksc.1100.0580
    Verified (2026-10-09). The BG/BB model, the discrete-time counterpart of the Pareto/NBD for transactions that can only happen at fixed occasions. (Module 2 briefing.)

  • Wübben, M., & v. Wangenheim, F. (2008). Instant customer base analysis: Managerial heuristics often “get it right”. Journal of Marketing, 72(3), 82–93. https://doi.org/10.1509/jmkg.72.3.082
    Verified (2026-10-09). Simple recency heuristics can match Pareto/NBD and BG/NBD at telling active from inactive customers; the models’ advantage is mainly in forecasting total future purchases. This is why Modules 1 and 3 measure the heuristic and the machine-learning alternative instead of assuming the model wins.

  • Bachmann, P., Meierer, M., & Näf, J. (2021). The role of time-varying contextual factors in latent attrition models for customer base analysis. Marketing Science, 40(4), 783–809. https://doi.org/10.1287/mksc.2020.1254
    Verified (2026-10-09). A Pareto/NBD extension with covariates, the method paper behind CLVTools’ covariate models. (Module 3.) CLVTools itself is cited as software under Library documentation.

Day 2 · What is a customer worth?

Monetary value, CLV and retention (Modules 4–5)

  • Fader, P. S., Hardie, B. G. S., & Lee, K. L. (2005). RFM and CLV: Using iso-value curves for customer base analysis. Journal of Marketing Research, 42(4), 415–430. https://doi.org/10.1509/jmkr.2005.42.4.415
    Verified (2026-10-09). CLV can be computed from recency, frequency and monetary value alone by combining a Pareto/NBD purchase model with a gamma-gamma spend model; applied to CDNOW. (Module 4.)

  • Fader, P. S., & Hardie, B. G. S. (2013). The Gamma-Gamma model of monetary value (note, February 2013). https://www.brucehardie.com/notes/025/
    Verified (2026-10-09; the PDF read again on 2026-10-10 for Module 13). The Gamma-Gamma spend model assumes spend per transaction is independent of the purchase process (“the distribution of average transaction values across customers is independent of the transaction process”); a customer’s expected spend is a weighted average of the population mean and their own average. Estimates on the 946 CDNOW repeat buyers: p = 6.25, q = 3.74, γ = 15.44. Module 4 uses the independence check and these estimates as a checkpoint. Module 13 runs the same check on Online Retail II, and its maximum-likelihood helper is tested against these estimates.

  • Fader, P. S., & Hardie, B. G. S. (2007). How to project customer retention. Journal of Interactive Marketing, 21(1), 76–90. https://doi.org/10.1002/dir.20074
    Verified (2026-10-09). The shifted-beta-geometric (sBG) model, fitted to seven years of retention data, projects survival to year 12 accurately, while curve-fitting regressions do not; aggregate retention rises only because customers differ. Table 1 gives the survival percentages for “Regular” and “High End” customers used in Module 4, and in Module 14, which sets Kaplan-Meier beside the sBG on the same table. The paper prints the High End α as 0.688 in the text and 0.668 in Appendix B; our own maximum-likelihood fit to the Appendix B counts gives 0.668 (β = 3.806).

  • Gupta, S., Hanssens, D., Hardie, B., Kahn, W., Kumar, V., Lin, N., Ravishanker, N., & Sriram, S. (2006). Modeling customer lifetime value. Journal of Service Research, 9(2), 139–155. https://doi.org/10.1177/1094670506293810
    Verified (2026-10-09). A review of CLV model families and their use in segmentation and in spending on acquisition, retention and cross-selling. Background for Module 5. (Crossref spells Kahn’s first name “Wiliam”.)

Day 3 · Did the marketing cause the sales?

Experiments (Module 6)

  • Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. https://doi.org/10.1017/9781108653985
    Verified (2026-10-09). A practitioner reference for A/B tests, including guardrail checks such as sample ratio mismatch (it has a chapter of that name).

  • Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. Proceedings of WSDM 2013, 123–132. https://doi.org/10.1145/2433396.2433413
    Verified (2026-10-09). CUPED: adjusting for a pre-experiment covariate reduces metric variance (the authors report about 50% at Bing), so a test needs fewer users for the same power. Abstract read at https://exp-platform.com/cuped/.

  • Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing sample ratio mismatch in online controlled experiments: A taxonomy and rules of thumb for practitioners. Proceedings of KDD 2019, 2156–2164. https://doi.org/10.1145/3292500.3330722
    Verified (2026-10-09). A sample ratio mismatch (observed split different from the designed split) is a common sign of a broken experiment and must be checked before reading results.

  • Deng, A., Knoblich, U., & Lu, J. (2018). Applying the delta method in metric analytics: A practical guide with novel ideas. Proceedings of KDD 2018, 233–242. https://doi.org/10.1145/3219819.3219919
    Verified (2026-10-09). When users are randomized but the metric is a ratio (for example revenue per visit), its variance must come from the delta method, not from treating visits as independent.

  • Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2022). Always valid inference: Continuous monitoring of A/B tests. Operations Research, 70(3), 1806–1821. https://doi.org/10.1287/opre.2021.2135
    Verified (2026-10-09). Always-valid p-values and confidence intervals let analysts monitor a test continuously and stop at any time without inflating false positives. (The arXiv preprint has a different title and three authors.)

  • Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B tests: Why it matters, and what to do about it. Proceedings of KDD 2017, 1517–1525. https://doi.org/10.1145/3097983.3097992
    Verified (2026-10-09), record only. We confirmed the bibliographic record (Crossref and the first author’s publication list) but could not open the abstract; for the claim, see the 2022 paper above.

  • Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193–225. https://doi.org/10.1287/mksc.2018.1135
    Verified (2026-10-09; again on 2026-10-10 for Module 15). Across 15 advertising experiments, common observational methods often failed to reproduce the lift measured by the randomized experiments. This is the evidence behind “attribution is not incrementality” in Module 6. Modules 15 (observational estimates against an experiment) and 16 (attribution credit against incrementality) cite it again. On 2026-10-10 the publisher’s page refused automated access; the Marketing Science Institute’s summary, read that day, says the 15 were US advertising experiments at Facebook and that matching and regression-based methods overestimated effectiveness relative to the experiments: https://msi.org/working-papers/a-comparison-of-approaches-to-advertising-measurement-evidence-from-big-field-experiments-at-facebook-2/.

Geo-experiments and quasi-experiments (Module 7)

  • Vaver, J., & Koehler, J. (2011). Measuring ad effectiveness using geo experiments. Google Inc. https://research.google/pubs/measuring-ad-effectiveness-using-geo-experiments/
    Verified (2026-10-09). Randomly assigning non-overlapping regions to treatment and control is a simple way to measure the incremental effect of advertising.

  • Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. https://doi.org/10.1198/jasa.2009.ap08746
    Verified (2026-10-09). A weighted combination of untreated units can stand in for a single treated unit; for Proposition 99 the authors estimate per-capita cigarette sales about 26 packs lower than the synthetic California by 2000. Module 7 replicates the method on these data.

  • Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The augmented synthetic control method. Journal of the American Statistical Association, 116(536), 1789–1803. https://doi.org/10.1080/01621459.2021.1929245
    Verified (2026-10-09). Adding an outcome model (ridge regression) corrects synthetic control’s bias when the treated unit’s pre-period cannot be matched closely; implemented in the augsynth R package, which GeoLift builds on.

  • Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015). Inferring causal impact using Bayesian structural time-series models. The Annals of Applied Statistics, 9(1), 247–274. https://doi.org/10.1214/14-AOAS788
    Verified (2026-10-09). A Bayesian structural time-series model forecasts the counterfactual from control series and estimates how a campaign’s effect evolves; implemented in the CausalImpact R package.

  • Cunningham, S. Causal inference: The remix (online second edition, work in progress), chapter “Difference-in-differences fundamentals”. https://mixtape.scunning.com/09-difference_in_differences
    Verified (2026-10-09). Difference-in-differences identifies the effect on the treated only under parallel trends, which can be checked only indirectly (event-study leads, falsification tests). The page shows no publication year.

Day 4 · Where should the next dollar go?

Marketing mix models, calibration and Bayesian workflow (Modules 8–10)

  • Jin, Y., Wang, Y., Sun, Y., Chan, D., & Koehler, J. (2017). Bayesian methods for media mix modeling with carryover and shape effects. Google Inc. https://research.google/pubs/bayesian-methods-for-media-mix-modeling-with-carryover-and-shape-effects/
    Verified (2026-10-09). Advertising has lagged effects (carryover, or adstock) and diminishing returns (shape, or saturation) that linear regression misses; a Bayesian MMM with flexible forms captures them, and priors matter a lot in small samples. (Module 8.)

  • Chan, D., & Perry, M. (2017). Challenges and opportunities in media mix modeling. Google. https://research.google/pubs/challenges-and-opportunities-in-media-mix-modeling/
    Verified (2026-10-09). MMMs are widely used but often struggle to give valid answers, because of limited data, correlated inputs and confounding. (Module 9.)

  • Wang, Y., Jin, Y., Sun, Y., Chan, D., & Koehler, J. (2017). A hierarchical Bayesian approach to improve media mix models using category data. Google Inc. https://research.google/pubs/a-hierarchical-bayesian-approach-to-improve-media-mix-models-using-category-data/
    Verified (2026-10-09). Pooling data across brands in a category, or passing what it learns on as informative priors, improves estimates when one brand’s data are thin. (Module 9 briefing.)

  • Zhang, Y., Wurm, M., Li, E., Wakim, A., Kelly, J., Price, B., & Liu, Y. (2024). Media mix model calibration with Bayesian priors. Google. https://research.google/pubs/media-mix-model-calibration-with-bayesian-priors/
    Verified (2026-10-09). Making ROAS a model parameter lets experiment results calibrate an MMM through its prior, which reduces bias and uncertainty in ROAS. Module 9 does the related thing in PyMC-Marketing, adding lift tests to the likelihood.

  • Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P.-C., & Modrák, M. (2020). Bayesian workflow. arXiv:2011.01808. https://doi.org/10.48550/arXiv.2011.01808
    Verified (2026-10-09). Bayesian analysis is an iterative workflow of prior checks, fitting, computational diagnostics, posterior predictive checks and model comparison. Modules 8 and 9 follow it.

  • Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., & Bürkner, P.-C. (2021). Rank-normalization, folding, and localization: An improved R̂ for assessing convergence of MCMC (with discussion). Bayesian Analysis, 16(2), 667–718. https://doi.org/10.1214/20-BA1221
    Verified (2026-10-09). The rank-normalized split-R̂ and bulk and tail effective sample sizes that the labs check before reading a posterior. Section 2 recommends running at least four chains and using the draws only if R̂ < 1.01 and the rank-normalized (bulk) effective sample size is above 400; Module 8’s gate uses these two thresholds (read in the arXiv version, arXiv:1903.08008, through its ar5iv HTML rendering).

  • Hyndman, R. J., & Athanasopoulos, G. (2021). Forecasting: Principles and practice (3rd ed.). OTexts. Section 5.10, “Time series cross-validation”. https://otexts.com/fpp3/tscv.html
    Verified (2026-10-09). Time series cross-validation uses a series of test sets, each training set holding only observations before its test set; the procedure is also known as evaluation on a rolling forecasting origin. Module 9’s time-slice cross-validation is this procedure with 8-week test sets. The book’s home page gives 2021 in its suggested citation.

Day 5 · Whom should we target?

Heterogeneous treatment effects and uplift (Module 11)

  • Wager, S., & Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228–1242. https://doi.org/10.1080/01621459.2017.1319839
    Verified (2026-10-09). Causal forests estimate how a treatment effect varies across individuals and give valid confidence intervals for those effects.

  • Athey, S., Tibshirani, J., & Wager, S. (2019). Generalized random forests. The Annals of Statistics, 47(2), 1148–1178. https://doi.org/10.1214/18-AOS1709
    Verified (2026-10-09). The general forest framework behind causal forests and the grf package.

  • Athey, S., & Wager, S. (2019). Estimating treatment effects with causal forests: An application. Observational Studies, 5(2), 37–51. https://doi.org/10.1353/obs.2019.0001
    Verified (2026-10-09). A worked application of causal forests to real data, covering propensity scores, clustering and tests for heterogeneity.

  • Yadlowsky, S., Fleming, S., Shah, N., Brunskill, E., & Wager, S. (2025). Evaluating treatment prioritization rules via rank-weighted average treatment effects. Journal of the American Statistical Association, 120(549), 38–51. https://doi.org/10.1080/01621459.2024.2393466
    Verified (2026-10-09). RATE metrics, with the Qini coefficient as a special case, measure how well a targeting rule picks out the people who benefit most; grf’s rank_average_treatment_effect implements them.

  • Künzel, S. R., Sekhon, J. S., Bickel, P. J., & Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10), 4156–4165. https://doi.org/10.1073/pnas.1804597116
    Verified (2026-10-09). Meta-learners turn any regression method into an estimator of individual treatment effects; the X-learner is efficient when one group is much larger than the other, as with marketing holdouts.

  • Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097
    Verified (2026-10-09; again on 2026-10-10 for Module 15). Orthogonal scores with cross-fitting remove the bias that regularized machine-learning models introduce, the basis of EconML’s CausalForestDML (Module 11) and of the double machine learning that Module 15 writes by hand and then fits with EconML’s LinearDML. The abstract, read at https://arxiv.org/abs/1608.00060, describes K-fold sample splitting (“cross-fitting”) and estimates that converge at the 1/√n rate, are approximately unbiased and normal, and give valid confidence intervals.

  • Radcliffe, N. J., & Surry, P. D. (2011). Real-world uplift modelling with significance-based uplift trees (Portrait Technical Report TR-2011-1, Stochastic Solutions white paper). https://stochasticsolutions.com/pdf/sig-based-up-trees.pdf
    Verified (2026-10-09). Uplift modelling targets the change in behaviour caused by a treatment, needs a randomized control group, and is judged with incremental gains (Qini) curves; Qini measures are based on the area under the incremental gains curve.

  • Radcliffe, N. J. (2007). Using control groups to target on predicted lift: Building and assessing uplift model. Direct Marketing Analytics Journal, 14–21.
    Not read. We could not find the text online. The citation details come from the reference list of Radcliffe & Surry (2011) above, which says Qini curves and measures are discussed in detail in this paper. The title ends “uplift model” in every record we found.

Extensions · What changes when the data are messy, observational, path-shaped or drifting, or when the channel is one you own?

The optional modules after the core week (see Extensions). The sources of Modules 13–17 were checked on 2026-10-10 and those of Module 18 on 2026-10-16. A source that a core module already cites keeps its place above and names the extension module as well; library documentation and datasets are in their own sections below.

Messy real data (Module 13)

Module 13’s sources are all listed elsewhere on this page: Fader, Hardie & Lee’s note 004 under Day 1, Fader & Hardie’s note 025 under Day 2, DuckDB’s guide to the AsOf join under Library documentation, and Online Retail II, with Olist (considered, not used), under Datasets.

Survival analysis for subscription churn (Module 14)

  • Kaplan, E. L., & Meier, P. (1958). Nonparametric estimation from incomplete observations. Journal of the American Statistical Association, 53(282), 457–481. https://doi.org/10.1080/01621459.1958.10501452
    Verified (2026-10-10), record only. We confirmed the Crossref record, but the publisher refused automated access, so we did not read the text. Cited as the origin of the product-limit (Kaplan-Meier) estimator; the formula Module 14 uses is the one in the lifelines documentation (Library documentation).

  • Cox, D. R. (1972). Regression models and life-tables. Journal of the Royal Statistical Society, Series B, 34(2), 187–220. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x
    Verified (2026-10-10), abstract only. The proportional-hazards model: the hazard is a function of the explanatory variables and regression coefficients multiplied by an arbitrary, unknown function of time, and inference comes from a conditional likelihood. Module 14 fits it to adjust a confounded comparison. Abstract read on the EPA’s HERO record https://hero.epa.gov/reference/8785/. Crossref gives pages 187–202, the paper without the discussion.

  • Grambsch, P. M., & Therneau, T. M. (1994). Proportional hazards tests and diagnostics based on weighted residuals. Biometrika, 81(3), 515–526. https://doi.org/10.1093/biomet/81.3.515
    Verified (2026-10-10), abstract only. A coefficient that changes over time (their example: a treatment effect that weakens) shows up when a residual plot is smoothed, and several proportional-hazards tests are weighted least-squares lines on that plot. The basis of lifelines’ proportional_hazard_test, which Module 14 uses to find the effect that fades. Abstract read on the publisher’s page.

  • Royston, P., & Parmar, M. K. B. (2013). Restricted mean survival time: an alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome. BMC Medical Research Methodology, 13, 152. https://doi.org/10.1186/1471-2288-13-152
    Verified (2026-10-10). The restricted mean survival time is “a measure of average survival from time 0 to a specified time point”, the area under the survival curve up to that point. It is valid whatever the distribution of the event times, so it needs no proportional hazards. Module 14 prices onboarding in these months of subscription. Read on PubMed Central: https://pmc.ncbi.nlm.nih.gov/articles/PMC3922847/.

  • Hernán, M. A. (2010). The hazards of hazard ratios. Epidemiology, 21(1), 13–15. https://doi.org/10.1097/EDE.0b013e3181c1ea43
    Verified (2026-10-10). A hazard ratio can change during follow-up, “period-specific HRs have a built-in selection bias”, and adjusted survival curves are a clearer summary. Why Module 14 ends in months of subscription rather than a hazard ratio. Read on PubMed Central: https://pmc.ncbi.nlm.nih.gov/articles/PMC3653612/.

  • Giobbie-Hurder, A., Gelber, R. D., & Regan, M. M. (2013). Challenges of guarantee-time bias. Journal of Clinical Oncology, 31(23), 2963–2969. https://doi.org/10.1200/JCO.2013.49.5283
    Verified (2026-10-10). A landmark analysis “selects a fixed time during follow-up as the landmark”; guarantee-time bias includes immortal-time bias. Module 14 estimates hazard ratios before and after a 3-month landmark. Read on PubMed Central: https://pmc.ncbi.nlm.nih.gov/articles/PMC3732313/.

  • Lee, H., & Nunan, D. (2020). Immortal time bias. Catalogue of Bias. https://catalogofbias.org/biases/immortal-time-bias/
    Verified (2026-10-10). Assigning groups with information from after time zero distorts the comparison; “assignment and time-zero must be aligned”. Why Module 14’s clock starts at the first billing date, when onboarding is already known.

Also for Module 14: Fader & Hardie (2007) under Day 2, and lifelines with its paper under Library documentation.

Causal effects without an experiment (Module 15)

  • Robinson, P. M. (1988). Root-N-consistent semiparametric regression. Econometrica, 56(4), 931–954. https://doi.org/10.2307/1912705
    Verified (2026-10-10), abstract only. Estimates based on a wrong parametric form for the nonparametric part of the model are generally inconsistent; inserting nonparametric regression estimates gives a root-N-consistent, asymptotically normal estimator of the slope. The ancestor of partialling out, which Module 15 writes by hand. Abstract read on RePEc: https://ideas.repec.org/a/ecm/emetrp/v56y1988i4p931-54.html.

  • Li, F., Morgan, K. L., & Zaslavsky, A. M. (2018). Balancing covariates via propensity score weighting. Journal of the American Statistical Association, 113(521), 390–400. https://doi.org/10.1080/01621459.2016.1260466
    Verified (2026-10-10), abstract only. Overlap weights, proportional to each unit’s probability of being assigned to the other group, are bounded and minimize the asymptotic variance of the weighted average treatment effect. Module 15 uses them to say whose effect double machine learning estimates; the weight e(1 − e), with e the propensity score, is derived in the module, not quoted from the paper. Crossref record, and the abstract on arXiv https://arxiv.org/abs/1609.07494, whose listing says the preprint was withdrawn by the author; the journal article stands.

  • Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1), 187–199. https://doi.org/10.1093/biomet/asn055
    Verified (2026-10-10), abstract only. For a wide range of distributions, a good approximation to the authors’ optimal rule is to discard units whose estimated propensity score lies outside [0.1, 0.9]. Module 15 trims at this rule and at [0.2, 0.8]. Abstract read on the publisher’s page.

  • LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604–620. https://ideas.repec.org/a/aea/aecrev/v76y1986i4p604-20.html
    Verified (2026-10-10), abstract only. Compares nonexperimental estimates with the results of a randomized employment and training experiment; many econometric procedures fail to replicate the experimental results. The design of Module 15’s Hillstrom benchmark. Abstract read on RePEc.

  • Chernozhukov, V., Cinelli, C., Newey, W., Sharma, A., & Syrgkanis, V. (2026). Long story short: Omitted variable bias in causal machine learning. Review of Economics and Statistics, published online 2026-02-04. https://doi.org/10.1162/REST.a.1705
    Verified (2026-10-10), abstract only. Bounds on omitted-variable bias for causal parameters, including in partially linear models, from plausibility judgments on how much the omitted variables could explain, with inference that can use machine-learning estimators. Module 15’s third stretch exercise only; we did not check EconML’s c_y and c_t against the paper. Abstract read on arXiv https://arxiv.org/abs/2112.13398; journal details from Crossref. For its sensitivity methods, EconML’s references page cites a 2022 NBER working paper of the same title (No. 30302), which it lists with N. Kallus as a sixth author.

Also for Module 15: Chernozhukov et al. (2018) under Day 5, Gordon et al. (2019) under Day 3, EconML’s LinearDML and LightGBM under Library documentation, and the Hillstrom experiment under Datasets.

Multi-touch attribution (Module 16)

  • Shao, X., & Li, L. (2011). Data-driven multi-touch attribution models. Proceedings of KDD 2011, 258–264. https://doi.org/10.1145/2020408.2020453
    Verified (2026-10-10), abstract only. Multi-touch attribution gives credit to more than one ad according to its contribution, instead of all of it to the last ad; the authors frame the task as interpreting each interaction’s influence accurately and stably, not only as classifying users. Crossref record, and the abstract through OpenAlex (the ACM page refused automated access).

  • Dalessandro, B., Perlich, C., Stitelman, O., & Provost, F. (2012). Causally motivated attribution for online advertising. Proceedings of the Sixth International Workshop on Data Mining for Online Advertising and Internet Economy (ADKDD 2012), 1–9. https://doi.org/10.1145/2351356.2351363
    Verified (2026-10-10), abstract only. Attribution posed as a causal estimation problem, with approximate methods that draw on cooperative game theory; where the causal assumptions are violated, these approximations are measures of variable importance. Crossref record, and the abstract through OpenAlex.

  • Anderl, E., Becker, I., von Wangenheim, F., & Schumann, J. H. (2016). Mapping the customer journey: Lessons learned from graph-based online attribution modeling. International Journal of Research in Marketing, 33(3), 457–474. https://doi.org/10.1016/j.ijresmar.2016.03.001
    Verified (2026-10-10), abstract only. Customer paths modeled as first- and higher-order Markov walks, applied to four large data sets with at least seven online channels each; the results differ substantially from last-click attribution. Abstract read on RePEc https://ideas.repec.org/a/eee/ijrema/v33y2016i3p457-474.html; ScienceDirect refused automated access, so the removal-effect formula on the Module 16 page is the module’s own statement, not quoted from the paper.

  • Shapley, L. S. (1953). A value for n-person games. In Contributions to the Theory of Games (AM-28), Volume II (pp. 307–318). Princeton University Press. https://doi.org/10.1515/9781400881970-018
    Verified (2026-10-10). We read the Crossref record and pages 1–7 of the earlier version, RAND paper P-295 (1952). The value is deduced from three axioms: symmetry, efficiency and additivity (Shapley’s “law of aggregation”: the value of the sum of two games is the sum of their values), which determine it uniquely. That a player who adds nothing to any coalition gets zero (the null-player property) is his Lemma 1. Module 16 computes the value exactly, with the conversion rate of each set of channels as the value of a coalition. Pages 307–318 as Crossref lists them.

  • Zhao, K., Mahboobi, S. H., & Bagheri, S. R. (2018). Shapley value methods for attribution modeling in online advertising. arXiv:1804.05327. https://arxiv.org/abs/1804.05327
    Verified (2026-10-10), abstract only. Shapley-value attribution across channels, a simplified formula, and an “ordered” variant that uses the order of touches. Mentioned in the Module 16 briefing; we did not read the full text, so the module does not describe its value function.

Also for Module 16: Gordon et al. (2019) under Day 3, and ChannelAttribution, considered and not used, under Library documentation.

MMM with drifting effects and a Prophet baseline (Module 17)

  • Taylor, S. J., & Letham, B. (2018). Forecasting at scale. The American Statistician, 72(1), 37–45. https://doi.org/10.1080/00031305.2017.1380080
    Verified (2026-10-10), abstract only. The paper behind Prophet; Prophet’s README cites it as “Prophet paper”. The abstract proposes “a modular regression model with interpretable parameters that can be intuitively adjusted by analysts with domain knowledge about the time series”. Crossref record, and the abstract on Semantic Scholar; the publisher’s page and the PeerJ preprint page refused automated access.

  • Riutort-Mayol, G., Bürkner, P.-C., Andersen, M. R., Solin, A., & Vehtari, A. (2023). Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming. Statistics and Computing, 33, 17. https://doi.org/10.1007/s11222-022-10167-2
    Verified (2026-10-10), record only. We confirmed the Crossref record (published online 2022-12-14, issue dated February 2023) but did not read the paper. Cited because PyMC-Marketing models Module 17’s drifting effects with a Hilbert-space approximation to a Gaussian process (HSGP): in the installed PyMC-Marketing 1.2.0, parameterize_from_data sets the number of basis functions and the boundary with approx_hsgp_hyperparams, whose docstring says it is “Based on recommendations from” this paper.

Also for Module 17: Prophet, PyMC-Marketing’s time-varying notebooks and Robyn’s use of Prophet under Library documentation.

Notifications (Module 18)

Production systems, in the order the Module 18 briefing discusses them, then the methods and the platform documents behind its channel table. All fetched on 2026-10-16.

  • Gauci, J., Conti, E., Liang, Y., Virochsiri, K., He, Y., Kaden, Z., Narayanan, V., Ye, X., Chen, Z., & Fujimoto, S. (2019). Horizon: Facebook’s open source applied reinforcement learning platform. arXiv:1811.00260, v5 (2019-09-04). https://arxiv.org/abs/1811.00260
    Verified (2026-10-16). The abstract page (ten authors, five versions, the last dated 4 September 2019) and the full text on ar5iv (https://ar5iv.labs.arxiv.org/html/1811.00260). Section 9.1, push notifications: the previous system used “supervised learning models for predicting click through rate (CTR)”; the new policy is a discrete-action DQN whose actions are “sending and dropping the notification”, trained with “rewards for interactions and activity on Facebook, with a penalty for sending the notification”, with a PID-tuned threshold and daily retraining; they report “a significant improvement in activity and meaningful interactions” and publish no numeric lift. Module 18’s per-send penalty in value iteration is this paper’s device.

  • ReAgent (Python package) 0.1, uploaded to PyPI 2020-05-27, BSD 3-Clause. https://pypi.org/project/reagent/
    Verified (2026-10-16). The project page says “The platform was once named”Horizon” but we have adopted the name “ReAgent” recently” and cites the Horizon paper. Cited as where the Horizon code now lives; no lab uses it.

  • Gao, Y., Gupta, V., Yan, J., Shi, C., Tao, Z., Xiao, P. J., Wang, C., Yu, S., Rosales, R., Muralidharan, A., & Chatterjee, S. (2018). Near real-time optimization of activity-based notifications. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2018), 283–292. https://doi.org/10.1145/3219819.3219880
    Verified (2026-10-16), abstract only. Crossref record (title, eleven authors, pages 283–292); all authors at LinkedIn Corporation and the abstract through OpenAlex and Semantic Scholar: sending a notification for every event “would result in too many notifications”; the trade-off between engagement and send volume is handled “by formulating the problem using constrained optimization”, run in a “stream computing system” making “multi-channel send decisions in near real-time”; online A/B tests on “tens of millions of members”. The ACM page refused automated access, and the talk’s page on videolectures.net is behind a bot check, so the full text was not read.

  • Zhao, B., Narita, K., Orten, B., & Egan, J. (2018). Notification volume control and optimization system at Pinterest. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2018), 1012–1020. https://doi.org/10.1145/3219819.3219906
    Verified (2026-10-16), abstract only. Crossref record (pages 1012–1020, all authors at Pinterest) and the abstract through OpenAlex: notifications across email, mobile and desktop push and SMS; the central question is how often each user should be notified; a machine-learned volume per user optimized for long-term engagement, deployed in mid-2017, which reduced volume while improving click-through rate. The ACM page refused automated access.

  • Pinterest Engineering (2020). User state based notification volume optimization. Medium post.
    Not read. Medium refused automated access to the post (a Cloudflare block, also with a browser user agent), and no archive copy could be reached, so it is cited by title only and not linked; the Module 18 briefing relies on the KDD 2018 paper above for what Pinterest’s system did.

  • Yancey, K. P., & Settles, B. (2020). A sleeping, recovering bandit algorithm for optimizing recurring notifications. Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2020), 3008–3016. https://doi.org/10.1145/3394486.3403351; PDF at https://research.duolingo.com/papers/yancey.kdd20.pdf
    Verified (2026-10-16). Crossref record and the full text from Duolingo’s research page. A bandit over hand-written reminder templates for “millions of daily reminders”; the reward “is 1 if the user completes a lesson within two hours of the reminder being sent, and 0 otherwise”; sleeping arms are templates with eligibility criteria that are not available every round, recovering arms are templates whose reward depends on how recently they were last used (a novelty effect, handled with a recency penalty and a 15-day half-life); the online experiment reports “a 0.5% increase in total daily active users (DAUs) and a 2% increase in new user retention”. The offline data were 34 days of logs, 88 million training and 114 million test rows. Module 18’s third stretch exercise and its briefing cite it as the production template bandit.

  • Yancey, K., & Settles, B. (2020). Replication data for: A sleeping, recovering bandit algorithm for optimizing recurring notifications. Harvard Dataverse, published 2020-06-11, CC BY-NC 4.0. https://doi.org/10.7910/DVN/23ZWVI
    Verified (2026-10-16), record only. Read through the Dataverse API: “Replication datasets (200 million rows)”, six archives (three training, three test, about 9.4 GB in all) and a readme, linked to the paper above. The field list is in the readme, which we did not open; the paper describes the logged data as the notification’s timestamp, user ID, eligible templates, selected template, the templates sent in the prior 30 days, and the reward. No lab downloads it; the stretch asks which exercises could run on it.

  • Yuan, Y., Muralidharan, A., Nandy, P., Cheng, M., & Prabhakar, P. (2022). Offline reinforcement learning for mobile notifications. Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM 2022), 3614–3623. https://doi.org/10.1145/3511808.3557083; arXiv:2202.03867, https://arxiv.org/abs/2202.03867
    Verified (2026-10-16). Crossref record (CIKM 2022, pages 3614–3623), the arXiv abstract page and the full text on ar5iv (https://ar5iv.labs.arxiv.org/html/2202.03867); every author at LinkedIn Corporation. Actions SEND and NOT-SEND (the candidate returns to the queue); the reward is “a user visit to the platform within the next time step”; a Double DQN trained offline on data from online exploration, with “a state-marginalized importance sampling algorithm for offline evaluation”; online A/B against the baseline: sessions +0.30%, notification cards −3.49%, click-through +4.53%, notification unfollows −4.37%.

  • Prabhakar, P., Yuan, Y., Yang, G., Sun, W., & Muralidharan, A. (2022). Multi-objective optimization of notifications using offline reinforcement learning. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2022), 3752–3760. https://doi.org/10.1145/3534678.3539193; arXiv:2207.03029, https://arxiv.org/abs/2207.03029
    Verified (2026-10-16). Crossref record (KDD 2022, pages 3752–3760), the arXiv abstract page and the full text on ar5iv (https://ar5iv.labs.arxiv.org/html/2207.03029); every author at LinkedIn Corporation (not Meta). The reward is a weighted linear combination of sessions, notification clicks and a volume penalty of −1 per send, with the weights as a preference vector; the learner is a Double DQN with conservative Q-learning, because “Directly utilizing existing value-based off-policy RL algorithms in an offline setting results in poor performance” under distribution shift. Online A/B against the baseline: without CQL, volume +7.72%, click-through −7.79%, weekly active users −0.69%; with CQL, sessions +0.24%, weekly active users +0.18%, volume −1.73%, click-through +2.26%. Disables appear as a guardrail, not in the reward. Module 18’s greedy_policy restriction is the lab’s small version of the same conservatism.

  • Ocejo, B., Wang, R., Liu, K., Patra, R. K., Shen, H., Liu, D., Yuan, Y., Mohanasundaram, G., Borisyuk, F., & Prabhakar, P. (2025). Generative sequential notification optimization via multi-objective decision transformers. arXiv:2509.02458, v1 (2025-09-02), CC BY 4.0. https://arxiv.org/abs/2509.02458
    Verified (2026-10-16). The abstract page and the full text at https://arxiv.org/html/2509.02458v1; every author at LinkedIn. A Decision Transformer treats policy learning as return-conditioned supervised learning, with a multi-objective reward (predicted notification value, actual visits, adaptive volume penalties) and quantile regression for the return-to-go; compared with a multi-objective CQL agent, the deployed policy “achieved a 0.72% increase in sessions while reducing notification volume”, with no significant change in click-through. No venue is listed on the page.

  • Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3–4), 285–294. https://doi.org/10.1093/biomet/25.3-4.285
    Verified (2026-10-16), record only. Crossref record; the text was not read. Cited as the origin of Thompson sampling, which Module 18’s Exercise 4 writes as one Normal draw per context and action.

  • Apple Developer Documentation. Asking permission to use notifications. https://developer.apple.com/documentation/usernotifications/asking-permission-to-use-notifications
    Verified (2026-10-16), through the page’s documentation JSON (the HTML page is script-rendered). “The first time your app makes this authorization request, the system prompts the person to grant or deny the request and records that response. Subsequent authorization requests don’t prompt the person.” Provisional authorization delivers notifications “quietly”, only in the notification center’s history, with Keep and Turn Off buttons. Module 18’s channel table.

  • Android Developers. Notification runtime permission. https://developer.android.com/develop/ui/views/notifications/notification-permission
    Verified (2026-10-16). Android 13 (API level 33) introduces the runtime permission POST_NOTIFICATIONS; on a fresh install on Android 13 or higher “your app’s notifications are off by default. Your app must wait to send notifications until after you request the new permission and the user grants that permission to your app”; for apps targeting 12L or lower, one tap of “Don’t allow” prevents further prompts until reinstall. Module 18’s channel table.

  • MDN Web Docs. Push API. https://developer.mozilla.org/en-US/docs/Web/API/Push_API
    Verified (2026-10-16). “The Push API gives web applications the ability to receive messages pushed to them from a server, whether or not the web app is in the foreground, or even currently loaded”; it needs an active service worker and reaches “users that opt in”; marked Baseline widely available since March

    1. Module 18’s channel table.

  • Federal Trade Commission. CAN-SPAM Act: A compliance guide for business. https://www.ftc.gov/business-guidance/resources/can-spam-act-compliance-guide-business
    Verified (2026-10-16). A commercial email must include “a clear and conspicuous explanation of how the recipient can opt out”, and opt-out requests must be honored within 10 business days; the guide states a per-email civil penalty maximum (an inflation-adjusted figure, updated on the page). Module 18’s channel table, for email’s permission model.

Also for Module 18: Module 6’s experiment references under Day 3 and Module 11’s uplift references under Day 5; the simulated app users under Datasets.

Library documentation

Versions are the ones the labs pin (_variables.yml, and environment/requirements.txt for libraries not listed there).

Datasets

  • CDNOW (1/10th sample), distributed by Bruce Hardie. https://www.brucehardie.com/datasets/
    Verified (2026-10-09). 2,357 customers, a systematic 1/10th sample of the 23,570 people who first bought at CDNOW in the first quarter of 1997, with their purchases to the end of June 1998 (6,919 records: customer, date, number of CDs, dollar value), according to the readme in the archive. Real data. Modules 1–4.

  • Online Retail II. Chen, D. (2012). UCI Machine Learning Repository. https://doi.org/10.24432/C5CG6D; dataset page https://archive.ics.uci.edu/dataset/502/online+retail+ii
    Verified (2026-10-09; again on 2026-10-10 for Module 13). “A real online retail transaction data set of two years”: transactions of a UK-based online gift-ware retailer, many of whose customers are wholesalers, from 01/12/2009 to 09/12/2011; 1,067,371 rows; license CC BY 4.0; donated 2019-09-20 (the citation line gives 2012). The variable descriptions say an invoice number that starts with the letter “c” marks a cancellation, a stock code is a five-digit number for each product, and prices are in sterling; the file names the columns Invoice and Price where the page says InvoiceNo and UnitPrice. Real data. Modules 1 and 13.

  • Online Retail (the one-year predecessor). Chen, D. (2015). UCI Machine Learning Repository. https://doi.org/10.24432/C5BW33. Introductory paper: Chen, D., Sain, S. L., & Guo, K. (2012). Data mining for the online retail industry: A case study of RFM model-based customer segmentation using data mining. Journal of Database Marketing & Customer Strategy Management, 19(3), 197–208. https://doi.org/10.1057/dbm.2012.17
    Verified (2026-10-09) for the dataset page and the paper’s record; we did not read the paper’s abstract. Background for Module 1 only.

  • Olist (considered for Module 13, not used). Brazilian E-Commerce Public Dataset by Olist, Kaggle. https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce; metadata from Kaggle’s API https://www.kaggle.com/api/v1/datasets/view/olistbr/brazilian-ecommerce
    Verified (2026-10-10). “100k orders from 2016 to 2018”; license CC BY-NC-SA 4.0; last updated 2021-10-01. Kaggle’s API documentation https://raw.githubusercontent.com/Kaggle/kaggle-api/main/docs/README.md says “First, you will need a Kaggle account”. Not used, for three reasons: the non-commercial license, the account a learner would need, and few repeat buyers (2.2% of customers bought on more than one day, measured on the files downloaded that day). Real data.

  • Hillstrom e-mail experiment. Hillstrom, K. (2008, March 20). The MineThatData e-mail analytics and data mining challenge. MineThatData blog. https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html
    Verified (2026-10-09; again on 2026-10-10 for Module 15). 64,000 customers who bought in the past twelve months, randomly split into thirds: Mens e-mail, Womens e-mail, no e-mail; outcomes are visit, conversion and spend in the following two weeks. Real, randomized data. Modules 6, 11 and 15; Module 15 draws a deliberately confounded sample from it and checks double machine learning against the randomized answer.

  • Proposition 99 panel (California and 38 control states, 1970–2000), as the smoking dataset in the tidysynth R package. Dunford, E. tidysynth, version 0.2.1 (CRAN, 2025-03-24). https://CRAN.R-project.org/package=tidysynth
    Verified (2026-10-09). 1,209 rows: state, year, cigarette sales and predictors; the package README replicates Abadie et al. (2010). The manual labels the unit of cigsale differently from the paper’s “per-capita packs”, so the lab states the unit it uses. The Synth R package does not contain these data. Real data. Module 7.

  • Fader–Hardie retention data, in Fader & Hardie (2007), above.
    Verified (2026-10-09). Table 1 gives the percentage of a cohort surviving at least 0–12 years for “Regular” and “High End” customers; Table B1 gives counts for 1,000 High End customers for years 1–7. Counts for Regular customers are not printed and are derived from the percentages (out of 1,000). Modules 4 and 14.

  • PyMC-Marketing mmm_example.csv. https://github.com/pymc-labs/pymc-marketing/blob/main/data/mmm_example.csv
    Verified (2026-10-09). A demo file in the PyMC-Marketing repository (179 weekly rows: date_week, y, x1, x2, event_1, event_2, dayofyear, t) used by its README and MMM quickstart. The pages that use it point to a “simulated example”, and no real-world source is given; the file itself is not labelled. We treat it as simulated demo data whose true parameters we do not have. Module 8.

  • Robyn dt_simulated_weekly, documented in the Robyn package source (https://github.com/facebookexperimental/Robyn).
    Verified (2026-10-09). Documented as “Simulated MMM data”; Robyn’s analyst guide says its outputs are based on simulated data and are for illustration only. Module 9’s R notebook.

  • Synthetic retailer (this workshop). Generated by mktstats.synth with the true parameters in data/synthetic/truth.json. Simulated, with known truth; Modules 1–13. The extensions’ own generated data (Module 13’s dirtied log, Module 14’s subscribers, Module 15’s app installs, Module 16’s customer journeys, Module 17’s weekly MMM with a fading channel and Module 18’s app users with their hidden engagement and fatigue sensitivity, whose send and volume experiments are generated live in the lab) keep their truth in separate files, data/synthetic/truth_<name>.json.

How the entries were checked

  • Every entry for the core week was opened on 2026-10-09, every entry added or read again for the Extensions (Modules 13–17) on 2026-10-10, and every entry for Module 18 on 2026-10-16: the DOI or publisher page, the Crossref record, arXiv, the author’s own page, or the documentation site. Several publisher pages (INFORMS, ACM, Elsevier’s ScienceDirect, Taylor & Francis, PNAS) refused automated access; for those we took the bibliographic details from Crossref and read the abstract on arXiv, an author or institution page, PubMed Central, RePEc, OpenAlex, the EPA’s HERO database, the Marketing Science Institute or Semantic Scholar.
  • A few pages we read are not linked, because they refuse automated clients (HTTP 403) and would fail the site’s monthly link check: RAND’s page for Shapley’s paper P-295, and the Oxford Academic abstract pages of Grambsch & Therneau (1994) and Crump et al. (2009). Their DOIs are linked instead.
  • The sentence after each tag says what the page we read supports in this workshop. Where the page differed from a commonly quoted version (a title, a year, an author list, a parameter value), we give what the page says and note the difference.
  • Not read marks an entry we cite because other sources rely on it, but whose text we could not open. Entries we could not confirm at all are left out. Record only means we confirmed the bibliographic record but read neither the abstract nor the text. Abstract only, a tag used on the entries checked on 2026-10-10, means we read the abstract but not the full text.
  • Where a source reports estimates that a lab uses as a checkpoint, we refitted them. On 2026-10-09, PyMC-Marketing 1.2.0 with flat priors (MAP) reproduced the BG/NBD estimates for CDNOW in note 004 and the Gamma-Gamma estimates in note 025, and a direct maximum-likelihood fit reproduced the sBG estimates in Fader & Hardie (2007). These were laptop runs while writing the lab briefs, not workshop run records. On 2026-10-10, Module 13’s maximum-likelihood helper (mktstats.mle) reproduced the estimates of notes 004 and 025 at their published precision (its unit tests), and a maximum-likelihood fit to years 0–7 of Table 1 in Fader & Hardie (2007) reproduced the published sBG estimates for both segments (Regular α = 0.704, β = 1.182; High End α = 0.668, β = 3.806, as in Appendix B), the fit Module 14 compares with Kaplan-Meier. Both were run again on the laptop for this page.
  • Versions of the libraries the labs use agree with _variables.yml and environment/requirements.txt; ChannelAttribution, which no lab uses, is described at the version we read. A library’s documentation changes with each release; the labs are checked against the installed version, not against these pages.