Teach this workshop

The instructor guide: the kit, the rhythm, pacing, debriefs and what goes wrong

This page is for the person running the room. It says what is in the kit, how a module and a day run, when to cut, what to ask in each debrief, and what to do when Colab, an install or a model fit goes wrong.

Read this first: what has run

The readiness page is generated from the run records in runs/. For every notebook it shows where it has run (Colab, the CI runner, the workshop’s Docker image), when, with which settings, how long it took, and whether that evidence is still current: a run counts for teaching only if it was on Colab, in worked mode, FULL, for the whole notebook, on the current code and dependencies, and no more than 21 days old. Every run time you plan with comes from that page. This guide gives no run times, except in the Extensions section, which names each extension lab’s slowest cell from its laptop and Docker run records, and the Colab Run all times of Modules 13 to 17 from the sweep of 2026-10-11; Module 18 has not run on Colab yet. A notebook that has not run is “written, not run”, and you should run it yourself before you teach it.

The instructor kit

• Module pages (modules/): each is the briefing for its module, with the objectives, the derivations, the decision, and a link to the lab. Teach from the page.
• Notebooks (Notebooks): one or two per module, Python or R, each opening in Colab from its badge at the release the workshop is pinned to. Notebooks are built from labs/src/ and never edited by hand.
• Solutions live in the notebooks, folded under each exercise. In Python notebooks on Colab, the solution cell shows as a collapsed form titled “Solution N — try it yourself first”. Colab’s R runtime has no forms; the build marks R solution cells as hidden, and whether Colab honors that is being checked (spike S3). If R solutions show, tell the room to scroll past them.
• Solutions never replace a learner’s code unless asked. Each reference is stored, not used, unless one of two switches is set:
  • WORKED_EXAMPLE in the harness cell at the top (or the environment variable MKTSTATS_WORKED=1): every exercise uses its reference, so Run all runs the whole lab. Use it to demonstrate, to prepare outputs before a session, and to make the capstone answer key. A worked run shows the lab works, not that anyone did it.
  • workshop.use_reference(n) in Python, use_reference(n) in R: exercise n only uses its reference, and the learner carries on.
• Every checkpoint says whose code it checked: “passed on your code” or “passed on the REFERENCE solution”. workshop.summary() (R: workshop_summary()) lists both at the end. Run all stops at the first unwritten exercise with a message that says how to continue.
• QUICK (switch in the harness cell, or MKTSTATS_QUICK=1) shortens sampling and subsamples data. It never skips a checkpoint. Intervals get wider and may not match the module page.
• Lab briefs (briefs/ on GitHub): for each lab, what each exercise asks for, the truth each checkpoint compares against, the data, and the decision. The lab standard says how every lab is built (Predict, Run, Explain, Check; checkpoint messages; the Decision cell).
• Knowledge checks: three to five questions per module with answers, for warm-ups, debriefs and an evening self-check.
• Day pages (Day 1, Day 2, Day 3, Day 4, Day 5): the timetable, the warm-up questions with answers, the clinics on Days 2 and 3, the capstone run of show on Day 5, and the wrap-up with the running results table.
• Extensions (Extensions): six optional modules beyond the week, Modules 13 to 18, with no timetable. The Extensions section below says how to swap one into a schedule and how each lab runs.
• Capstone task and rubric (on GitHub): the task for pairs, the five-slide brief, timing checkpoints, a rubric scored in 5 minutes per pair, and a score sheet.
• Before Day 1 (Before Day 1): the entry check and remediation you send participants a week ahead.

The shape of a day

Every teaching day answers one question and runs on the same frame; the schedule has the clock times.

• Warm-up (15 minutes). Four questions on the previous day, on the day page with folded answers. Participants answer alone for 5 minutes with notes closed and compare with a neighbor for 4. In your 5 minutes, ask for a show of hands on each question and explain the most-missed from its folded answer; do not reteach the module. Use the last minute for the day page’s “Why today”. Nothing is collected or graded.
• Modules. Three a day on Days 1 and 4; two on Days 2 and 3, which then have a 60-minute clinic after lunch (Day 2: the CFO memo clinic; Day 3: the geo-test design review). Day 5 is Module 11 in the morning and the capstone from late morning to the end of the day.
• Wrap-up (20 minutes). On the day page: what each module fixed and left open; the running results table, which you keep on the board or a shared sheet all week; and one decision claim each participant writes. Day 5’s wrap-up closes the week.

The shape of a module

Each module is a briefing (45 to 55 minutes), a lab (50 to 55 minutes) and a debrief (10 minutes); the exact minutes per module are on the schedule.

The lab’s first 5 minutes are for opening the notebook from the module page and running the install and harness cells. While the install runs, everyone reads the header and Exercise 1’s Predict prompt.

Every exercise runs Predict → Run → Explain → Check. Your moves:

1. Predict. Say “write your guess down” and mean it: on paper or in the cell. Ask two people for their guess before anyone runs anything. Do not correct predictions.
2. Run. The learner writes the function and runs it.
3. Explain. Ask someone whose prediction was wrong to explain the result. That is where the learning is. The folded “Why this solution works” is the answer to check against.
4. Check. The checkpoint passes or fails. When it fails, ask the learner to read the message aloud: it says what failed, what to check and how to recover.

While the lab runs, circulate. Do not type on a learner’s keyboard; point at the message. Say at the start of every lab that use_reference(n) is normal: it keeps people with the room, and the summary shows what they wrote themselves.

Each lab ends with a Decision cell (number → rule → recommendation with uncertainty) and then an optional stretch. Everyone reaches the Decision cell; nobody needs the stretch.

Two-notebook labs (Modules 3, 7 and 11) fit both notebooks into the one lab slot. The first cell of each notebook gives the order of work, the minutes in each notebook and the minute to switch; call the switch aloud, and anyone not finished uses the reference for what is left. Module 3: R first, then Python about 30 minutes in. Module 7: Python Parts A and B first, R’s Part C about 33 minutes in, then back to Python for the decision. Module 11: R first, then Python about 28 minutes in. In each, learners start the slower install first (the Python one in Modules 3 and 11, the R one in Module 7), so they hold two Colab runtimes at once.

Pacing rules

Apply these in order. A module’s own notes in its brief come first.

1. Start the lab on time. If the briefing runs long, cut its last section to the main result and send the room to the module page for the rest. Keep any demo that sets up the lab’s predictions.
2. The stated minutes plus 3. A learner stuck on one exercise for its stated minutes plus 3 runs use_reference(n) and moves on. They can come back to it in the stretch time or that evening.
3. The first checkpoint is a room check. If fewer than half the room has passed Checkpoint 1 by the end of Exercise 1’s stated minutes, stop the room for 2 minutes: show the key line of the solution, or ask everyone to use the reference for Exercise 1.
4. Long fits. While a model fits, learners read the next Predict prompt. If a fit takes more than twice the time the readiness page records for it, interrupt it, switch on QUICK and run the fit cell again; or use the reference for that exercise.
5. Ten minutes before the lab ends, call “Decision cell in five minutes”. Anyone not yet on the last core exercise uses the reference for what is left. At five minutes before the end, everyone is in the Decision cell.
6. The stretch is for individuals, never the room, and only for those who have passed every core checkpoint and written their decision sentence.
7. Shrink the debrief; never drop it. If the lab runs over, give it 5 minutes: the numbers on the board and the bridge to the next module.
8. Breaks and lunch keep their length. A module that runs over ends at its break; the next starts on time.
9. Warm-up stops at 15 minutes; wrap-up starts on time. If the last module ran over, skip the pair discussion in the wrap-up and go straight to the results table.
10. On any day, the stretch is the first thing dropped.

Debriefs (10 minutes per module)

Every debrief has three parts:

• The room’s numbers (4 minutes). Ask three or four people for the headline number from their Decision cell. Write each on the board with its interval (type and probability) and whether it was FULL or QUICK, own code or reference. The day’s wrap-up table is filled from these.
• Two or three questions (4 minutes) from the list below, tied to the module’s decision.
• The bridge (2 minutes). What this module could not do, and which module takes it up next. The day pages’ “Fixed and left open” lists them.

Do not reteach the briefing and do not debug code in front of the room.

Module 1 · Transaction logs to customer-base analysis

1. What share of customers bought only once? What does that do to any average taken over “customers”?
2. Which “recency” did you compute: time since the last purchase, or age at the last purchase? Where would mixing them up go wrong tomorrow?
3. Would you stop retargeting customers who have been inactive for N days? For the N you chose, how many still-active customers does the rule drop, and does the retargeting cost it saves outweigh the margin they would have brought? What does the rule ignore?

Module 2 · Buy-till-you-die models: BG/NBD and Pareto/NBD

1. Where did the BG/NBD parameters land against the truth, and which had the widest interval relative to its value? Why that one?
2. Two customers with the same number of repeat purchases have very different P(alive). What drives the difference?
3. Your threshold, τ = contact cost ÷ (margin per purchase × response rate): how many customers fall below it, and how much contact cost does stopping them save? How far did that count move between a 2% and a 10% response rate? Is a low P(alive) the same as “retargeting would not bring them back”? (Day 3 measures the response rate; Module 11 asks whom an offer changes.)

Module 3 · Covariates in CLVTools, and BTYD versus machine learning

1. Which acquisition channel’s customers buy more often or leave sooner, and how sure are you? Is that the channel’s effect, or who the channel reaches?
2. Did gradient boosting beat the BTYD model on the holdout period, on which metric? What does the BTYD model give you that the machine-learning model does not?
3. Which channels would you treat differently, by the lab’s rule (a rate more than 20% away from search, with a 90% interval that excludes no difference)? Which forecaster would you use for next quarter, and did the bootstrap interval of the error difference exclude 0? Would you change acquisition spend on this alone? (Day 2 adds what each channel’s customers are worth and cost.)

Module 4 · Monetary value and dollar CLV

1. Did the check that spend per transaction is unrelated to purchase frequency pass? What would you do if heavy buyers spent less per order?
2. How far does average customer value move when you change the discount rate or the horizon? Which assumption matters most for your number?
3. What are an average customer and the whole customer base worth over 12 and over 36 months, with 94% intervals? Which assumption moved the number most: the horizon, the discount rate or the spend model? Is that the number you would put in front of finance?

Module 5 · From CLV to decisions

1. What acquisition cost cap did you set per channel, and from which quantile of customer value? Why not the mean?
2. Two segments’ intervals overlap. Does that mean their values are not different? How did you check?
3. Would you run the proposed retention campaign? Which number in that calculation has nobody measured yet? (Day 3 measures it.)

Module 6 · Experiments done right

1. How many customers per arm did you need for the effect you care about? What did halving the detectable effect do?
2. What did peeking do to the false-positive rate in the simulation, and what fixes it?
3. Would you roll out the email? Give the incremental revenue per customer with its 95% interval after the SRM check and CUPED, and say whether the margin per email cleared the cost per email at the interval’s lower end. What did you check before reading the effect at all?

Module 7 · Geo-experiments and quasi-experiments

1. Did your interval contain the true lift? Where did synthetic control and difference-in-differences disagree, and why?
2. What did the placebo tests show? How large an effect could this design have detected?
3. Was the campaign incremental, and by how much? Give the incremental sales with their interval (the 90% placebo-in-time interval, and CausalImpact’s 95% interval if you ran it) and the return on the campaign spend at the interval’s lower end. Would you scale it?

Module 8 · Bayesian marketing mix models

1. Which channels’ true ROAS fell inside their interval? Which channel had the widest interval, and what in its spend history explains it?
2. Were there divergences? What do they mean, and what did you do about them?
3. Which channels pay back (probability at least 0.9 that ROAS is above break-even, 1 ÷ gross margin), which do not (at least 0.9 that it is below), and which are undecided? What would settle an undecided channel?

Module 9 · Calibrating and validating an MMM

1. How did the lift test change the tested channel’s ROAS: did the interval narrow, move, or both?
2. How did the model do on weeks it had not seen, and were its parameters stable across time windows?
3. Which channel’s ROAS is reliable enough to move budget on now, and which needs a new lift test first? Why that one?

Module 10 · Budget allocation under uncertainty

1. How far did the optimizer move from the current split, and which constraint stopped it?
2. How did the allocation that accounts for risk differ from the one computed at the posterior mean? Why?
3. Which weekly split would you recommend within the agreed bounds, with its expected incremental sales and 94% interval, and what probability that it beats the current plan? How did the risk-averse and the CLV-weighted objectives change it?

Module 11 · Uplift and heterogeneous treatment effects

1. Are the customers with the highest predicted uplift the ones most likely to buy? Why not?
2. On held-out customers, did targeting by predicted uplift beat random targeting? What did the RATE estimate say about whether the differences are real?
3. How many customers clear margin × predicted uplift > offer cost, and how much more (or less) does mailing only them earn than mailing everyone, with its interval? What happens to the rule if the offer costs twice as much?

Module 12 · Capstone: one decision pipeline for the retailer

The presentations are the debrief. After the last pair, ask the room:

1. Which stage’s uncertainty mattered most for the final recommendation?
2. Which assumption in your pipeline would you test first at your company?

Clinics and the capstone

• Day 2, CFO memo clinic. Print one CFO card and one Analyst checklist (both on the Day 2 page) per person. Groups of three; you keep time and call each step aloud. Write down the gaps you hear for the wrap-up.
• Day 3, geo-test design review. Before the session, run Module 7’s R notebook on a Colab R runtime with WORKED_EXAMPLE on and keep its market-selection output open for the demo; do not run market selection live. If a team asks how to analyze a design with several treated markets, the answer is to aggregate them into one series (as Lab 7 does) or use CausalImpact, not GeoLift() (see Common failures seen in testing). The plan and checklist are on the Day 3 page.
• Day 5, capstone. Announce the pairs at the end of Day 4. Pair people whose strengths differ (one at ease with the code, one with the business logic), and avoid pairing two people who both needed the reference through most of Day 4. Before presentations, prepare the answer key (the rubric says how) and decide who scores. With more than 12 pairs, split into two rooms with one scorer each. The run of show is on the Day 5 page.

What goes wrong, and what to do

Symptom	Likely cause	What to do
The install cell says to restart the session, then Run all.	Colab had already loaded an older version of a package the labs pin (for example PyMC or ArviZ), and the runtime guard refuses to continue with it.	Runtime → Restart session, then Runtime → Run all, once. If it asks again, Runtime → Disconnect and delete runtime, reconnect and Run all.
The install is slow or pip fails.	Network load in the room, or Colab’s image changed since the last recorded run.	Re-run the install cell once. Check the readiness page: if the last Colab run is older than the age limit, Colab may have moved; note the COLAB_RELEASE_TAG the run record prints and tell the Technical Expert. For a demo, use the workshop’s Docker image.
An R notebook fails at once with errors such as SyntaxError on <-.	The notebook is running on a Python runtime.	Runtime → Change runtime type → R, then Run all. Opening the notebook from its badge should select R; check it anyway at the start of each R lab.
R packages install slowly or fail.	R packages come as binaries from a dated Posit Package Manager snapshot, which serves binaries only when R identifies itself correctly; the install cell sets this. A restart in the middle of the install leaves it half done. CLVTools needs the GSL system library, which the install cell installs. GeoLift and augsynth are not on CRAN; the install cell fetches them from pinned GitHub archives.	Re-run the install cell from the top, not from the middle. Never install.packages("GeoLift"). If GitHub is unreachable from the room, use the reference outputs prepared in advance for the GeoLift demo.
A model fit runs far past its expected time.	A slow shared runtime, FULL settings, or a model that is struggling (see divergences).	Learners read the next Predict prompt while it runs. Past twice the recorded time: Runtime → Interrupt execution, switch on QUICK in the harness cell, re-run the fit cell; or use the reference for that exercise. Do not edit sampler settings by hand: QUICK is the short setting the checkpoints were tested with.
The sampler warns of divergences.	The sampler could not explore part of the posterior; estimates may be biased.	Do not ignore it, and do not just draw more samples. Follow the lab: its checkpoint states how many divergences it accepts and which setting to change (for example, a higher target acceptance rate). In Module 8 the warning may be part of the lesson.
A data download fails.	The source site is down or slow, or the room’s network blocks it.	The workshop’s loaders try each listed source in order, check every file’s SHA-256 hash and keep a cache; the message lists each URL it tried and the error. If every source fails, one person downloads the file from one of those URLs and shares it. Learners upload it in Colab’s Files panel (it lands in /content), move it into the loaders’ cache with !mkdir -p ~/.cache/mktstats && mv /content/FILE ~/.cache/mktstats/ (in R: system("mkdir -p ~/.cache/mktstats && mv /content/FILE ~/.cache/mktstats/")), and rerun the cell. FILE is the name the loader expects: cdnow_transactions.csv, online_retail_ii.zip, hillstrom.csv or mmm_example.csv. A cached file with the right hash is used without a download.
Colab disconnected, and variables are gone.	Idle timeout or a runtime reset (for example over lunch).	Re-run from the top; switch on QUICK if the clock is short. For the capstone, keep the tab open during a fit.
Colab warns that the notebook was not authored by Google.	Any notebook opened from GitHub.	Run anyway: the notebooks come from the workshop’s public repository.
Colab will not connect a new runtime and mentions too many sessions.	Every open Colab notebook holds its own runtime. The two-notebook labs (Modules 3, 7 and 11) need two at once, and tabs from earlier labs may still hold theirs. Seen while recording the Colab runs.	Runtime → Manage sessions, terminate the sessions of earlier labs, then connect again. Keep connected only the notebooks the current lab needs.
The runtime runs out of memory.	Several large objects, or several Colab tabs each holding a runtime.	Close other Colab tabs; Runtime → Restart session; switch on QUICK, which subsamples the data.
A learner cannot find the checkpoint’s message.	In Python notebooks the harness prints its [workshop] line before the error, so the checkpoint’s own message is the last line of the red error below it, as the line says.	Point at the last line of the error and ask the learner to read it aloud.
A checkpoint fails on the reference solution.	A bug, or a package that changed.	Not the learner’s fault. Note the notebook, the checkpoint, the full message and the run’s COLAB_RELEASE_TAG, and report it. Tell the room to run the following cells one at a time instead of Run all.

Common failures seen in testing, by module

What went wrong, or surprised us, while the labs were built and run, from the run records in runs/, DECISIONS.md and the lab build notes, with what to tell the room. Where each notebook has run, and how long it took, is on the readiness page; check it before each day, because a lab that has not yet run on Colab can still surprise you there.

• Every two-notebook lab (Modules 3, 7 and 11). Learners hold two Colab runtimes at once. While the Colab runs were recorded, Colab refused a new runtime with a “too many sessions” message when earlier notebooks were still connected; the fix is in the table above.
• The workshop’s Docker image with a CPU limit. Under docker run --cpus 2 on a laptop with more cores, the linear-algebra library started one thread per visible core and SciPy’s optimizer in Module 7 slowed about 75-fold; the lab now limits it to one thread with threadpoolctl (DECISIONS.md). Colab and the GitHub runner are not expected to be affected (not verified on Colab). Run records made in such a container report the limit (rounded up, read from the container’s cgroup); records made before v2026.10.2 report the host’s core count.
• Module 0. Nothing beyond the general table above.
• Module 1. The first load of Online Retail II downloads and converts a 45 MB file from the UCI repository; later loads read the cache. Learners read Exercise 2 meanwhile. Checkpoint 1b compares invoice counts with counts measured on the hash-checked file, so a different count means a cleaning rule differs, not that the data changed.
• Module 2. The parameter-recovery check (3 of the 4 true BG/NBD parameters inside their 94% HDI) passed on 17 of 20 simulated cohorts in a scan of seeds; the misses came in pairs (r with α, a with b). The notebook fixes a cohort that passes in QUICK and FULL. If a learner changes the cohort seed and Checkpoint 3 fails, that is chance, not their code. The Pareto/NBD is fitted by MAP because sampling it was too slow for class on Colab (DECISIONS.md, spike S2).
• Module 3. CLVTools needs the GSL system library, which Colab lacks; the R install cell installs it (spike S3). If library(CLVTools) fails, rerun the install cell from the top.
• Module 6. Hillstrom downloads from minethatdata.com over plain HTTP, with a gzip copy on Amazon S3 as the fallback; both are hash-checked. If minethatdata.com is unreachable from the room, the loader moves to the fallback after its retries, so the first load can take longer.
• Module 7. GeoLift() fails with two or more test markets at the pinned commits (reproduced on GeoLift’s own example; DECISIONS.md). Market selection and power, which the clinic demo uses, work. The augsynth stretch with progfunc = "ridge" cross-validates its penalty and is much slower than the default; the stretch says so.
• Module 8. FULL draws 2 × 2,000 because 2 × 1,000 left the smallest bulk ESS below the lab’s bar of 400 in testing; with QUICK the diagnostics gate usually fails, which the notebook says is expected. target_accept = 0.9 because 0.8 left a divergence in one of six test fits. The main MMM fit is the notebook’s longest cell.
• Module 10. The risk-averse plan (SLSQP maximizing the 10% quantile) barely moves from the mean-optimal plan, in a direction that depends on the draws; in one Docker run it stopped at its start, because the quantile is nearly flat around that plan. The notebook prints the iterations and the dollars moved: “it hardly moved” is the finding, not a failure. The CLV-weighted plan rests on made-up acquisition costs and is shown as a sensitivity only.
• Module 11. The Python install is the longer of the two, so learners start it first. The grf forests grow 1,000 trees rather than grf’s default 2,000, to keep each fit short on 2 CPUs. Hillstrom downloads as in Module 6.

The Extensions track: Modules 13 to 18

The Extensions are six optional modules beyond the core week: messy real data, survival analysis for churn, double machine learning, multi-touch attribution, an MMM whose effects drift, and push notifications as a policy over a quarter. They have no timetable. Participants can take them on their own after the week, or you can swap one into a custom schedule, in place of a core module or after the week. Each ends in its own decision; none feeds the capstone or the day pages’ results table.

Swapping one in

Each keeps the core module format, so it needs a core module’s slot: a briefing from its module page, a lab in one Python notebook with the same rhythm, harness and pacing rules, and a debrief whose third question is its decision.

Module	Builds on	Briefing / lab / debrief (minutes)
Module 13	Modules 1, 2 and 4 (Module 5 helps)	45 / 55 / 10
Module 14	Modules 4 and 6	45 / 55 / 10
Module 15	Modules 6 and 11	45 / 55 / 10
Module 16	Modules 6 and 8	45 / 55 / 10
Module 17	Modules 8 and 9	45 / 55 / 10
Module 18	Modules 6 and 11 (Module 5 helps)	45 / 55 / 10

• Teach it after the modules it builds on. Modules 15 and 18 need Module 11, so in the five-day order they can only follow Day 5.
• What to drop to make room is your call. Before dropping a core module, check which later modules list it under “Before you start” on their pages.
• Warm-ups and debriefs. The day pages cover the core modules only; for an extension, take questions from its knowledge checks.

Before you teach one

• Colab. The Colab buttons open these notebooks at the release the workshop is pinned to, v2026.10.3. The readiness page says whether each has a current Colab run.
• Times. The slowest cells below come from the laptop and Docker run records of 2026-10-10 (worked, FULL; an Apple Silicon laptop, natively and in the Docker image limited to 2 CPUs); they are not Colab times. On Colab (worked, FULL, 2026-10-11, at v2026.10.2) Run all took 38 s for Module 13, 39 s for 14, 161 s for 15, 13 s for 16 and 642 s for 17, which includes the two stretch fits (135 s and 95 s). The Colab times written in the Module 13 and 17 notebooks are estimates made before that sweep: Module 17’s static fit took 136 s (the notebook says about a minute and a half), its drifting fit 141 s (about three minutes) and its Prophet-baseline fit 84 s (about a minute); Module 13’s fit took 25 s (about a minute). Module 18’s times come from its records of 2026-10-16 (worked, FULL; the same laptop natively, and the Docker image at 2 CPUs): Run all took about 12 s natively and 14 s in Docker, and no cell took more than 1.8 s natively (3.0 s in Docker, the imports). Module 18 has no Colab time: its notebook is not in v2026.10.3, so it has no Colab button until the next release.
• No pilot yet. The minute marks are the lab briefs’ estimates; the stall points are what the reviews of the briefs and notebooks expect, not what a class has shown. Pilot with two or three people first.
• Solutions are folded under each exercise, as in the core. After use_reference(n), rerun that exercise’s checkpoint cell, then the cells below it: several checkpoints compute values that later cells use (flat_12 and boot in Module 14, mine in Module 15, P and removal in Module 16, effect, curve and uplift in Module 18), so skipping one ends in a NameError. The notebook’s message says the same.
• QUICK shortens a fit only in Module 17; the other labs take seconds either way (in Module 18 it keeps 5,000 of the 20,000 users, so intervals are about twice as wide and a choice near the guardrail can flip). Do not use QUICK in Module 17 on Colab at v2026.10.2: with QUICK on, Checkpoint 4 fails on the reference solution, because social’s band covers 48% of weeks (Colab, 2026-10-11) and the bar is 50%. The CI runner gives 49%; FULL settings pass on Colab, in Docker and on the laptop. If the room is short of time, use use_reference(n) instead.

Module 13 · Messy real data: cleaning a real log for BTYD and spend models

Module page · lab brief · knowledge checks

• Timing (minutes from the start of the lab): open 0–5; Exercises 1 to 5 end at 13, 23, 30, 37 and 45; Decision 45–50; slack 50–55. Switch at 25: anyone still on Exercise 2 runs workshop.use_reference(2). Still behind: everyone runs use_reference(3) and reads Exercise 3’s tables.
• Slowest cell: the PyMC-Marketing cross-check that opens Part B (2.8 s laptop, 10.5 s Docker; the notebook guesses a minute on Colab); learners write Exercise 3’s guesses meanwhile. D1 downloads Online Retail II on first use.
• Expected stalls. Exercise 1’s pattern sits in an f-string, so its braces are doubled, '^[0-9]{{5}}'. Exercise 2’s ASOF join is new SQL to most: point at the worked example above it. Part D’s counts check can fail after Checkpoints 1 and 2 passed; its message says what to rerun. Exercise 3’s change of tens of millions of percent is not a bug (the notebook says why). Exercise 4: steer anyone who picks “no cap” to the P99 row against the oracle on p, q and v.
• Solutions: folded under each exercise; none for the stretch.
• Debrief.
  1. Module 1’s rules moved the 52-week total by a few percent but hundreds of customers’ scores by more than 10%. Which matters for a forecast, and which for a contact list? Why give a customer’s change in money, not percent?
  2. On Online Retail II, why was the rank correlation of frequency and spend about twice the Pearson one, and what does a positive link do to the Gamma-Gamma’s estimate for a customer with few purchases?
  3. How many Online Retail II customers get a retention contact at a 10% lift, and at 5% and 20%? Which cleaning or counting choice moves the list most, and by how many customers? Would you spend next week on cleaning or on measuring the lift?
  Bridge: Module 14 turns to subscribers, whose churn is observed.

Module 14 · Survival analysis for subscription churn

Module page · lab brief · knowledge checks

• Timing: open 0–5; Exercises 1 to 5 end at 15, 22, 29, 37 and 45; Decision 45–50; slack 50–55. Room check at 15: if fewer than half have passed Checkpoint 1, show the at-risk line (a reversed cumulative sum) or have everyone run use_reference(1). Behind at 22: write the Exercise 2 guess, run use_reference(2), rerun Checkpoint 2 and read the years 8–12 table together. Keep Exercise 4 hands-on.
• Slowest cell: Checkpoint 5’s bootstrap (5.0 s laptop, 5.9 s Docker); learners read the Explain meanwhile.
• Expected stalls. Exercise 1: the number at risk, and ties (two cancellations at one time make one factor, 1 − 2/3); Checkpoint 1 names each mistake. Exercise 3: passing all of subs fails inside lifelines (TypeError: Cannot cast DatetimeArray to dtype float64) before any checkpoint message; the Task’s hint covers it. Listen for “33% less likely to churn”: a hazard ratio of 0.67 means “cancels at 67% of the rate”. Exercise 4: changing subs in place; the checkpoint says how to rebuild it.
• Solutions: folded under each exercise; none for the stretch.
• Debrief.
  1. Where did the true 12-month retention fall against the two dashboard numbers and your Kaplan-Meier interval? Which subscribers did each dashboard mishandle?
  2. The intro offer’s full-period hazard ratio was about 1.36. What did the landmark at month 3 show, and what would you write in the campaign report instead?
  3. Is an onboarding call worth its cost? Give the months onboarding adds within 24 for subscribers who did not onboard (standardized by plan, with its 95% bootstrap interval) and the share of called subscribers the call must get to onboard. What would the unadjusted comparison have said, and why is the estimate still an association?
  Bridge: onboarding was not randomized. Module 15 estimates causal effects from observational data; a randomized pilot (Module 6) settles it.

Module 15 · Causal effects without an experiment: double machine learning

Module page · lab brief · knowledge checks

• Timing: open 0–5; Exercises 1 to 5 end at 13, 21, 30, 37 and 45; Decision 45–50; slack 50–55. Call each mark; at 45, anyone still in Exercise 5 runs use_reference(5). If the room runs long, hand over Exercise 1 first: use_reference(1) after its Predict and plot.
• Slowest cell: the provided run without cross-fitting after Exercise 3 (12.6 s laptop with two threads, 12.9 s Docker). Each attempt at Checkpoint 5 refits ten models. Hillstrom downloads on first use. During the build, on a busy laptop without a thread limit, that run took minutes (not measured on a quiet one); if it does, restart Jupyter with OMP_NUM_THREADS=2.
• Expected stalls. Comparing the estimate with the ATE: the sentence before Checkpoint 3 names the overlap-weighted target. Exercise 2: column 1 of predict_proba. Exercise 3: no intercept; HC0, not HC1. Exercise 5: discrete_treatment=True, cv=folds, and W=W with X=None. The checkpoints name each. Exercise 4: reading the notebook’s LOW and HIGH instead of the arguments low and high; Checkpoint 4 names it. With QUICK the no-cross-fitting estimate is meaningless; its standard error is the lesson.
• Solutions: folded under each exercise and at the end of the stretch.
• Debrief.
  1. The naive comparison and a regression with all nine covariates both missed the truth by a wide margin. What did the regression miss that double machine learning caught?
  2. Your estimate weights customers by e(1 − e). Who are they (Exercise 4’s profile), and are they the customers a credit would reach?
  3. Would you pay customers the $2.50 credit for installing the app? Give the incremental margin per persuaded install with its 95% interval and its lower end against $2.50, and name the unobserved confounder that could overturn it. What did leaving out mobile_share do? QUICK and FULL runs can disagree: that is sample size.
  Bridge: Module 16 tests another observational claim made every day: the credit multi-touch attribution gives each channel.

Module 16 · Multi-touch attribution: Markov chains, Shapley values and incrementality

Module page · lab brief · knowledge checks

• Timing: open 0–5; Exercises 1 to 5 end at 13, 21, 29, 38 and 45; Decision 45–50; slack 50–55. Exercise 2, the transition matrix, is the likeliest to overrun: use_reference(2) at 21, and use_reference(3) for anyone behind at 29. Call the Decision at 45: it is the lab’s heaviest reading.
• Slowest cell: nothing to wait for (the bootstrap, 1.2 s on the laptop; the imports, 2.6 s in Docker).
• Expected stalls. Exercise 1: crediting journeys that did not convert; linear credit per touch, not per channel. Exercise 2: a table with every state, start, conv and null included (pd.crosstab also works). Exercise 3: deleting the channel and rescaling the other rows instead of sending its journeys to null, which gives a negative removal effect. Exercise 4: an off-by-one factorial in the weights. Exercise 5: counts instead of shares, or a share error without its factor 1/2. The checkpoints name each.
• Solutions: folded under each exercise and at the end of the stretch; the absorbing-chain solve is a provided cell.
• Debrief.
  1. How much credit did retargeting get under last-click, Markov and Shapley, and how many conversions did it cause? What made the data-driven methods agree with last-click?
  2. Your removal effects added up to about 2, not 1: what does dividing by their sum assume? Which split did you predict for B in the three-player game, and why does Shapley split a joint gain equally rather than in proportion?
  3. Would you move budget toward retargeting, the paid channel cheapest per credited conversion, or run a holdout first? Give its cost per credited conversion with the 95% bootstrap interval, the holdout’s minimum detectable effect against the break-even, the margin at risk, and the result that would justify the move.
  Bridge: attribution cannot tell a channel’s effect from the intent of the users it reaches; experiments (Modules 6 and 7) and lift-calibrated MMMs (Module 9) can. Module 17 returns to the MMM.

Module 17 · MMM with drifting effects and a Prophet baseline

Module page · lab brief · knowledge checks

• Timing: open 0–5; Exercises 1 to 5 end at 12, 20, 26, 35 and 45; Decision 45–50; slack 50–55. During each of the three fits, learners read the next Predict and start its function, as the notebook says; after each fit, run the cells below it in order. If the room runs long, everyone runs use_reference(3) after its Predict. QUICK (300 draws per chain instead of 1,000) must be on before the data cell, which sets the draws; switching it on later means rerunning from there, the static fit included.
• Slowest cells: the drifting fit (49.1 s on the laptop) and the static fit, the first of three (37.1 s in Docker); each record names only its slowest cell. The notebook guesses about three minutes on Colab for the drifting fit. In worked mode, Run all also runs the two stretch fits.
• Expected stalls. Exercise 1 (7 minutes) is tight for anyone new to xarray: select and sum by name, .sel(date=slice(start, end)) and .sum("date"); Checkpoint 1 names the usual mistakes. The static model’s very wide intervals for search and display are expected (the note after Exercise 4 says why). Exercise 5 is the longest function. With QUICK the verdict can come out “lift test first”: use it in the debrief.
• Solutions: folded under each exercise and under stretches A and B.
• Debrief.
  1. Social’s ROAS over all weeks and in the last quarter, by the truth, the static model and the drifting model: which interval contained the truth? Why did the static model give about the same number for both windows?
  2. Which channel’s correlation between residuals and spend changed sign between 2023 and 2025? Could you run that check on your own MMM tomorrow, without a truth, and what else could produce the same pattern?
  3. Cut, keep or lift-test social next quarter? Give the probability that its current-quarter ROAS is below break-even, how that differs from the all-period answer, and what would change it.
  Bridge: the Prophet baseline, fitted to the same sales, took credit from social and TV. A lift test run now (Module 9) checks the current level, and Module 10’s optimizer can take the current-quarter response curves. Module 18 leaves paid media for a channel the retailer owns and can fire every day.

Module 18 · Notifications: volume, targeting and a policy for the long run

Module page · lab brief · knowledge checks

• Timing: open 0–5; Exercises 1 to 5 end at 13, 20, 28, 37 and 45; Decision 45–50; slack 50–55. Nothing waits on compute, so the marks are about reading and writing. Exercise 4 (9 minutes) carries the most new ideas (contexts, the Normal draw, the penalty); the Decision table (fourteen policies) is the lab’s heaviest reading. Behind at 28: anyone still in Exercise 3 runs use_reference(3), reruns Checkpoint 3 and reads the channel table. At 45, call the Decision: everyone writes the sentence from the table before running the 180-day reveal cell below it. The reveal is the point of the Decision, and it only works if the sentence is written first.
• Slowest cells: nothing to wait for. On the laptop (2026-10-16, worked, FULL) the 180-day reveal took 1.8 s, the Decision’s evaluation of ten policies 1.8 s, the four bandit runs 1.5 s and the stretch’s three bandit runs 1.1 s; Run all took about 12 s. In the Docker image (2 CPUs) Run all took 14 s and the slowest cell was the imports (3.0 s). No Colab time exists yet (see Before you teach one).
• Expected stalls. Exercise 1: comparing all user-days instead of eligible ones (opted-out users sit in the not-sent group but could never be sent); Checkpoint 1 names it, and names ddof=1 and the 1.96. Exercise 2: open_rate as the mean of per-user rates instead of total opens over total sends, which weights a user with one send like one with sixty; Checkpoint 2 names it. Exercise 3: the rows with an incomplete 7-day window (margin_fwd NaN) must be dropped for every column, p_optout included; a learner who drops them for the effect but keeps them for the opt-out rate gets a p_optout that Checkpoint 3 rejects, with the rule in its message. Exercise 4: drawing from np.random instead of the rng argument (two seeds then give the same choices), or returning the draws rather than argmax(axis=1); Checkpoint 4 names both. Exercise 5: one sweep from zeros instead of n_iter, or returning R; Checkpoint 5 names both. With QUICK (5,000 of the 20,000 users) the intervals are about twice as wide and the choice can flip near the guardrail: in the FULL run the chosen policy loses 8.5% of the base against a 10% guardrail.
• Solutions: folded under each exercise and at the end of the stretch.
• Debrief.
  1. The openers-versus-non-openers gap was about $1.81 of same-day margin; the experiment’s estimate was $0.14 (95% interval $0.13 to $0.15) against a true $0.14. What did the gap measure? Then the volume curve: daily sends won week-1 opens (0.42 per user) and three a week won the quarter ($21.88 per user against $17.93 daily, which lost 46% of its users). Which two mechanisms, both absent on day 1, separate the two rankings?
  2. With penalty 0 the bandit sent 61 notifications per user and lost 41% of the base; with penalty 10 V, 34 per user and 8.5%. What did the penalty change in what the bandit sees, and what does it still not see about a send? The value-iteration policies sent fewer than three a week, lost under 9% and earned less than three a week: which approximation in the model explains it?
  3. Which policy next quarter? Within the 10% guardrail, the bandit with penalty 10 V: $21,964 per 1,000 users and 8.5% lost, +$865 against three a week (95% paired interval +$730 to +$999). The reveal: over 180 days the 3 V bandit still earned the most ($44,401 per 1,000 against $43,567) and had lost 19.7% of the base. Does that overturn the guardrail, and who should decide where it sits?
  Bridge: this is the last extension. The simulation has no real tolerance curve: on a real app the opt-out rates, the habit effect and the value at stake come from the two experiments this lab runs (a randomized send experiment, as in Module 6, and a volume experiment over a quarter), and the bandit’s penalty and the guardrail are decisions, not parameters the data return.

Before the workshop

• Three weeks before, check that repo.ref in _variables.yml is the release tag you will teach from; the Colab badges and installs point at it.
• Within 21 days of Day 1, run every notebook on a fresh Colab runtime in worked mode, FULL, whole notebook. Record each run with scripts/add_run_record.py (its --help lists the arguments) so the readiness page shows it.
• Check the readiness page. Every notebook should be ready to teach: a current Colab run, CI passing on the current commit. Rerun anything marked stale (code or dependencies changed since its run, or older than the age limit). To check ahead for a class date, run the release workflow with that date; it also runs on the 1st of each month.
• In the week before Day 1, check the data URLs: run the link check by hand (Run workflow), which downloads every dataset source and checks its hash, or run each lab’s data-loading cells once. The monthly run can be weeks old.
• A week before, send participants the Before Day 1 page: entry check, review, setup, Module 0.
• The day before, remind them to check their setup and to send any error they cannot fix.
• Prepare: the GeoLift demo output (Day 3), printed CFO cards and Analyst checklists (Day 2), the capstone pairs and answer key (Day 5).
• If you swap in an extension, read Before you teach one first, and check the extension’s line on the readiness page; Module 18 has not run on Colab yet.
• The room: Wi-Fi that can take the whole room installing packages at once at the start of each lab, and power at every table.
• First delivery only: pilot each lab with two or three people and note where they stall. A run record measures compute time, not how long a person takes.

Reuse and license

The teaching text, slides, figures and notebook Markdown are licensed CC BY 4.0; the code is licensed MIT. You may teach, adapt and share the workshop with attribution. Datasets keep their own licenses; the References page lists their sources. The source is on GitHub.