Promotion experiments and sales outcomes
Input: Hillstrom’s randomized email experiment in context/.
| File | What the agent reads |
|---|---|
hillstrom.csv |
64,000 customers, assignment and two-week outcomes |
notes.md |
Design, data dictionary, provenance, limitations |
sources.bib |
Experiment and documentation references |
Output: one verified analysis in three report formats: report.pdf, report.docx, and report.ipynb. Also deliver slides.html, exactly 8 slides including the title.
Decision: which campaign increases sales per assigned customer? Keep report.qmd, slides.qmd, unchanged inputs, and build instructions.
Use a continuing coding agent conversation to analyze a real promotion experiment and communicate a campaign recommendation.
| Activity | Time | Observable result |
|---|---|---|
| Orientation and agent session | 10 min | Inputs copied, agent running |
| Turns 1–2: evidence and brief | 15 min | brief.md: question and outcomes |
| Turn 3: build and export | 20 min | report.pdf, .docx, runnable .ipynb |
| Turns 4–5: audit claim and figure | 15 min | One corrected claim, one better figure |
| Turns 6–8: slides and review | 15 min | Eight-slide slides.html, checks |
| Partner audit | 15 min | A verified correction or passed check |
🏠 before the lab: install one coding agent · ⭐ = extension, self-paced · Prerequisites: basic Markdown and Git; means and percentages · Audience: graduate students in marketing, business analytics, or research methods
Kevin Hillstrom published a 64,000-customer randomized email experiment in 2008. Customers had purchased within the previous year.
| Assigned campaign | Outcomes over the next two weeks |
|---|---|
| Men’s merchandise email | Website visit, purchase, dollars spent |
| Women’s merchandise email | The same outcomes |
| No email | The same outcomes; control group |
Classroom decision: advise a marketing manager on the next email campaign.
One request could say (an example: the lab splits it into eight turns):
Read the inputs in context/. Build an executable Quarto report for a marketing manager. Compare each email campaign with no email on two-week sales per assigned customer and purchase conversion. Explain the experiment and limitations. Export report.pdf, report.docx, and report.ipynb from report.qmd. Create exactly 8 slides including the title. Render, fix failures, and report checks. Preserve the raw inputs.
The coding agent can inspect files, write source, prepare dependencies, and run Quarto within its tools and permissions.
Your job: evaluate the outcome definition and campaign recommendation.
One conversation, eight reviewable requests. Each turn ends with a check you can see.
Turns 1 and 2 define the question. Turns 3 to 5 build and audit the report. Turns 6 to 8 produce and review the slides.
🏠 before the lab
The workshop deck lists the installers for Claude Code, Codex and Antigravity: install a coding agent. Each asks you to sign in on first start.
Start claude, codex or agy inside the lab folder. Use the same plain-English task prompts with any of them.
Check: confirm the session’s working folder and record the tool version.
Question: how much additional sales does each email generate compared with sending no email?
~/quarto-promotions/
context/
hillstrom.csv # original 64,000 customer rows
notes.md # design, dictionary, provenance
sources.bib # experiment and Quarto references
brief.md # current decision and deliverables
AGENTS.md # recurring project conventions
The unit of analysis is one assigned customer. A directory organizes context; tool permissions govern access.
Uses the clone from the workshop’s setup check (~/quarto-workshop) · or codex / agy instead of claude
context/ holds hillstrom.csv, notes.md and sources.bib, and the agent reports quarto-promotions as its working folder.
the agent’s summary agrees with the files:
segment is randomized campaign assignment; groups are independent.spend is dollars per customer over two weeks, including zeros.conversion marks any purchase; visit marks a website visit.Primary outcome: mean two-week sales per assigned customer, in dollars.
Secondary outcomes: purchase conversion and website visit rates.
For campaign \(a\) and no-email control \(0\):
\[\widehat{\Delta}_a = \overline{Y}_a - \overline{Y}_0\]
\(Y_i\) is customer \(i\)’s two-week spending, including zero purchases. Positive \(\widehat{\Delta}_a\) estimates additional sales per assigned customer.
Check: the denominator includes every customer assigned to that arm.
Write brief.md using the three context files. Require report.pdf (Typst), report.docx, and report.ipynb from report.qmd, plus exactly 8 slides including the title. Compare each email with no email on sales/customer and conversion in percentage points. Include uncertainty, provenance, business limitations, and completion checks. Label planning assumptions.Your addition: a criterion that catches a misleading claim, such as treating all campaign revenue as incremental sales.
you have reviewed the question and outcomes in brief.md, and it contains your criterion.
brief.md describes this decision. AGENTS.md holds recurring project rules.
the agent shows that it reads the project’s instructions.
Implement brief.md as report.qmd (formats typst and docx, bibliography context/sources.bib). Read context/hillstrom.csv in Python and validate rows, assignments, and outcomes. Compute arm sizes, sales means, conversion, and contrasts with control. State uncertainty assumptions. Add labeled figures, captions, alt text, and citations. Prepare dependencies and record build commands.writes report.qmd · installs what Python needs · renders it
report.qmd reads the CSV and computes every number it reports. The next slides export and inspect it.
In report.qmd, the coding agent selects PDF (via Typst) and Word:
Typst is bundled with Quarto: no LaTeX install, the same engine as the workshop’s PDFs. The notebook conversion preserves Markdown and executable Python cells. Ask the agent to include readable reference links in the notebook.
report.pdf, report.docx and report.ipynb exist and every command finished without errors.
📕
PDF: page breaks · captions · numbered references at the end
📄
DOCX: tables · figures · citations, in Word or LibreOffice
📓
IPYNB: code cells, not Markdown · saved results · reference links
open report.pdf and report.docx · open report.ipynb: restart the kernel, run all cells from the lab root
each format passes its check, and a fresh-kernel run reproduces the notebook’s results.
⭐ self-paced
Expected configuration pattern:
Check: errors stop the build; results recompute from local inputs.
⭐ self-paced
One possible cell in the student’s report:
Check: use every assigned customer, including nonbuyers. Conversion differences become percentage points when multiplied by 100.
Two-week outcomes computed from the original customer data:
| Assigned arm | Customers | Sales/customer | Conversion |
|---|---|---|---|
| No E-Mail | 21,306 | $0.653 | 0.573% |
| Mens E-Mail | 21,307 | $1.423 | 1.253% |
| Womens E-Mail | 21,387 | $1.077 | 0.884% |
Check: calculate each campaign’s mean minus the no-email mean. Rounded display values can differ slightly from full-precision calculations.
| Campaign | Added sales/customer | 95% interval | Conversion change |
|---|---|---|---|
| Mens E-Mail | +$0.77 | $0.49 to $1.05 | +0.68 pp |
| Womens E-Mail | +$0.42 | $0.17 to $0.68 | +0.31 pp |
Men’s email minus women’s email: +$0.35 per assigned customer (95% interval $0.03 to $0.66).
pp = percentage points. The head-to-head interval barely excludes zero: the ranking is less certain than the point estimates suggest.
Check: the entire treatment-arm sales mean includes purchases that might have happened without an email.
Check: both intervals exclude zero, and they overlap. Overlap does not settle the ranking; the direct contrast on the previous slide does.
Suppose the draft says: “Every dollar of email-arm revenue was caused by the promotion.”
observed sales and additional sales are labeled separately, with values recomputed from the CSV.
Check: purchasing is an outcome of the promotion. Selecting only buyers breaks the original randomized comparison.
you can explain why more visits, more purchases, and more revenue are different outcomes.
If the figure shows only total campaign revenue:
Plot each campaign's additional sales per assigned customer versus no email. Show an uncertainty interval and a zero reference line. Label the two-week window and dollars/customer. Put conversion differences in a separate panel with percentage-point units. Update the caption and alt text, then render again.Check: arm size does not determine the ranking; units and reference are explicit.
your one visual revision rendered and the underlying estimates did not change.
⭐ self-paced
Check: most customers spend zero; a few spend much more. Intervals describe estimation uncertainty under stated assumptions.
⭐ self-paced
Check: a recommendation distinguishes measured sales from projected sales and profit. Any new audience size or cost is a labeled planning assumption.
you have chosen one unresolved business question and defined an outcome for the next test.
writes slides.qmd · quarto render slides.qmd --to revealjs → slides.html
slides.html has eight slides, title included, and every number matches the report.
If a figure and explanation are crowded:
no browser tool in the agent? open slides.html yourself and describe the defect · ask which checks are done and which are still needed
the whole caption fits, and the notes keep the necessary interpretation.
source, unchanged inputs, build instructions, and all four outputs are present and checked.
⭐ self-paced From a fresh workspace with the report and context:
quarto create project website campaign-site --no-open --no-prompt
navigation works and results recompute from the copied inputs.
⭐ self-paced In another fresh workspace:
quarto create project blog marketing-blog --no-open --no-prompt
data paths work from the post’s new location.
| Output | Ask the coding agent to change | Verify yourself |
|---|---|---|
| PDF / DOCX | Develop experiment and results | Page layout, figures, references |
| IPYNB | Retain narrative and executable cells | Run all cells and compare results |
| Slides | Shorten visible explanation | Readability and recommendation |
| Website | Link decision and evidence pages | Navigation and desktop layout |
| Blog | Make a self-contained case study | Provenance, date, and listing |
Keep the computation consistent while adapting the explanation.
⭐ self-paced For a larger project, you can ask for bounded roles:
| Role | Required return |
|---|---|
| Evidence reviewer | Supported claims and source locations |
| Author | Draft, assumptions, and checks performed |
| Render reviewer | Defects with slide IDs and viewport |
Use your tool’s delegation support, or perform these stages sequentially.
Exchange all three report formats and slides. Each partner checks:
Use the feedback for a plain-English revision request.
you can show the defect, the request, and evidence of correction; or a check that passed.
Submit report.pdf, report.docx, report.ipynb, and the eight-slide slides.html, plus both .qmd sources, unchanged input copies, and dependency/build instructions.
| Record | Example |
|---|---|
| Request | “Check which revenue is incremental.” |
| Defect found | Total email-arm sales described as additional sales |
| Verification | Recomputed arm-minus-control means and checked prose |
If no defect is found, report a check that passed. Do not invent an agent mistake.
| Dimension | Evidence of success |
|---|---|
| Reproducibility | A partner rebuilds all formats and reruns the notebook |
| Experimental reasoning | Assignment, outcomes, units, and assumptions are explicit |
| Provenance | Claims trace to customer data or verified sources |
| Communication | Figures and slides support the campaign decision |
| Agent supervision | Follow-ups address observed issues |
Which business conclusions could a successful build never validate?
Case and installation sources checked 2026-10-06. Word and notebook documentation checked 2026-10-07; Typst checked 2026-10-30.
Quarto Workshop