AI Workflows for Quarto

Promotion experiments and sales outcomes

Ravi Kalia

Workflow Inputs and Outputs

Input: Hillstrom’s randomized email experiment in context/.

File What the agent reads
hillstrom.csv 64,000 customers, assignment and two-week outcomes
notes.md Design, data dictionary, provenance, limitations
sources.bib Experiment and documentation references

Output: one verified analysis in three report formats: report.pdf, report.docx, and report.ipynb. Also deliver slides.html, exactly 8 slides including the title.

Decision: which campaign increases sales per assigned customer? Keep report.qmd, slides.qmd, unchanged inputs, and build instructions.

Learning Outcomes

Use a continuing coding agent conversation to analyze a real promotion experiment and communicate a campaign recommendation.

  • Define a sales outcome and a randomized comparison.
  • Delegate implementation, dependency setup, and rendering.
  • Produce PDF, DOCX, and IPYNB reports plus eight slides.
  • Audit numerical claims, causal reasoning, and visual presentation.

Return to Quarto foundations

A 90-Minute Marketing Lab

Activity Time Observable result
Orientation and agent session 10 min Inputs copied, agent running
Turns 1–2: evidence and brief 15 min brief.md: question and outcomes
Turn 3: build and export 20 min report.pdf, .docx, runnable .ipynb
Turns 4–5: audit claim and figure 15 min One corrected claim, one better figure
Turns 6–8: slides and review 15 min Eight-slide slides.html, checks
Partner audit 15 min A verified correction or passed check

🏠 before the lab: install one coding agent · ⭐ = extension, self-paced · Prerequisites: basic Markdown and Git; means and percentages · Audience: graduate students in marketing, business analytics, or research methods

The Case: Which Promotion Should We Send?

Kevin Hillstrom published a 64,000-customer randomized email experiment in 2008. Customers had purchased within the previous year.

Assigned campaign Outcomes over the next two weeks
Men’s merchandise email Website visit, purchase, dollars spent
Women’s merchandise email The same outcomes
No email The same outcomes; control group

Classroom decision: advise a marketing manager on the next email campaign.

Original experiment and challenge.

One Request Can Cover Several Steps

One request could say (an example: the lab splits it into eight turns):

Read the inputs in context/. Build an executable Quarto report for a marketing manager. Compare each email campaign with no email on two-week sales per assigned customer and purchase conversion. Explain the experiment and limitations. Export report.pdf, report.docx, and report.ipynb from report.qmd. Create exactly 8 slides including the title. Render, fix failures, and report checks. Preserve the raw inputs.

The coding agent can inspect files, write source, prepare dependencies, and run Quarto within its tools and permissions.

Your job: evaluate the outcome definition and campaign recommendation.

The Lab in Eight Turns

One conversation, eight reviewable requests. Each turn ends with a check you can see.

  1. Understand the evidence
  2. Write the brief
  3. Create the report
  4. Correct an overclaim
  1. Improve the figure
  2. Adapt the report into slides
  3. Fix a layout defect
  4. Review the whole deliverable

Turns 1 and 2 define the question. Turns 3 to 5 build and audit the report. Turns 6 to 8 produce and review the slides.

Installation

Choose One Coding Agent

🏠 before the lab

The workshop deck lists the installers for Claude Code, Codex and Antigravity: install a coding agent. Each asks you to sign in on first start.

Start claude, codex or agy inside the lab folder. Use the same plain-English task prompts with any of them.

Check: confirm the session’s working folder and record the tool version.

The Directory Method

A Workspace for One Question

Question: how much additional sales does each email generate compared with sending no email?

~/quarto-promotions/
  context/
    hillstrom.csv       # original 64,000 customer rows
    notes.md            # design, dictionary, provenance
    sources.bib         # experiment and Quarto references
  brief.md              # current decision and deliverables
  AGENTS.md             # recurring project conventions

The unit of analysis is one assigned customer. A directory organizes context; tool permissions govern access.

Copying the Lab Inputs

mkdir -p ~/quarto-promotions/context && cd ~/quarto-promotions
cp ~/quarto-workshop/examples/marketing-context/* context/
claude

Uses the clone from the workshop’s setup check (~/quarto-workshop) · or codex / agy instead of claude

Exercise · ⏱️ 5 min

context/ holds hillstrom.csv, notes.md and sources.bib, and the agent reports quarto-promotions as its working folder.

Turn 1: Understand the Evidence

Read every file in context/. Summarize assignment, row meaning, outcomes, units, and observation window. Identify pre-campaign attributes and missing business information. Check the source citation before proposing an analysis.

Exercise · ⏱️ 5 min

the agent’s summary agrees with the files:

  • segment is randomized campaign assignment; groups are independent.
  • spend is dollars per customer over two weeks, including zeros.
  • conversion marks any purchase; visit marks a website visit.
  • Costs, margin, delivery/open status, and later sales are unavailable.

Define Success Before Looking for a Winner

Primary outcome: mean two-week sales per assigned customer, in dollars.

Secondary outcomes: purchase conversion and website visit rates.

For campaign \(a\) and no-email control \(0\):

\[\widehat{\Delta}_a = \overline{Y}_a - \overline{Y}_0\]

\(Y_i\) is customer \(i\)’s two-week spending, including zero purchases. Positive \(\widehat{\Delta}_a\) estimates additional sales per assigned customer.

Check: the denominator includes every customer assigned to that arm.

Turn 2: Have the Agent Write the Brief

Write brief.md using the three context files. Require report.pdf (Typst), report.docx, and report.ipynb from report.qmd, plus exactly 8 slides including the title. Compare each email with no email on sales/customer and conversion in percentage points. Include uncertainty, provenance, business limitations, and completion checks. Label planning assumptions.

Your addition: a criterion that catches a misleading claim, such as treating all campaign revenue as incremental sales.

Exercise · ⏱️ 5 min

you have reviewed the question and outcomes in brief.md, and it contains your criterion.

Durable Instructions

Initialize Git if needed and work on a draft branch. Create AGENTS.md with rules to preserve raw inputs, compute claims, verify citations, keep execution enabled, and inspect rendered outputs. Connect these rules to the instruction file your tool reads. Record the build commands.

brief.md describes this decision. AGENTS.md holds recurring project rules.

Exercise · ⏱️ 3 min

the agent shows that it reads the project’s instructions.

Building and Revising

Turn 3: Create the Report

Implement brief.md as report.qmd (formats typst and docx, bibliography context/sources.bib). Read context/hillstrom.csv in Python and validate rows, assignments, and outcomes. Compute arm sizes, sales means, conversion, and contrasts with control. State uncertainty assumptions. Add labeled figures, captions, alt text, and citations. Prepare dependencies and record build commands.

writes report.qmd · installs what Python needs · renders it

Exercise · ⏱️ 8 min

report.qmd reads the CSV and computes every number it reports. The next slides export and inspect it.

One Source, Three Report Formats

In report.qmd, the coding agent selects PDF (via Typst) and Word:

format:
  typst: default        # writes report.pdf
  docx: default
bibliography: context/sources.bib
jupyter: python3

Typst is bundled with Quarto: no LaTeX install, the same engine as the workshop’s PDFs. The notebook conversion preserves Markdown and executable Python cells. Ask the agent to include readable reference links in the notebook.

Render the Three Formats

Export report.pdf and report.docx from report.qmd, then convert it to report.ipynb and execute the notebook from the lab root. Show each command and fix any errors.
quarto render report.qmd --to typst      # report.pdf
quarto render report.qmd --to docx
quarto convert report.qmd --output report.ipynb
jupyter nbconvert --to notebook --execute --inplace report.ipynb

Exercise · ⏱️ 3 min

report.pdf, report.docx and report.ipynb exist and every command finished without errors.

Inspect Each Format

📕

PDF: page breaks · captions · numbered references at the end

📄

DOCX: tables · figures · citations, in Word or LibreOffice

📓

IPYNB: code cells, not Markdown · saved results · reference links

open report.pdf and report.docx · open report.ipynb: restart the kernel, run all cells from the lab root

Exercise · ⏱️ 4 min

each format passes its check, and a fresh-kernel run reproduces the notebook’s results.

What to Inspect in the Source

⭐ self-paced

Show where report.qmd reads the raw CSV, computes each numerical claim, and sets execution options. Explain these parts briefly.

Expected configuration pattern:

execute:
  echo: true
  error: false
  cache: false
  freeze: false

Check: errors stop the build; results recompute from local inputs.

A Calculation to Inspect

⭐ self-paced

One possible cell in the student’s report:

```{python}
import pandas as pd

df = pd.read_csv("context/hillstrom.csv")
summary = df.groupby("segment").agg(
    n=("spend", "size"), sales=("spend", "mean"),
    conversion=("conversion", "mean"))
summary["incremental_sales"] = (
    summary["sales"] - summary.loc["No E-Mail", "sales"])
print(summary)
```

Check: use every assigned customer, including nonbuyers. Conversion differences become percentage points when multiplied by 100.

An Independent Numerical Check

Two-week outcomes computed from the original customer data:

Assigned arm Customers Sales/customer Conversion
No E-Mail 21,306 $0.653 0.573%
Mens E-Mail 21,307 $1.423 1.253%
Womens E-Mail 21,387 $1.077 0.884%

Check: calculate each campaign’s mean minus the no-email mean. Rounded display values can differ slightly from full-precision calculations.

Additional Sales Versus No Email

Campaign Added sales/customer 95% interval Conversion change
Mens E-Mail +$0.77 $0.49 to $1.05 +0.68 pp
Womens E-Mail +$0.42 $0.17 to $0.68 +0.31 pp

Men’s email minus women’s email: +$0.35 per assigned customer (95% interval $0.03 to $0.66).

pp = percentage points. The head-to-head interval barely excludes zero: the ranking is less certain than the point estimates suggest.

Check: the entire treatment-arm sales mean includes purchases that might have happened without an email.

See the Sales Effect

Men's merchandise email adds about 0.77 dollars per assigned customer, interval roughly 0.49 to 1.06; women's merchandise email adds about 0.42 dollars, interval roughly 0.17 to 0.68. Both intervals exclude zero and overlap each other.
Figure 1: Hillstrom (2008): two-week sales gains versus no email, with 95% intervals.

Check: both intervals exclude zero, and they overlap. Overlap does not settle the ranking; the direct contrast on the previous slide does.

Turn 4: Correct a Marketing Overclaim

Suppose the draft says: “Every dollar of email-arm revenue was caused by the promotion.”

Check that claim against the no-email control. Replace it with the computed difference in sales per assigned customer. Audit denominators and distinguish percentage points from percent change. Re-render and show the values that support the corrected interpretation.

Exercise · ⏱️ 5 min

observed sales and additional sales are labeled separately, with values recomputed from the CSV.

A Tempting but Misleading Winner

Compare average spending among buyers with sales per assigned customer. Explain why the buyer-only ranking cannot identify the campaign with the largest causal sales effect. Keep the primary analysis on all assigned customers.

Check: purchasing is an outcome of the promotion. Selecting only buyers breaks the original randomized comparison.

Exercise · ⏱️ 3 min

you can explain why more visits, more purchases, and more revenue are different outcomes.

Turn 5: Improve the Figure

If the figure shows only total campaign revenue:

Plot each campaign's additional sales per assigned customer versus no email. Show an uncertainty interval and a zero reference line. Label the two-week window and dollars/customer. Put conversion differences in a separate panel with percentage-point units. Update the caption and alt text, then render again.

Check: arm size does not determine the ranking; units and reference are explicit.

Exercise · ⏱️ 5 min

your one visual revision rendered and the underlying estimates did not change.

What Does Uncertainty Mean Here?

⭐ self-paced

Explain the assumptions behind the intervals. For independent arms, use customer-level sampling and preserve assignment groups. Compare men's and women's emails directly before declaring a winner. State how multiple comparisons and skewed spending affect interpretation.

Check: most customers spend zero; a few spend much more. Intervals describe estimation uncertainty under stated assumptions.

From a Sales Effect to a Business Decision

⭐ self-paced

Draft a campaign recommendation supported by the experiment. State which assumptions would be needed to project sales to a future audience. List the margin, campaign cost, and later outcomes needed to assess profit. Propose a follow-up randomized test.

Check: a recommendation distinguishes measured sales from projected sales and profit. Any new audience size or cost is a labeled planning assumption.

Exercise · ⭐ self-paced

you have chosen one unresolved business question and defined an outcome for the next test.

Turn 6: Adapt the Report into Slides

Create slides.qmd with exactly 8 slides including the title. Reuse the report's verified computation. Cover the decision, experiment, outcome definitions, sales results, uncertainty, recommendation, and follow-up test. Use a dark Reveal.js theme and speaker notes. Render and inspect every slide.

writes slides.qmd · quarto render slides.qmd --to revealjs → slides.html

Exercise · ⏱️ 6 min

slides.html has eight slides, title included, and every number matches the report.

Turn 7: Fix an Observed Layout Defect

If a figure and explanation are crowded:

On the "Sales Results" slide, the caption falls below the visible area. Move the detailed interpretation to speaker notes and shorten the caption. Keep the figure readable and the total at 8 slides. Re-render at desktop and laptop presentation sizes.

no browser tool in the agent? open slides.html yourself and describe the defect · ask which checks are done and which are still needed

Exercise · ⏱️ 3 min

the whole caption fits, and the notes keep the necessary interpretation.

Turn 8: Review the Whole Deliverable

Rebuild all three report formats and the slides. Run the notebook from a fresh kernel in the lab root. Recompute claims from the CSV. Audit citations, figures, denominators, and business claims. Review the diff and report commands, limitations, and files ready for review. Keep this work local.
quarto render slides.qmd --to revealjs   # after the four report exports
git diff --check

Exercise · ⏱️ 5 min

source, unchanged inputs, build instructions, and all four outputs are present and checked.

Adapting the Workflow

Optional: A Campaign Evidence Website

⭐ self-paced From a fresh workspace with the report and context:

Create a Quarto website in `campaign-site` with Decision, Experiment, and Results pages. Adapt the verified report and copy its inputs. Preserve the outcome definitions and limitations. Render the site and check all local links and its desktop layout.

quarto create project website campaign-site --no-open --no-prompt

navigation works and results recompute from the copied inputs.

Optional: A Marketing Analytics Blog

⭐ self-paced In another fresh workspace:

Create a Quarto blog in `marketing-blog`. Adapt the report into posts/promotion-sales/index.qmd with title, date, categories, executable analysis, experiment citation, and limitations. Copy the inputs and fix relative paths. Render and check the listing's post link.

quarto create project blog marketing-blog --no-open --no-prompt

data paths work from the post’s new location.

Different Outputs Need Different Reviews

Output Ask the coding agent to change Verify yourself
PDF / DOCX Develop experiment and results Page layout, figures, references
IPYNB Retain narrative and executable cells Run all cells and compare results
Slides Shorten visible explanation Readability and recommendation
Website Link decision and evidence pages Navigation and desktop layout
Blog Make a self-contained case study Provenance, date, and listing

Keep the computation consistent while adapting the explanation.

Audit and Collaboration

Requests for a Larger Team

⭐ self-paced For a larger project, you can ask for bounded roles:

Have an evidence reviewer inspect sources read-only, an author edit only report.qmd, and a render reviewer report visual defects. One lead should integrate changes and run the final build.
Role Required return
Evidence reviewer Supported claims and source locations
Author Draft, assumptions, and checks performed
Render reviewer Defects with slide IDs and viewport

Use your tool’s delegation support, or perform these stages sequentially.

Partner Review

Exercise · ⏱️ 12 min

Exchange all three report formats and slides. Each partner checks:

  1. One additional-sales estimate against the CSV.
  2. One causal or business assumption against the notes.
  3. One figure or slide in the browser.

Use the feedback for a plain-English revision request.

you can show the defect, the request, and evidence of correction; or a check that passed.

Lab Submission

Submit report.pdf, report.docx, report.ipynb, and the eight-slide slides.html, plus both .qmd sources, unchanged input copies, and dependency/build instructions.

Record Example
Request “Check which revenue is incremental.”
Defect found Total email-arm sales described as additional sales
Verification Recomputed arm-minus-control means and checked prose

If no defect is found, report a check that passed. Do not invent an agent mistake.

Evaluation Rubric

Dimension Evidence of success
Reproducibility A partner rebuilds all formats and reruns the notebook
Experimental reasoning Assignment, outcomes, units, and assumptions are explicit
Provenance Claims trace to customer data or verified sources
Communication Figures and slides support the campaign decision
Agent supervision Follow-ups address observed issues

Which business conclusions could a successful build never validate?

References and Questions

Case and installation sources checked 2026-10-06. Word and notebook documentation checked 2026-10-07; Typst checked 2026-10-30.

Back to Quarto foundations