Day 5
Agents and the Capstone
How do we build reliable systems out of these models?
Day 5 builds systems. Module 14 turns the hand-written tool loop of Module 8 into an agent graph with state, human approval and confidence-gated control. The capstone takes the rest of the day: in pairs, you improve one component of a research assistant and measure whether the system got better. The wrap-up closes the workshop.
Warm-Up: What Carried Over (09:00–09:15)
Answer alone on paper for 6 minutes, notes closed; compare with a neighbor for 4; the facilitator takes the two most-missed questions for 5. Each link opens the question with its folded answer. Three questions come from Day 4 and two from Day 3.
| Question | Topic | From | Why today |
|---|---|---|---|
| 12.2 | Act, ask and escalate thresholds | Day 4 · Module 12 | Module 14’s guard uses these two thresholds |
| 11.2 | Over- or underconfident; ECE on 100 cases | Day 4 · Module 11 | Lab 14 and the capstone report every rate with its denominator |
| 13.3 | Faithful and wrong | Day 4 · Module 13 | The capstone’s “supported by the sources?” check cannot catch this case |
| 8.1a | Who runs the tool | Day 3 · Module 8 | Module 14 turns this loop into an explicit graph |
| 8.2 | Invalid counts as wrong; a standard error on every rate | Day 3 · Module 8 | The capstone compares your change with the baseline on the same questions |
09:15–11:15
Module 14
Tool-using agents as explicit graphs in LangGraph, with state, human-in-the-loop interrupts and confidence-gated control.
Open the briefing Open in Colab LangChain LangGraph Jev
11:30–12:25
13:25–14:30
14:45–16:45
Module 15
Combine retrieval, an agent graph and calibrated control into one system, evaluate it, and explain the design.
Open the briefing Open in Colab LlamaIndex LangGraph Jev OpenAI Anthropic
Run Sheet
09:15–11:15 · Module 14 · Agents
09:15 briefing · 10:10 lab · 11:05 debrief
- In the room: the live plan, 45 minutes of exposition and 10 of checks, demos and predictions.
- Lab: open in Colab, a checkpoint after each exercise; about 6 minutes of compute (Google Colab, free T4 GPU, planned).
- When something goes wrong: stuck on exercise N, run its folded Solution cell, then
workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.
11:30–12:25 · 13:25–14:30 · 14:45–16:45 · Module 15 · Capstone
- In the room: hands-on, in pairs (the plan).
- Lab: open in Colab, a checkpoint after each exercise; about 15 minutes of compute (Google Colab, free T4 GPU, planned).
- When something goes wrong: stuck on exercise N, run its folded Solution cell, then
workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.
Compute times are planning estimates, not measurements on the teaching runtime; the readiness page has what has been run, where.
Day Wrap-Up (16:45–17:00)
The last 15 minutes of the capstone wrap-up (Module 15, Part IV), and the close of the workshop. The whole room, with the boards from Days 2 to 4. The numbers are the room’s own, from its runs.
The whole line. Complete the table in Five Days, One Line of Ideas: add what Modules 14 and 15 fixed and left open, and set the room’s numbers from the boards beside the build’s.
Today’s table. Fill in the room’s results. Write the path beside each number, as the notebook’s last cell prints it (real open models, commercial APIs, or offline stand-ins, which measure no model); the results worksheet has a table per comparison group.
Lab Rows Metric Lab 14, decision set and the injection test the router; the guard route accuracy; unsafe-action rate and injection success rate, each with its denominator Lab 15, the capstone’s testsplitthe baseline; each pair’s final system accuracy, both abstention rates, cost per question; questions gained and lost, with the sign test’s p What is still open. Each pair names one problem the workshop left open (see Open Problems) and the measurement from the last five days it would start with.
Times are on the schedule.