Day 4

Calibration, Decisions and Retrieval

When should a model’s answer be trusted, and how do we ground it in sources?

Day 4 asks when a model’s answer can be trusted. Calibration measures whether a model’s probabilities mean what they say; Module 12 turns calibrated probabilities into act, ask or escalate decisions, and asks what changes when calibration itself is the training objective: TypeSafe names this RLCD but has not published the method, so the module keeps what TypeSafe has stated apart from third-party reports and from our own illustration. Module 13 grounds answers in retrieved sources and measures retrieval and answers separately.

Warm-Up: What Carried Over (09:00–09:15)

Answer alone on paper for 6 minutes, notes closed; compare with a neighbor for 4; the facilitator takes the two most-missed questions for 5. Each link opens the question with its folded answer. Three questions come from Day 3, one from Day 2 and one from Day 1.

Question Topic From Why today
8.1b What a valid schema guarantees Day 3 · Module 8 Form, not truth: Module 11 asks what a probability should guarantee
9.3 Bradley–Terry and its ceiling Day 3 · Module 9 Module 12 sets this preference reward beside an outcome reward
10.4 Reward hacking at \beta = 0 Day 3 · Module 10 A learned reward cannot see its own errors; verified labels can
6.3 The loss before fine-tuning Day 2 · Module 6 \ln 4 is a log loss, one of Module 11’s proper scoring rules
1.3 Which classifier is overconfident Day 1 · Module 1 Lab 11 measures the calibration of a classifier you have already trained

09:15–11:15

Module 11

Calibration

What a probability should mean, how to measure it, and how to use confidence to decide when to abstain.

Open the briefing Open in Colab PyTorch scikit-learn OpenAI Anthropic

11:30–12:25
13:25–14:30

Module 12

Calibrated Decisions: RLCD and Jev

Training and using models for calibrated decisions: outcome rewards, thresholds from costs, and TypeSafe’s RLCD and Jev as a case study in what is and is not public.

Open the briefing Open in Colab PyTorch TypeSafe SDK

14:45–16:45

Module 13

Retrieval-Augmented Generation

Building a RAG pipeline with LlamaIndex and LangChain, and evaluating retrieval and answers separately.

Open the briefing Open in Colab LlamaIndex LangChain Jev

Run Sheet

09:15–11:15 · Module 11 · Calibration
09:15 briefing · 10:10 lab · 11:05 debrief

  • In the room: the live plan, 44 minutes of exposition and 11 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 7 minutes of compute (Google Colab, CPU runtime, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.

11:30–12:25 · 13:25–14:30 · Module 12 · Calibrated Decisions: RLCD and Jev
11:30 briefing · 13:25 lab · 14:20 debrief

  • In the room: the live plan, 45 minutes of exposition and 10 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 3 minutes of compute (Google Colab, CPU runtime, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, a toy model stands in for the real system; its numbers only illustrate.

14:45–16:45 · Module 13 · Retrieval-Augmented Generation
14:45 briefing · 15:40 lab · 16:35 debrief

  • In the room: the live plan, 45 minutes of exposition and 10 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 8 minutes of compute (Google Colab, free T4 GPU, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.

Compute times are planning estimates, not measurements on the teaching runtime; the readiness page has what has been run, where.

Day Wrap-Up (16:45–17:00)

In pairs for 5 minutes, then the room for 10. The facilitator adds today’s rows to the board and keeps them for Day 5. The numbers are the room’s own, from its runs.

  1. Fixed and left open. For Modules 11, 12 and 13, which limitation of the step before did each remove, and which did it leave for the next? Start from Module 10: RLHF optimizes what raters prefer, and raters do not judge confidence.

  2. The running table. Fill in the room’s results. Write the path beside each number, as the notebook’s last cell prints it (real open models, Lab 12’s toy model, commercial APIs, or offline stand-ins, which measure no model); the results worksheet has a table per comparison group.

    Lab Rows Metric
    Lab 11, arXiv Topics v1 the Lab 6 classifier (or Lab 1’s, where the lab fell back to it), before and after temperature scaling accuracy, ECE, Brier score, log loss
    Lab 11, decision set the language model’s stated confidence accuracy, ECE, Brier score; the abstention threshold chosen and its coverage
    Lab 12, decision set the toy model with an accuracy reward and with a Brier reward (our illustration, not TypeSafe’s method); thresholds chosen on dev; Jev, where someone had a key accuracy, ECE, Brier score; cost per case beside act-all, ask-all and escalate-all
    Lab 13, workshop briefings chunk size and top-k at a fixed context budget; with and without reranking recall@k, MRR; faithfulness and the two-by-two table

  3. Ranking or values. Which of today’s numbers depend only on how the cases are ranked, and which on the probability values themselves? Which would temperature scaling change? And which rows of Lab 12 are evidence about our toy model, and which about Jev?

If time allows, a transfer question from today’s modules (T11, T12 or T13) can replace item 3, in the same minutes. It changes the example, so the answer has to be reasoned out rather than recalled.

Times are on the schedule.