Day 3

Using and Aligning LLMs

How do we use these models, and what are they trained to want?

Day 3 starts with the commercial models you reach through an API, and with what their answers do not tell you. It then asks what these models are trained to want: Module 9 frames generation as reinforcement learning and learns a reward from human comparisons, and Module 10 optimizes a model against that reward, which is RLHF, and shows how it fails.

Warm-Up: What Carried Over (09:00–09:15)

Answer alone on paper for 6 minutes, notes closed; compare with a neighbor for 4; the facilitator takes the two most-missed questions for 5. Each link opens the question with its folded answer. Three questions come from Day 2 and two from Day 1.

Question Topic From Why today
5.3 A tie within seed noise Day 2 · Module 5 Lab 8 prints a standard error beside every score
6.1b Masked language modeling Day 2 · Module 6 The models behind Module 8’s APIs are causal language models
7.3 What perplexity and ROUGE miss Day 2 · Module 7 Lab 8 scores outputs on a fixed evaluation set instead
2.2a The negative-sampling loss Day 1 · Module 2 Module 9’s reward-model loss has the same log-sigmoid form
4.1 The teacher-forced loss Day 1 · Module 4 Module 9 asks what training on demonstrations cannot teach

09:15–11:15

Module 8

LLMs Through APIs

OpenAI and Claude side by side: prompting, structured output, tool use, cost and evaluation.

Open the briefing Open in Colab OpenAI Anthropic Hugging Face Pydantic

11:30–12:25
13:25–14:30

Module 9

Reinforcement and Preference Learning

Text generation as a reinforcement-learning problem, the policy gradient, and reward models learned from comparisons.

Open the briefing Open in Colab PyTorch Hugging Face

14:45–16:45

Module 10

RLHF

The full reinforcement-learning-from-human-feedback pipeline, DPO, and the ways it goes wrong.

Open the briefing Open in Colab PyTorch Hugging Face

Run Sheet

09:15–11:15 · Module 8 · LLMs Through APIs
09:15 briefing · 10:10 lab · 11:05 debrief

  • In the room: the live plan, 44 minutes of exposition and 11 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 3 minutes of compute (Google Colab, free T4 GPU, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.

11:30–12:25 · 13:25–14:30 · Module 9 · Reinforcement and Preference Learning
11:30 briefing · 13:25 lab · 14:20 debrief

  • In the room: the live plan, 45 minutes of exposition and 10 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 10 minutes of compute (Google Colab, CPU runtime, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, the lab always runs its real models.

14:45–16:45 · Module 10 · RLHF
14:45 briefing · 15:40 lab · 16:35 debrief

  • In the room: the live plan, 45 minutes of exposition and 10 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 7 minutes of compute (Google Colab, free T4 GPU, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.

Compute times are planning estimates, not measurements on the teaching runtime; the readiness page has what has been run, where.

Day Wrap-Up (16:45–17:00)

In pairs for 5 minutes, then the room for 10. The facilitator adds today’s rows to the board and keeps them for Day 5. The numbers are the room’s own, from its runs.

  1. Fixed and left open. For Modules 8, 9 and 10, which limitation of the step before did each remove, and which did it leave for the next? Start from Module 7: a small model can be tuned to follow instructions, but the strongest models are reached only through an API.

  2. The running table. Fill in the room’s results. Write the path beside each number, as the notebook’s last cell prints it (the lab’s own code and models, real open models, commercial APIs, or offline stand-ins, which measure no model); the results worksheet has a table per comparison group.

    Lab Rows Metric
    Lab 8, extraction set each provider the room ran: the open model, and OpenAI or Claude where someone had a key schema-validity rate, exact match with its standard error, cost per run, mean latency
    Lab 9, toy task and preference pairs REINFORCE without and with a baseline; the reward model gradient variance; held-out pairwise accuracy beside its ceiling
    Lab 10, Lab 6’s small GPT-2 against Lab 9’s reward model KL-penalized policy; the same with \beta = 0; DPO reward gain, KL drift from the reference, gold reward, distinct-bigram share; DPO’s held-out preference accuracy

  3. What the numbers do not say. In which row did the number being optimized rise while the thing you care about fell? And which number in today’s table, or in Day 2’s, tells you how sure a model is of one particular answer?

If time allows, a transfer question from today’s modules (T8, T9 or T10) can replace item 3, in the same minutes. It changes the example, so the answer has to be reasoned out rather than recalled.

Times are on the schedule.