Day 2

Transformers and Pretrained Models

How does one architecture, pretrained at scale, become a general tool?

Day 2 builds the transformer, then follows it through pretraining to fine-tuning with LoRA. Like Days 3 to 5, it opens with a warm-up on earlier days and closes with a wrap-up of the day’s results. Each module ends with a short debrief of its lab, and the middle module’s lab comes after lunch.

Warm-Up: What Carried Over (09:00–09:15)

Answer alone on paper for 6 minutes, notes closed; compare with a neighbor for 4; the facilitator takes the two most-missed questions for 5. Each link opens the question with its folded answer. Day 1 is the only earlier day, so three questions come from its afternoon and two from its morning.

Question Topic From Why today
4.3a Dot-product attention with a mask Day 1 · Module 4 Module 5’s self-attention is this lookup, with queries, keys and values from one sequence
4.2 Two dates through one vector Day 1 · Module 4 Attention removed the bottleneck; Module 5 removes the recurrence too
3.3 When two perplexities compare Day 1 · Module 3 Lab 5 scores the mini-GPT beside the LSTM and the n-gram model
1.2a Perplexity of a uniform model Day 1 · Module 1 The trivial baseline every model in Lab 5 must beat
2.3 Why the dense model need not win Day 1 · Module 2 Lab 6 fine-tunes an encoder on the same task and compares it with Labs 1 and 2

09:15–11:15

Module 5

The Transformer

Self-attention, multi-head attention and positional encodings, assembled into a small GPT.

Open the briefing Open in Colab PyTorch

11:30–12:25
13:25–14:30

Module 6

Pretraining and the Hugging Face Stack

Subword tokenization, masked and causal pretraining, and using pretrained models through Hugging Face.

Open the briefing Open in Colab Hugging Face PyTorch

14:45–16:45

Module 7

Fine-Tuning and LoRA

Turning a pretrained language model into an instruction follower, efficiently.

Open the briefing Open in Colab Hugging Face PEFT PyTorch

Run Sheet

09:15–11:15 · Module 5 · The Transformer
09:15 briefing · 10:10 lab · 11:05 debrief

  • In the room: the live plan, 45 minutes of exposition and 10 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 10 minutes of compute (Google Colab, free T4 GPU, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, the lab always runs its real models.

11:30–12:25 · 13:25–14:30 · Module 6 · Pretraining and the Hugging Face Stack
11:30 briefing · 13:25 lab · 14:20 debrief

  • In the room: the live plan, 45 minutes of exposition and 10 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 7 minutes of compute (Google Colab, free T4 GPU, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.

14:45–16:45 · Module 7 · Fine-Tuning and LoRA
14:45 briefing · 15:40 lab · 16:35 debrief

  • In the room: the live plan, 44 minutes of exposition and 11 of checks, demos and predictions.
  • Lab: open in Colab, a checkpoint after each exercise; about 7 minutes of compute (Google Colab, free T4 GPU, planned).
  • When something goes wrong: stuck on exercise N, run its folded Solution cell, then workshop.use_reference(N), and go on. Without keys, an open model runs in place of the commercial one.

Compute times are planning estimates, not measurements on the teaching runtime; the readiness page has what has been run, where.

Day Wrap-Up (16:45–17:00)

In pairs for 5 minutes, then the room for 10. The facilitator writes the table on the board and keeps it for Day 5. The numbers are the room’s own, from its runs.

  1. Fixed and left open. For Modules 5, 6 and 7, which limitation of the step before did each remove, and which did it leave for the next? Start from Module 4: attention still read from a recurrent encoder, one step at a time.

  2. The running table. Fill in the room’s results (test split, held-out examples for Lab 7). Write the path beside each number, as the notebook’s last cell prints it (the lab’s own code and models, real open models, commercial APIs, or offline stand-ins, which measure no model); the results worksheet has a table per comparison group.

    Thread Rows Metric
    Tiny Shakespeare, character level 5-gram (Lab 1); LSTM (Lab 3); LSTM retrained and mini-GPT (Lab 5) nats per character
    arXiv Topics v1 TF-IDF with logistic regression (Lab 1); averaged embeddings (Lab 2); fine-tuned encoder (Lab 6) accuracy, macro-F1
    Dolly instructions the base model before and after LoRA (Lab 7) response perplexity, ROUGE-1, trainable parameters

  3. One claim. Which row would you defend as an improvement on the row above it, and what must be stated beside it to make the claim fair? Why does Lab 7’s perplexity not belong in the same column as Lab 5’s?

If time allows, a transfer question from today’s modules (T5, T6 or T7) can replace item 3, in the same minutes. It changes the example, so the answer has to be reasoned out rather than recalled.

Times are on the schedule.