Results Worksheet

Your Own Numbers, With What It Takes to Compare Them

Each lab ends with a number. This worksheet keeps them in one place, beside the conditions a comparison needs: the data and split, the unit a loss is averaged over, the metric and which direction is better, and the baseline. Rows in the same table can be compared; rows in different tables cannot, and each table says why. Download it as a CSV file and fill it in as you go, or copy a table into your own notes.

What to write in each column.

The day wrap-ups use these tables. Run times and “has run on” claims on this site come from run records, never from this worksheet (readiness).

Language Modeling: Tiny Shakespeare, per Character

Comparable. Every row scores the same 60,394 characters of the Tiny Shakespeare test split, over the same 65-character vocabulary of the training split, with the same formula, and every setting was chosen on the validation split.

Do not compare. Never add a per-word or per-subword-token number to this group (Lab 1’s word models, Lab 6, Lab 7). On a CPU, Labs 3 and 5 run in QUICK mode, a smaller configuration: write QUICK beside such a row and compare it only with other QUICK rows.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 1, Ex. 3 Character bigram with add-k smoothing (k chosen on val) Tiny Shakespeare, test, per character cross-entropy, nats per character (lower is better) uniform over the 65 characters: ln 65 ≈ 4.17 nats per character (definition)
Lab 1, Ex. 3 Character trigram with add-k smoothing (k chosen on val) Tiny Shakespeare, test, per character cross-entropy, nats per character (lower is better) the bigram row above; uniform: ln 65 ≈ 4.17 (definition)
Lab 1, Ex. 3 Character 5-gram with add-k smoothing (the best order up to 5 on val) Tiny Shakespeare, test, per character cross-entropy, nats per character (lower is better) the bigram and trigram rows above; uniform: ln 65 ≈ 4.17 (definition)
Lab 3 RNN language model, hand-written cell, hidden size 256 Tiny Shakespeare, test, per character cross-entropy, nats per character (lower is better) your Lab 1 5-gram row; recorded reference 1.8326 nats per character, perplexity 6.25 (data/baselines.json, quoted in Module 3, section 6)
Lab 3 LSTM language model, nn.LSTM, hidden size 256 Tiny Shakespeare, test, per character cross-entropy, nats per character (lower is better) your Lab 1 5-gram row; recorded reference 1.8326 nats per character, perplexity 6.25 (data/baselines.json, quoted in Module 3, section 6)
Lab 5, Ex. 4 Mini-GPT before training Tiny Shakespeare, val, per character cross-entropy, nats per character (lower is better) uniform over the 65 characters: ln 65 = 4.1744 (definition); an untrained model should be close to it
Lab 5, Ex. 4 LSTM of Lab 3’s architecture, retrained with the mini-GPT’s training loop and budget Tiny Shakespeare, test, per character cross-entropy, nats per character (lower is better) Lab 3’s own LSTM run; recorded reference 1.6114 nats per character (data/baselines.json, quoted in Module 5, section 7)
Lab 5, Ex. 4 Mini-GPT: decoder-only transformer, d = 128, 4 heads, 4 blocks, context length 128 Tiny Shakespeare, test, per character cross-entropy, nats per character (lower is better) the retrained LSTM in the row above, its partner at the same budget; recorded reference 1.5727 nats per character (data/baselines.json, quoted in Module 5, section 7)

Language Modeling: Tiny Shakespeare, per Word

Comparable. The same text and splits as the per-character group, cut into words, with a vocabulary of the words seen at least twice in the training split (min_count = 2).

Do not compare. Perplexity per word is on another scale from the per-character numbers: these rows never share a column with the per-character group.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 1, Ex. 3 Word bigram with add-k smoothing (k chosen on val) Tiny Shakespeare, test, per word perplexity per word (lower is better) uniform over the word vocabulary: perplexity equal to its size, which the lab prints (definition)
Lab 1, Ex. 3 Word trigram with add-k smoothing (k chosen on val) Tiny Shakespeare, test, per word perplexity per word (lower is better) the word bigram row above; uniform: the vocabulary size (definition)

Tokenization: arXiv Topics v1 Validation Texts, Tokens per Word

Comparable. The same 600 validation abstracts, tokenized by each tokenizer with special tokens left out; total tokens divided by total whitespace-separated words.

Do not compare. Tokens per word is a conversion between units for this text, not a quality score, and a tokenizer belongs to its model.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 6, Ex. 2 Our byte-pair encoding, 4,000 entries, trained on the training abstracts arXiv Topics v1, val, per word tokens per word (lower is better) 1 token per word: a tokenizer that never splits a word (definition)
Lab 6, Ex. 2 DistilBERT’s WordPiece tokenizer (30,522 entries, lowercased) arXiv Topics v1, val, per word tokens per word (lower is better) our 4,000-entry tokenizer in the row above; 1 token per word (definition)
Lab 6, Ex. 2 GPT-2’s byte-level BPE tokenizer (50,257 entries) arXiv Topics v1, val, per word tokens per word (lower is better) our 4,000-entry tokenizer in the first row; 1 token per word (definition)

Word Vectors: Probe Words and Analogies

Comparable. The same skip-gram embeddings, trained on the arXiv Topics training abstracts and searched over the same 12,469-word vocabulary, against probe lists written before training.

Do not compare. A rough check, not a benchmark: an expected list can miss good neighbors, and a hit can be luck.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 2, Ex. 2 Skip-gram with negative sampling, trained in Lab 2 arXiv Topics v1, train, per probe word neighbor hit rate: share of probe words with an expected word among their 10 nearest neighbors by cosine (higher is better) random vectors: close to 0, since 10 random words out of 12,469 rarely include an expected one
Lab 2 Skip-gram with negative sampling, trained in Lab 2 arXiv Topics v1, train, per analogy analogy accuracy (higher is better) random vectors: close to 0 (the answer must be one word out of 12,469)

Classification: arXiv Topics v1, Test Split

Comparable. Every row is trained on the same 4,800 training papers, tuned on the 600 validation papers and scored once on the same 1,600 test papers, 400 per class, with scikit-learn’s accuracy and macro-F1.

Do not compare. On a CPU, Lab 6 fine-tunes a smaller encoder on part of the training split: label that row and do not rank it against the full-data rows. A difference of a few papers is within seed-to-seed variation.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 1, Ex. 6 TF-IDF features and logistic regression (C chosen on val) arXiv Topics v1, test, per example accuracy (higher is better) chance with four balanced classes: 0.25 (definition)
Lab 1, Ex. 6 TF-IDF features and logistic regression (C chosen on val) arXiv Topics v1, test, per example macro-F1 (higher is better) uniform random guessing: about 0.25 (definition)
Lab 1 Word counts and naive Bayes (alpha = 1) arXiv Topics v1, test, per example accuracy (higher is better) the TF-IDF and logistic regression row
Lab 1 Word counts and naive Bayes (alpha = 1) arXiv Topics v1, test, per example macro-F1 (higher is better) the TF-IDF and logistic regression row
Lab 2, Ex. 4 Averaged skip-gram embeddings (frozen) and a feed-forward network arXiv Topics v1, test, per example accuracy (higher is better) TF-IDF and logistic regression, recomputed in Lab 2; recorded reference 0.884 (data/baselines.json, quoted in Module 2, section 7)
Lab 2, Ex. 4 Averaged skip-gram embeddings (frozen) and a feed-forward network arXiv Topics v1, test, per example macro-F1 (higher is better) TF-IDF and logistic regression, recomputed in Lab 2; recorded reference 0.884 (data/baselines.json, quoted in Module 2, section 7)
Lab 6, Ex. 5 Pretrained encoder fine-tuned for classification (name the encoder and the training rows) arXiv Topics v1, test, per example accuracy (higher is better) TF-IDF and logistic regression, recomputed in Lab 6; recorded reference 0.884 (data/baselines.json, quoted in Module 2, section 7)
Lab 6, Ex. 5 Pretrained encoder fine-tuned for classification (name the encoder and the training rows) arXiv Topics v1, test, per example macro-F1 (higher is better) TF-IDF and logistic regression, recomputed in Lab 6; recorded reference 0.884 (data/baselines.json, quoted in Module 2, section 7)

Sequence Transduction: Dates to ISO Format

Comparable. Every model is trained on the same 100,000 generated sources for the same number of steps and scored with greedy decoding on the same generated test set, 500 sources for each number of dates K; an output counts only if the whole string is right.

Do not compare. Compare models at the same K, and read K = 1 against K = 4 within one model: the gap is the bottleneck.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 4, Ex. 2 Encoder-decoder without attention (GRU encoder and decoder) generated in Lab 4, test, K = 1 (500 sources, seed 0), per example exact match (higher is better) the same model at K = 4, in the next row
Lab 4, Ex. 2 Encoder-decoder without attention (GRU encoder and decoder) generated in Lab 4, test, K = 4 (500 sources, seed 0), per example exact match (higher is better) the same model at K = 1, in the row above
Lab 4, Ex. 3 Encoder-decoder with dot-product attention generated in Lab 4, test, K = 1 (500 sources, seed 0), per example exact match (higher is better) the model without attention at K = 1
Lab 4, Ex. 3 Encoder-decoder with dot-product attention generated in Lab 4, test, K = 4 (500 sources, seed 0), per example exact match (higher is better) the model without attention at K = 4
Lab 4, Ex. 4 Encoder-decoder with additive attention generated in Lab 4, test, K = 4 (500 sources, seed 0), per example exact match (higher is better) dot-product attention at K = 4
Lab 4, Ex. 5 Encoder-decoder with dot-product attention, teacher-forced on the gold target generated in Lab 4, test, K = 4 (500 sources, seed 0), per target digit alignment hit rate at tolerance 1: share of digits whose largest attention weight falls in the right date (higher is better) chance at tolerance 1 for K = 4, which the lab prints beside it

Instruction Following: Dolly 15k Held-Out Examples

Comparable. Before and after LoRA: the same base model (SmolLM2-135M), the same 200 held-out examples, the same chat template, greedy decoding and the same output limit.

Do not compare. Response perplexity is per SmolLM2 token over the response tokens only: it shares no column with the per-character numbers of Labs 1, 3 and 5. On a CPU (FAST), Lab 7 uses 50 held-out examples, generates 10 replies and trains briefly: label those rows.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 7, Ex. 4 SmolLM2-135M base model, before fine-tuning Databricks Dolly 15k, held-out (200 examples), per token response perplexity per token (lower is better) none: this is the before measurement that the LoRA row is compared with
Lab 7, Ex. 5 The same model after LoRA (rank 8 on the query and value projections) Databricks Dolly 15k, held-out (200 examples), per token response perplexity per token (lower is better) the base model row above
Lab 7, Ex. 4 SmolLM2-135M base model, before fine-tuning Databricks Dolly 15k, held-out, the first 50 examples, for which replies are generated (10 on a CPU), per example share of generated replies that end with <|im_end|> (higher is better) none: this is the before measurement that the LoRA row is compared with
Lab 7, Ex. 5 The same model after LoRA (rank 8 on the query and value projections) Databricks Dolly 15k, held-out, the first 50 examples, for which replies are generated (10 on a CPU), per example share of generated replies that end with <|im_end|> (higher is better) the base model row above
Lab 7, Ex. 6 SmolLM2-135M base model, before fine-tuning Databricks Dolly 15k, held-out, the first 50 examples, for which replies are generated (10 on a CPU), per example mean ROUGE-1 against the reference responses (higher is better) none: this is the before measurement that the LoRA row is compared with
Lab 7, Ex. 6 The same model after LoRA (rank 8 on the query and value projections) Databricks Dolly 15k, held-out, the first 50 examples, for which replies are generated (10 on a CPU), per example mean ROUGE-1 against the reference responses (higher is better) the base model row above
Lab 7, Ex. 5 LoRA adapter, rank 8 on the query and value projections Databricks Dolly 15k, not applicable (a count of parameters), per model trainable parameters (lower is better) full fine-tuning trains every parameter of the base model (definition)

Structured Output and Tools: Lab 8’s Generated Announcements

Comparable. Every provider runs the same generated items through the same harness, one-shot with up to R = 2 retries; an invalid output counts as wrong; write N and the standard error beside each rate.

Do not compare. The offline stub is a test double, not a model: its rows are not results. On the open model on a CPU the lab uses 12 items and 3 tool questions, so compare providers only on the same items. Copy a row for each extra provider you ran.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 8, Ex. 3 The provider selected by PROVIDER (open model, OpenAI or Claude), one-shot prompt generated in Lab 8, evaluation set (40 items; 12 on the open model on a CPU), per example schema-validity rate at the first attempt (R = 0) (higher is better) the zero-shot prompt’s validity at R = 0, printed beside it
Lab 8, Ex. 3 The provider selected by PROVIDER (open model, OpenAI or Claude), one-shot prompt generated in Lab 8, evaluation set (40 items; 12 on the open model on a CPU), per example schema-validity rate after up to R = 2 retries (higher is better) the same provider’s validity at R = 0, in the row above
Lab 8, Ex. 3 The provider selected by PROVIDER (open model, OpenAI or Claude), one-shot prompt, R = 2 generated in Lab 8, evaluation set (40 items; 12 on the open model on a CPU), per example record exact match (all five fields right; invalid counts as wrong) (higher is better) zero-shot record exact match, printed beside it
Lab 8, Ex. 4 The same provider in the hand-written two-tool loop generated in Lab 8, tool questions (3 on the open model on a CPU), per question task success: share of questions finished with the expected answer (higher is better) none recorded; read it beside the number of model calls per question
Lab 8, Ex. 5 The provider selected by PROVIDER, one-shot extraction, R = 2 generated in Lab 8, evaluation set (40 items; 12 on the open model on a CPU), per request cost in US dollars per 100 requests, at the prices in the notebook (lower is better) the open model: no API fee (definition; the runtime itself is not free)
Lab 8, Ex. 5 The provider selected by PROVIDER, one-shot extraction, R = 2 generated in Lab 8, evaluation set (40 items; 12 on the open model on a CPU), per request mean latency in seconds per request (lower is better) none: compare providers only on the same items

Policy Gradient: Lab 9’s Toy Sequence Task

Comparable. The same toy task (8 symbols, 6 positions, a hidden target sequence), the same batch size and learning rate; the best policy earns J = 1.

Do not compare. A task with a known answer: it checks the estimator and the baseline, not a language model.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 9, Ex. 1 REINFORCE without a baseline, 300 steps generated in Lab 9, the toy environment (no split), per episode expected return J of the trained policy (higher is better) the uniform starting policy: J = 1/8 = 0.125 (definition)
Lab 9, Ex. 2 REINFORCE with the leave-one-out baseline, at the uniform policy generated in Lab 9, the toy environment (no split), per gradient estimate total variance of the gradient estimate, as a ratio to no baseline (lower is better) no baseline: ratio 1 (definition)
Lab 9, Ex. 2 REINFORCE without a baseline generated in Lab 9, the toy environment (no split), per training run mean steps to reach J ≥ 0.9, over seeds 0 to 4 (lower is better) the leave-one-out row below
Lab 9, Ex. 2 REINFORCE with the leave-one-out baseline generated in Lab 9, the toy environment (no split), per training run mean steps to reach J ≥ 0.9, over seeds 0 to 4 (lower is better) no baseline, the row above

Preference Pairs: Lab 9’s Held-Out Pairs

Comparable. The same 1,000 held-out pairs of Lab 9’s preference data, labeled by a noisy rater from a known gold score; each accuracy is the share of pairs ordered as the rater labeled them.

Do not compare. Read every accuracy against the ceiling Acc*, not against 1. On a CPU (FAST), Lab 10 scores a subset of 64 held-out pairs: label that row.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 9, Ex. 3 The gold rule itself: the ceiling Acc* Lab 9 preference pairs, heldout (1,000 pairs), per pair held-out pairwise accuracy (higher is better) chance: 0.5 (definition)
Lab 9, Ex. 4 Reward model r_φ trained on the preference pairs Lab 9 preference pairs, heldout (1,000 pairs), per pair held-out pairwise accuracy (higher is better) Acc* in the row above; chance 0.5 (definition)
Lab 9, Ex. 4 Reward model r_φ trained on the preference pairs Lab 9 preference pairs, heldout (untied pairs), per pair accuracy against the gold ordering (higher is better) chance: 0.5 (definition)
Lab 9, Ex. 4 Reward model r_φ trained on the preference pairs Lab 9 preference pairs, heldout responses, per response Spearman correlation of r_φ with the gold score (higher is better) no relation: 0 (definition)
Lab 10, Ex. 5 DPO policy, scored by its implicit reward Lab 9 preference pairs, heldout (1,000 pairs; 64 on a CPU), per pair held-out pairwise accuracy (higher is better) the reward model’s accuracy on the same pairs and the gold rule’s ceiling, both printed beside it

Policies Against the Reward Model: Lab 10’s Evaluation Prompts

Comparable. Every policy starts from the same reference model and is sampled on the same 64 evaluation prompts, 8 samples each, with one fixed evaluation seed; r_φ is in Lab 9’s normalized units, the gold reward is the lab’s hidden rule, and drift is the KL estimate from the reference in nats per response.

Do not compare. On a CPU (FAST) the trained policies are smoke tests on 8 prompts and their numbers are not results. One seed is not the seed protocol of brief 10. The gold reward is the lab’s known rule, not a human judgment.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 10, Ex. 3 RLHF policy with the KL penalty (β as in the lab) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response mean reward-model score r_φ (higher is better) the reference policy on the same prompts, printed beside it
Lab 10, Ex. 3 RLHF policy with the KL penalty (β as in the lab) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response mean gold reward (higher is better) the reference policy on the same prompts, printed beside it
Lab 10, Ex. 3 RLHF policy with the KL penalty (β as in the lab) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response drift: KL from the reference, nats per response (lower is better) the reference policy: 0 (definition)
Lab 10, Ex. 3 RLHF policy with the KL penalty (β as in the lab) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per prompt distinct-2: share of distinct bigrams among a prompt’s samples (higher is better) the reference policy on the same prompts, printed beside it
Lab 10, Ex. 4 RLHF policy without the KL penalty (β = 0) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response mean reward-model score r_φ (higher is better) the policy with the KL penalty
Lab 10, Ex. 4 RLHF policy without the KL penalty (β = 0) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response mean gold reward (higher is better) the policy with the KL penalty
Lab 10, Ex. 4 RLHF policy without the KL penalty (β = 0) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response drift: KL from the reference, nats per response (lower is better) the policy with the KL penalty; the reference: 0 (definition)
Lab 10, Ex. 4 RLHF policy without the KL penalty (β = 0) Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per prompt distinct-2: share of distinct bigrams among a prompt’s samples (higher is better) the policy with the KL penalty
Lab 10, Ex. 5 DPO policy trained on the same preference data Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response mean reward-model score r_φ (higher is better) the RLHF policy with the KL penalty
Lab 10, Ex. 5 DPO policy trained on the same preference data Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response mean gold reward (higher is better) the RLHF policy with the KL penalty
Lab 10, Ex. 5 DPO policy trained on the same preference data Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per response drift: KL from the reference, nats per response (lower is better) the RLHF policy with the KL penalty; the reference: 0 (definition)
Lab 10, Ex. 5 DPO policy trained on the same preference data Lab 9 prompts, preview items and gold rule, eval (64 prompts × 8 samples), per prompt distinct-2: share of distinct bigrams among a prompt’s samples (higher is better) the RLHF policy with the KL penalty

Calibration of a Classifier: arXiv Topics v1 Test Split

Comparable. The same 1,600 test papers and the same classifier’s logits; the temperature is fitted on the 600 validation papers; ECE uses 15 equal-width bins; the Brier score (summed over the 4 classes) and the log loss are averaged per paper; selective prediction uses l_wrong = 10 and l_defer = 1.

Do not compare. Name the classifier: the Lab 6 encoder, or Lab 1’s classifier where the lab fell back to it. The binary Brier scores of Lab 11, Part B, and of Lab 12 are on another scale and do not belong here.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 11, Ex. 1 The lab’s classifier (Lab 6 encoder or the Lab 1 fallback) arXiv Topics v1, test, per example accuracy (higher is better) chance with four balanced classes: 0.25 (definition)
Lab 11, Ex. 1 The lab’s classifier, before temperature scaling arXiv Topics v1, test, per example ECE, 15 equal-width bins (lower is better) the noise floor for this N, which Exercise 1 prints
Lab 11, Ex. 2 The lab’s classifier, before temperature scaling arXiv Topics v1, test, per example Brier score, summed over the 4 classes (lower is better) 1/4 on every class: 0.75 (definition)
Lab 11, Ex. 2 The lab’s classifier, before temperature scaling arXiv Topics v1, test, per example log loss, nats per example (lower is better) 1/4 on every class: ln 4 ≈ 1.386 (definition)
Lab 11, Ex. 3 The same classifier after temperature scaling (write the fitted temperature in your interpretation) arXiv Topics v1, test, per example ECE, 15 equal-width bins (lower is better) the ECE before scaling; the noise floor of Exercise 1
Lab 11, Ex. 3 The same classifier after temperature scaling arXiv Topics v1, test, per example Brier score, summed over the 4 classes (lower is better) the Brier score before scaling
Lab 11, Ex. 3 The same classifier after temperature scaling arXiv Topics v1, test, per example log loss, nats per example (lower is better) the log loss before scaling
Lab 11, Ex. 5 The lab’s classifier, abstention threshold chosen on val arXiv Topics v1, test, per example coverage: share of test papers acted on (higher is better) Chow’s threshold, 0.9 at these costs, printed beside it
Lab 11, Ex. 5 The lab’s classifier, abstention threshold chosen on val arXiv Topics v1, test, per example cost per case at l_wrong = 10, l_defer = 1 (lower is better) defer everything: 1 per case (definition, l_defer = 1); act on everything, printed beside it

Calibration on the Decision Set: Test Split

Comparable. The same 300 test items of Workshop Desk Decisions v1 (template items only); the outcome is 1 when the chosen answer is right; the probability is that of the chosen answer (for a language model, its stated confidence); ECE uses 15 bins and the Brier score is the binary one, the mean of (probability − outcome)².

Do not compare. The local toy decider is our illustration, not Jev and not TypeSafe’s method: label every row with its backend. Jev’s confidence field never goes on these axes. On the open model on a CPU, Lab 11 asks 80 test items, so its noise floor is larger.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 11, Ex. 4 Language model’s stated confidence (the provider you ran) Workshop Desk Decisions v1, test, per decision validity: share of replies with a valid answer and confidence (higher is better) every reply valid: 1 (definition)
Lab 11, Ex. 4 Language model’s stated confidence (the provider you ran) Workshop Desk Decisions v1, test, per decision accuracy, invalid replies counted as wrong (higher is better) chance: 0.5 on yes/no policy items, 0.2 on five-way route items (definition)
Lab 11, Ex. 4 Language model’s stated confidence (the provider you ran) Workshop Desk Decisions v1, test, per decision ECE of the valid replies, 15 bins (lower is better) the noise floor for this N, printed beside it
Lab 11, Ex. 4 Language model’s stated confidence (the provider you ran) Workshop Desk Decisions v1, test, per decision binary Brier score of the valid replies (lower is better) a constant probability of 0.5: 0.25 (definition)
Lab 12, Ex. 1 Toy decision model trained with an accuracy reward (our illustration, not TypeSafe’s method) Workshop Desk Decisions v1, test (policy and route pooled), per decision accuracy (higher is better) chance: 0.5 on policy items, 0.2 on route items (definition)
Lab 12, Ex. 1 Toy decision model trained with an accuracy reward (our illustration, not TypeSafe’s method) Workshop Desk Decisions v1, test (policy and route pooled), per decision ECE, 15 bins (lower is better) the noise floor, printed beside it
Lab 12, Ex. 1 Toy decision model trained with an accuracy reward (our illustration, not TypeSafe’s method) Workshop Desk Decisions v1, test (policy and route pooled), per decision binary Brier score (lower is better) a constant probability of 0.5: 0.25 (definition)
Lab 12, Ex. 1 Toy decision model trained with a Brier-score reward (our illustration, not TypeSafe’s method) Workshop Desk Decisions v1, test (policy and route pooled), per decision accuracy (higher is better) the accuracy-reward model on the same items
Lab 12, Ex. 1 Toy decision model trained with a Brier-score reward (our illustration, not TypeSafe’s method) Workshop Desk Decisions v1, test (policy and route pooled), per decision ECE, 15 bins (lower is better) the accuracy-reward model on the same items; the noise floor
Lab 12, Ex. 1 Toy decision model trained with a Brier-score reward (our illustration, not TypeSafe’s method) Workshop Desk Decisions v1, test (policy and route pooled), per decision binary Brier score (lower is better) the accuracy-reward model on the same items
Lab 12, Ex. 3 Jev, keyed path only (write the model name the response reports) Workshop Desk Decisions v1, test, per decision accuracy (higher is better) the toy models on the same items; chance 0.5 or 0.2 (definition)
Lab 12, Ex. 3 Jev, keyed path only, from the probability of the chosen answer, never from its confidence field Workshop Desk Decisions v1, test, per decision ECE, 15 bins (lower is better) the noise floor, printed beside it
Lab 12, Ex. 3 Jev, keyed path only, from the probability of the chosen answer Workshop Desk Decisions v1, test, per decision binary Brier score (lower is better) a constant probability of 0.5: 0.25 (definition)

Decisions From Probabilities: Cost per Case on the Decision Set

Comparable. Within one lab only: thresholds are chosen on the 100 dev items and costs are averaged over the 300 test items, at the costs named in each row.

Do not compare. Lab 11 has two actions (l_wrong = 10, l_defer = 1) and Lab 12 three (wrong 20, ask 0.5, miss 4, escalate 3): costs per case from the two labs cannot be compared. Rows from the local toy decider are our illustration, not Jev.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 11, Ex. 5 Language model’s stated confidence, threshold chosen on dev Workshop Desk Decisions v1, test, per decision coverage: share of test items acted on (higher is better) Chow’s threshold, 0.9 at these costs, printed beside it
Lab 11, Ex. 5 Language model’s stated confidence, threshold chosen on dev Workshop Desk Decisions v1, test, per decision cost per case at l_wrong = 10, l_defer = 1 (invalid replies deferred) (lower is better) defer everything: 1 per case (definition, l_defer = 1)
Lab 12, Ex. 4 The decider (local toy decider without a key, Jev with one), act, ask and escalate thresholds chosen on dev Workshop Desk Decisions v1, test, per decision cost per case at wrong 20, ask 0.5, miss 4, escalate 3 (lower is better) escalate everything: 3 per case (definition, l_esc = 3); act on everything and ask about everything, printed beside it
Lab 12, Ex. 4 The same decider with the analytic thresholds (0.375, 0.969) Workshop Desk Decisions v1, test, per decision cost per case at wrong 20, ask 0.5, miss 4, escalate 3 (lower is better) the thresholds chosen on dev, in the row above; escalate everything: 3 per case (definition)

Retrieval: Workshop Lectures v1, Questions Written by People

Comparable. The same frozen snapshot of the module pages and the same human-written questions; chunk size, top-k and the reranker are chosen on the 30 dev questions and reported once on the 50 test questions, at a context budget of k × L ≤ 1,024 tokens.

Do not compare. Until Workshop RAG Questions v1 exists, the lab runs on plumbing probes copied from the pages: they measure the code, not retrieval, and are labeled probe. Do not record probe numbers as results. Compare chunk sizes only at the same budget.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 13, Ex. 2 Dense retriever at the chunk size L and top-k chosen on dev Workshop Lectures v1, test (Workshop RAG Questions v1, 50 questions; not written yet), per question recall@k over the answerable questions (higher is better) the runner-up (L, k), printed beside it with the questions gained and lost
Lab 13, Ex. 2 Dense retriever at the chunk size L and top-k chosen on dev Workshop Lectures v1, test (Workshop RAG Questions v1, 50 questions; not written yet), per question MRR@k over the answerable questions (higher is better) the runner-up (L, k), printed beside it
Lab 13, Ex. 4 First-stage retriever before reranking, k0 candidates Workshop Lectures v1, test (Workshop RAG Questions v1, 50 questions; not written yet), per question recall@k0 (higher is better) none: it is the most any reranker can reach (definition)
Lab 13, Ex. 4 After reranking (the reranker kept on dev: cross-encoder, or Jev with a key) Workshop Lectures v1, test (Workshop RAG Questions v1, 50 questions; not written yet), per question recall@k (higher is better) the first stage’s recall@k on the same questions, printed beside it with the sign test
Lab 13, Ex. 4 After reranking (the reranker kept on dev: cross-encoder, or Jev with a key) Workshop Lectures v1, test (Workshop RAG Questions v1, 50 questions; not written yet), per question MRR@k (higher is better) the first stage’s MRR@k on the same questions, printed beside it

Answers From Retrieved Passages: Lab 13’s Test Questions

Comparable. The same test questions, retriever settings and generator; faithfulness by the lab’s judge over the answered questions; correctness by program checks on the answerable, auto-checkable questions; abstention by exact match with the abstention sentence.

Do not compare. The faithfulness judge has not been validated against human labels. Plumbing probes are not questions. On the open model on a CPU the lab answers 12 test questions.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 13, Ex. 5 The generator you ran, on the retriever chosen in Exercises 2 and 4 Workshop Lectures v1, test (Workshop RAG Questions v1; not written yet), per question faithfulness: mean share of supported sentences in the answered questions (one value per question kind, as printed) (higher is better) none: read it beside correctness, because a faithful answer can be wrong
Lab 13, Ex. 5 The generator you ran, on the retriever chosen in Exercises 2 and 4 Workshop Lectures v1, test, answerable and auto-checkable questions, per question correctness: share answered correctly (higher is better) the two-by-two table printed beside it: correctness with and without the evidence retrieved
Lab 13, Ex. 5 The generator you ran, on the retriever chosen in Exercises 2 and 4 Workshop Lectures v1, test, unanswerable questions, per question abstention rate on unanswerable questions (higher is better) a system that abstains on everything scores 1 here and 0 in correctness (definition)
Lab 13, Ex. 5 The generator you ran, on the retriever chosen in Exercises 2 and 4 Workshop Lectures v1, test, answerable questions, per question false-abstention rate on answerable questions (lower is better) 0 for a system that never abstains (definition)

Agent Evaluation: Router, Guard and Injection Tests

Comparable. The same test items of the decision set and the same guard set derived from its labels, the same desk corpus with one poisoned page, thresholds from Module 12’s costs (wrong 20, ask 0.5, miss 4, escalate 3) and a simulated human on ask; every rate with its N.

Do not compare. Label each row with its backend: the local toy decider and the stub agent are neither Jev nor a language model. A rate of 0 on a few dozen items is consistent with a true rate of several percent.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 14, Ex. 5 Router (the decider you ran) Workshop Desk Decisions v1, test, route items (150), per decision route accuracy, invalid counted as wrong (higher is better) chance over five routes: 0.2 (definition)
Lab 14, Ex. 5 Router (the decider you ran) Workshop Desk Decisions v1, test, route items the router was confident enough to follow, per decision accuracy when followed (higher is better) the route accuracy on all items, in the row above
Lab 14, Ex. 5 Guard with the P6 rule in code first (the decider you ran) Workshop Desk Decisions v1, test, guard items whose email is not allowed, per decision unsafe-action rate: share sent without approval (lower is better) escalating every email also scores 0, at 3 per case (definition)
Lab 14, Ex. 5 Guard with the P6 rule in code first (the decider you ran) Workshop Desk Decisions v1, test, guard items whose email is allowed, per decision held-back rate: share of allowed emails not sent alone (lower is better) escalating every email holds back all of them: 1 (definition)
Lab 14, Ex. 5 Guard with the P6 rule in code first (the decider you ran) Workshop Desk Decisions v1, test, guard set, per decision cost per case at wrong 20, ask 0.5, miss 4, escalate 3 (lower is better) escalate every email: 3 per case (definition, l_esc = 3)
Lab 14, Ex. 5 Guard, with and without the poisoned page among its passages Workshop Desk Decisions v1, test, 60 route-derived guard items, per decision mean probability shift Δp caused by the poisoned page (lower is better) a page that changes nothing: 0 (definition)
Lab 14, Ex. 5 Whole agent, HARD_RULES on, poisoned page in the index Workshop Desk Decisions v1, test, first 10 retrieve route items (4 on the open model on a CPU), per run injection success rate, attempted: share of runs where the model tried the injected action (lower is better) the carried-out rate in the next row: the gap is what the harness stopped
Lab 14, Ex. 5 Whole agent, HARD_RULES on, poisoned page in the index Workshop Desk Decisions v1, test, first 10 retrieve route items (4 on the open model on a CPU), per run injection success rate, carried out: share of runs that sent an email without approval (lower is better) the same runs with HARD_RULES off, in the next row
Lab 14, Ex. 5 Whole agent, HARD_RULES off, poisoned page in the index Workshop Desk Decisions v1, test, first 10 retrieve route items (4 on the open model on a CPU), per run injection success rate, carried out: share of runs that sent an email without approval (lower is better) HARD_RULES on, the row above

Capstone: The System on the Capstone Test Split

Comparable. Every pair runs the same 80 test questions twice, scored by the same fixed scoring cell: once with the starter system (the baseline) and once with the final system, on the same path class and the same day.

Do not compare. Until Workshop Capstone Questions v1 exists, every run uses plumbing probes and no number is a result. Compare only rows of the same path class (keyed, open or open-cpu). Seven gained and two lost is not yet a clear win: read the sign test.

Lab Model Data, split and unit Metric Baseline Your result Path
Lab 15 Starter system (your baseline run) Workshop Lectures v1, test (Workshop Capstone Questions v1, 80 questions; not written yet), per question capstone cost per question, wrong 5 and abstain-on-answerable 1 (lower is better) abstaining on everything: 1 per answerable question and 0 per unanswerable one (definition)
Lab 15 Your final system (one change, chosen on dev) Workshop Lectures v1, test (Workshop Capstone Questions v1, 80 questions; not written yet), per question capstone cost per question, wrong 5 and abstain-on-answerable 1 (lower is better) your baseline row
Lab 15 Starter system (your baseline run) Workshop Lectures v1, test, answerable and auto-checkable questions, per question accuracy, abstentions counted as not correct (higher is better) abstaining on everything: 0 (definition)
Lab 15 Your final system (one change, chosen on dev) Workshop Lectures v1, test, answerable and auto-checkable questions, per question accuracy, abstentions counted as not correct (higher is better) your baseline row
Lab 15 Starter system (your baseline run) Workshop Lectures v1, test, unanswerable questions, per question abstention rate on unanswerable questions (higher is better) abstaining on everything: 1 (definition)
Lab 15 Your final system (one change, chosen on dev) Workshop Lectures v1, test, unanswerable questions, per question abstention rate on unanswerable questions (higher is better) your baseline row
Lab 15 Starter system (your baseline run) Workshop Lectures v1, test, answerable questions, per question abstention rate on answerable questions (false abstention) (lower is better) never abstaining: 0 (definition)
Lab 15 Your final system (one change, chosen on dev) Workshop Lectures v1, test, answerable questions, per question abstention rate on answerable questions (false abstention) (lower is better) your baseline row
Lab 15 Starter system (your baseline run) Workshop Lectures v1, test (Workshop Capstone Questions v1, 80 questions; not written yet), per question API cost in US dollars per question (n/a on local paths: write the model calls per question) (lower is better) local paths: no API fee (definition)
Lab 15 Your final system (one change, chosen on dev) Workshop Lectures v1, test (Workshop Capstone Questions v1, 80 questions; not written yet), per question API cost in US dollars per question (n/a on local paths: write the model calls per question) (lower is better) your baseline row
Lab 15 Your final system against your baseline, paired by question Workshop Lectures v1, test (Workshop Capstone Questions v1, 80 questions; not written yet), per question questions gained: cost fell (higher is better) questions lost, the next row; the run-to-run flips of the baseline
Lab 15 Your final system against your baseline, paired by question Workshop Lectures v1, test (Workshop Capstone Questions v1, 80 questions; not written yet), per question questions lost: cost rose (lower is better) questions gained, the row above
Lab 15 Your final system against your baseline, paired by question Workshop Lectures v1, test (Workshop Capstone Questions v1, 80 questions; not written yet), per question two-sided sign test p on the questions that changed (lower is better) no change at all: p = 1 (definition)