References

Readings by Module, the Courses and Documentation This Workshop Draws on, and How Each Entry Was Checked

Every module page ends with five readings and a note on why each matters. This page collects them, together with the other works the briefings cite, as full citations with stable links. A source appears in full once, under the first module that lists it as a reading; later modules point back to it.

For a term or a symbol rather than a source, use the glossary and notation index; for your own numbers, the results worksheet.

How the Entries Were Checked

Each entry ends with a note in italics saying how its citation was checked on 2026-10-05:

• Verified (with the source): the authors, title, year, venue and pages match a primary or authoritative record that we opened. These records are the ACL Anthology’s own metadata files, the metadata files of the Proceedings of Machine Learning Research (PMLR), PyPI, a project’s own repository, or the documentation page itself.
• Search summary: the details match several independent web search results, but we could not open the publisher’s page. The build environment cannot reach arXiv, the ACL Anthology website, the NeurIPS, ICLR and OpenReview proceedings, or most publishers.
• Not read: the source exists, but no one writing the workshop has read it from the build environment. The entry says what is missing.

No entry is cited from memory alone. Page ranges and DOIs are given only where a check confirmed them.

Courses

• Stanford University. CS224N: Natural Language Processing with Deep Learning. The model for this workshop’s order of ideas: derive, implement, then use the library. Search summary.
• Carnegie Mellon University. CS 11-747: Neural Networks for NLP, Graham Neubig, 2021 edition. The model for code-first briefings in which every equation maps to code. Search summary.
• Massachusetts Institute of Technology. 6.S191: Introduction to Deep Learning, Alexander Amini and Ava Amini. The model for short briefings followed at once by a lab. Search summary.

Textbooks

• Jurafsky, D. and Martin, J. H. Speech and Language Processing, 3rd edition draft (the August 2026 release). Free chapter PDFs. Chapter numbers change between draft releases, so the briefings name chapters by title. Modules 1, 2, 4, 5 and 6 use the chapters “Words and Tokens”, “N-gram Language Models”, “Logistic Regression and Text Classification”, “Embeddings”, “Neural Networks”, “RNNs and LSTMs”, “Transformers and Pretraining” and “Masked Language Models”. Search summary: the draft and its August 2026 release confirmed; the chapter titles were not checked against the current table of contents.

Day 1 · Foundations: From Counts to Attention

Module 1 · Text as Data

• Manning, C. D., Raghavan, P. and Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. Chapter 6 (term weighting and TF-IDF), chapter 13 (text classification and naive Bayes), and section 11.4.3, “Okapi BM25: a non-binary model”, for the BM25 stretch exercise of Lab 1. Search summary.
• Chen, S. F. and Goodman, J. (1999). An empirical study of smoothing techniques for language modeling. Computer Speech & Language 13(4), 359–394. The standard comparison of smoothing methods, for the optional box on interpolation, backoff and Kneser–Ney. An earlier version appeared at ACL 1996: An Empirical Study of Smoothing Techniques for Language Modeling, Proceedings of the 34th Annual Meeting of the ACL, 310–318 (preprint arXiv:cmp-lg/9606011). Journal version: search summary. ACL 1996 version: verified, ACL Anthology.
• Wang, S. and Manning, C. (2012). Baselines and Bigrams: Simple, Good Sentiment and Topic Classification. Proceedings of the 50th Annual Meeting of the ACL (Volume 2: Short Papers), 90–94. Why a well-tuned linear model is a serious baseline. Verified, ACL Anthology.
• Robertson, S. E. and Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4), 333–389. The derivation of BM25; also used in Module 13. Search summary.
• Also used: Jurafsky and Martin, the chapters “Words and Tokens”, “N-gram Language Models” and “Logistic Regression and Text Classification” (Textbooks).

Module 2 · Word Vectors and Neural Networks

• Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111–3119. Skip-gram with negative sampling, the loss implemented in Lab 2. Search summary.
• Goldberg, Y. and Levy, O. (2014). word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv:1402.3722. A slower version of the briefing’s derivation. Search summary.
• Pennington, J., Socher, R. and Manning, C. (2014). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532–1543. Verified, ACL Anthology.
• Bengio, Y., Ducharme, R., Vincent, P. and Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research 3, 1137–1155. The bridge from n-grams to neural language models. Search summary.
• Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781. Skip-gram and CBOW. Search summary.
• Levy, O. and Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization. Advances in Neural Information Processing Systems 27 (NIPS 2014). Skip-gram with negative sampling as a factorization of a shifted PMI matrix. Search summary.
• Levy, O., Goldberg, Y. and Dagan, I. (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. Transactions of the Association for Computational Linguistics 3, 211–225. Verified, ACL Anthology.
• Harris, Z. S. (1954). Distributional Structure. Word 10(2–3), 146–162. The distributional hypothesis. Search summary.
• Firth, J. R. (1957). A Synopsis of Linguistic Theory, 1930–1955. In Studies in Linguistic Analysis (special volume of the Philological Society), 1–32. Oxford: Blackwell. Search summary; no stable link checked.
• Also used: Jurafsky and Martin, the chapters “Embeddings” and “Neural Networks” (Textbooks).

Module 3 · Sequence Models

• Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8), 1735–1780. Search summary.
• Pascanu, R., Mikolov, T. and Bengio, Y. (2013). On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3), 1310–1318. The Jacobian analysis of vanishing and exploding gradients, and gradient norm clipping. Verified, PMLR.
• Karpathy, A. (2015). The Unreasonable Effectiveness of Recurrent Neural Networks. Blog post, May 21, 2015. Character-level models on Shakespeare, the model for Lab 3. Search summary.
• Olah, C. (2015). Understanding LSTM Networks. Blog post, August 27, 2015. Search summary.
• Holtzman, A., Buys, J., Du, L., Forbes, M. and Choi, Y. (2020). The Curious Case of Neural Text Degeneration. International Conference on Learning Representations (ICLR 2020). Nucleus sampling, for the Lab 3 stretch section. Search summary.
• Gers, F. A., Schmidhuber, J. and Cummins, F. (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation 12(10), 2451–2471. The forget gate. Search summary.
• Also used: Cho et al. (2014), the GRU (Module 4).

Module 4 · Seq2seq and Attention

• Sutskever, I., Vinyals, O. and Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems 27 (NIPS 2014), 3104–3112. The LSTM encoder–decoder. Search summary.
• Bahdanau, D., Cho, K. and Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations (ICLR 2015). The bottleneck and additive attention. Search summary.
• Luong, T., Pham, H. and Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1412–1421. Dot-product and general scores. Verified, ACL Anthology.
• Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. and Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1724–1734. The RNN encoder–decoder and the GRU; also cited in Module 3. Verified, ACL Anthology.
• Also used: Jurafsky and Martin, the chapter “RNNs and LSTMs” (Textbooks).

Day 2 · Transformers and Pretrained Models

Module 5 · The Transformer

• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017). Search summary.
• Rush, A. (2018), revised by Huang, A., Subramanian, S., Sum, J., Almubarak, K. and Biderman, S. (2022). The Annotated Transformer. The paper, line by line, as working PyTorch code. Verified: the author lines in the page’s source, harvardnlp/annotated-transformer on GitHub.
• Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L. and Liu, T.-Y. (2020). On Layer Normalization in the Transformer Architecture. Proceedings of the 37th International Conference on Machine Learning, PMLR 119, 10524–10533. Pre-norm against post-norm. Verified, PMLR.
• Karpathy, A. nanoGPT. A compact pre-norm GPT in PyTorch with a character-level Shakespeare example. Its README (November 2025 update) marks it as deprecated in favor of a successor project. Verified, GitHub repository.
• Ba, J. L., Kiros, J. R. and Hinton, G. E. (2016). Layer Normalization. arXiv:1607.06450. Search summary.
• Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B. and Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864; later published in Neurocomputing (2024). Rotary position embeddings. Search summary.
• Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI technical report (GPT-2). Search summary; no stable link checked.
• Also used: Jurafsky and Martin, the chapter “Transformers and Pretraining” (Textbooks).

Module 6 · Pretraining and the Hugging Face Stack

• Sennrich, R., Haddow, B. and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the ACL (Volume 1: Long Papers), 1715–1725. Byte-pair encoding for words. Verified, ACL Anthology.
• Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI technical report (GPT). Search summary.
• Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the ACL: Human Language Technologies, Volume 1, 4171–4186. Verified, ACL Anthology.
• Hugging Face. LLM Course, chapter 6: the Tokenizers library, and the Transformers documentation page Tokenization algorithms. BPE, WordPiece and Unigram side by side. Search summary.
• Gage, P. (1994). A New Algorithm for Data Compression. The C Users Journal 12(2), February 1994. Byte-pair encoding as a compression method. Search summary; no stable link checked.
• Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L. and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692. Search summary.
• Sanh, V., Debut, L., Chaumond, J. and Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 (a NeurIPS 2019 workshop paper). The encoder fine-tuned in Lab 6. Search summary.
• Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21(140), 1–67. T5. Search summary.
• Brown, T. B. et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). GPT-3 and in-context learning. Search summary.
• Also used: Jurafsky and Martin, the chapters “Words and Tokens”, “Transformers and Pretraining” and “Masked Language Models” (Textbooks).

Module 7 · Fine-Tuning and LoRA

• Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR 2022). Search summary.
• Aghajanyan, A., Gupta, S. and Zettlemoyer, L. (2021). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. Proceedings of the 59th Annual Meeting of the ACL and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 7319–7328. Verified, ACL Anthology.
• Hugging Face. PEFT documentation: LoRA, the developer guide for LoraConfig. Search summary.
• Hugging Face. Transformers documentation: Chat templates. Search summary.
• Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E. and Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. Search summary.
• Dettmers, T., Pagnoni, A., Holtzman, A. and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). Search summary.
• Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74–81. Verified, ACL Anthology.
• Papineni, K., Roukos, S., Ward, T. and Zhu, W.-J. (2002). Bleu: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the ACL, 311–318. Verified, ACL Anthology.
• Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M. and Gelly, S. (2019). Parameter-Efficient Transfer Learning for NLP. Proceedings of the 36th International Conference on Machine Learning, PMLR 97, 2790–2799. Adapters. Verified, PMLR.
• Li, X. L. and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the ACL and the 11th IJCNLP (Volume 1: Long Papers), 4582–4597. Verified, ACL Anthology.
• Lester, B., Al-Rfou, R. and Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045–3059. Verified, ACL Anthology.
• Kalajdzievski, D. (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv:2312.03732. Rank-stabilized LoRA. Search summary.
• Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T. and Chen, M.-H. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. Proceedings of the 41st International Conference on Machine Learning, PMLR 235, 32100–32121. Verified, PMLR.

Day 3 · Using and Aligning LLMs

Module 8 · LLMs Through APIs

The readings are the providers’ own documentation. Documentation pages move and change; if a link breaks, search the provider’s site for the page title.

• OpenAI. Function calling. Not read here: developers.openai.com was blocked from the build environment on 2026-10-05. The briefing’s author opened it on 2026-10-04.
• OpenAI. Structured outputs. Not read here, as above.
• OpenAI. Reasoning guide, for the status field of a response. Not read here, as above.
• OpenAI. Pricing. Not read here, as above.
• Anthropic. Tool use with Claude. The tool_use and tool_result blocks and a complete round trip. Verified, page opened.
• Anthropic. Structured outputs. Verified, page opened.
• Anthropic. Stop reasons and fallback. Every way a response can end. Verified, page opened; this is the page’s current title.
• Anthropic. Pricing. Verified, page opened.

Module 9 · Reinforcement and Preference Learning

• Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press. Chapter 13, sections 13.1 to 13.4: the policy-gradient theorem, REINFORCE, and REINFORCE with a baseline. Search summary.
• Williams, R. J. (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning 8, 229–256. The original REINFORCE paper. Search summary.
• Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S. and Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems 30 (NIPS 2017). A reward learned from pairwise comparisons. Search summary.
• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J. and Lowe, R. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). InstructGPT; Module 9 reads the reward-model section and Module 10 the rest. Search summary.
• Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A. and Hooker, S. (2024). Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. Proceedings of the 62nd Annual Meeting of the ACL (Volume 1: Long Papers), 12248–12267. Verified, ACL Anthology.
• Bradley, R. A. and Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39(3/4), 324–345. Search summary.
• Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D. and Christiano, P. (2020). Learning to Summarize from Human Feedback. arXiv:2009.01325; published in Advances in Neural Information Processing Systems 33 (NeurIPS 2020) as “Learning to summarize with human feedback”. Also a Module 10 reading. Search summary.
• Gao, L., Schulman, J. and Hilton, J. (2023). Scaling Laws for Reward Model Overoptimization. Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 10835–10866 (preprint arXiv:2210.10760). Also a Module 10 reading. Verified, PMLR.

Module 10 · RLHF

• Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. Search summary.
• Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D. and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). Search summary.
• Schulman, J., Moritz, P., Levine, S., Jordan, M. and Abbeel, P. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. International Conference on Learning Representations (ICLR 2016). Search summary.
• Gheshlaghi Azar, M., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M. and Calandriello, D. (2024). A General Theoretical Paradigm to Understand Learning from Human Preferences. Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 238, 4447–4455. IPO. Verified, PMLR.
• Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E. and Raileanu, R. (2024). Understanding the Effects of RLHF on LLM Generalisation and Diversity. International Conference on Learning Representations (ICLR 2024). Search summary.
• Perez, E. et al. (2023). Discovering Language Model Behaviors with Model-Written Evaluations. Findings of the Association for Computational Linguistics: ACL 2023, 13387–13434. Verified, ACL Anthology.
• Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R. et al. (2024). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations (ICLR 2024). Search summary.
• Singhal, P., Goyal, T., Xu, J. and Durrett, G. (2023). A Long Way to Go: Investigating Length Correlations in RLHF. arXiv:2310.03716; published at COLM 2024. Search summary.
• OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774. Calibration before and after post-training; also cited in Module 11. Search summary.
• Also used: Ouyang et al. (2022), Stiennon et al. (2020), Gao et al. (2023) and Christiano et al. (2017) (Module 9).

Day 4 · Calibration, Decisions and Retrieval

Module 11 · Calibration

• Guo, C., Pleiss, G., Sun, Y. and Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning, PMLR 70, 1321–1330. Reliability diagrams, ECE and temperature scaling. Verified, PMLR.
• Gneiting, T. and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477), 359–378. Also a Module 12 reading. Search summary.
• Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G. and Tran, D. (2019). Measuring Calibration in Deep Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 38–41. Search summary.
• Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C. and Manning, C. D. (2023). Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 5433–5442. Verified, ACL Anthology.
• Geifman, Y. and El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. Advances in Neural Information Processing Systems 30 (NIPS 2017). Risk–coverage curves. Search summary.
• Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1), 1–3. Search summary; no stable link checked.
• Chow, C. K. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16(1), 41–46. Also cited in Module 12. Search summary; no stable link checked.
• DeGroot, M. H. and Fienberg, S. E. (1983). The Comparison and Evaluation of Forecasters. Journal of the Royal Statistical Society, Series D (The Statistician) 32(1/2), 12–22. Search summary.
• Desai, S. and Durrett, G. (2020). Calibration of Pre-trained Transformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 295–302. Verified, ACL Anthology.
• El-Yaniv, R. and Wiener, Y. (2010). On the Foundations of Noise-free Selective Classification. Journal of Machine Learning Research 11, 1605–1641. Search summary.
• Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society, Series B 69(2), 243–268. Search summary.
• Kadavath, S. et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221. Search summary.
• Lin, S., Hilton, J. and Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words. Transactions on Machine Learning Research. Search summary.
• Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D. and Lucic, M. (2021). Revisiting the Calibration of Modern Neural Networks. Advances in Neural Information Processing Systems 34 (NeurIPS 2021). Search summary.
• Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology 12, 595–600. The decomposition of the Brier score. Search summary; no stable link checked.
• Naeini, M. P., Cooper, G. F. and Hauskrecht, M. (2015). Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proceedings of the AAAI Conference on Artificial Intelligence 29(1). Search summary.
• Niculescu-Mizil, A. and Caruana, R. (2005). Predicting good probabilities with supervised learning. Proceedings of the 22nd International Conference on Machine Learning (ICML), 625–632. Search summary.
• Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B. and Snoek, J. (2019). Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). Search summary.
• Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, MIT Press, 61–74. Search summary; no stable link checked.
• Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J. and Hooi, B. (2024). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. International Conference on Learning Representations (ICLR 2024). Search summary.
• Zadrozny, B. and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 694–699. Search summary; no stable link checked.
• Also used: OpenAI (2023), the GPT-4 Technical Report (Module 10).

Module 12 · Calibrated Decisions: RLCD and Jev

TypeSafe has not published how RLCD works. The TypeSafe sources the briefing relies on are listed under TypeSafe and Jev: Public Sources below; nothing in this module’s toy experiment is TypeSafe’s method.

• Damani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y. and Andreas, J. (2026). Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. International Conference on Learning Representations (ICLR 2026). RLCR, a published reinforcement-learning method with a correctness-plus-Brier reward, by authors unrelated to TypeSafe. RLCR is not RLCD. Search summary.
• Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux. Part I, “Two Systems”, for the System 1 and System 2 framing only. Search summary; no stable link checked.
• Melnikoff, D. E. and Bargh, J. A. (2018). The Mythical Number Two. Trends in Cognitive Sciences 22(4), 280–293. A critique of the dual-process typology. Search summary.
• Stanovich, K. E. and West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences 23(5), 645–665. Search summary; no stable link checked.
• Also used: Gneiting and Raftery (2007) and Chow (1970) (Module 11).

Module 13 · Retrieval-Augmented Generation

• Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S. and Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). Search summary.
• Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D. and Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 6769–6781. The bi-encoder and in-batch negatives. Verified, ACL Anthology.
• Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. and Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, 157–173. Verified, ACL Anthology.
• Es, S., James, J., Espinosa Anke, L. and Schockaert, S. (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. Proceedings of the 18th Conference of the European Chapter of the ACL: System Demonstrations, 150–158. Verified, ACL Anthology.
• Thakur, N., Reimers, N., Rücklé, A., Srivastava, A. and Gurevych, I. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 Datasets and Benchmarks Track. Why BM25 remains the baseline to beat. Search summary.
• Cormack, G. V., Clarke, C. L. A. and Büttcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 758–759. Search summary.
• Nogueira, R. and Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. The cross-encoder reranker. Search summary.
• Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K. and Harshman, R. (1990). Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science 41(6), 391–407. Latent semantic analysis, the offline encoder of Lab 13. Search summary; no stable link checked.
• Also used: Robertson and Zaragoza (2009) (Module 1); Greshake et al. (2023) (Module 14); the source of llama-index-core and langchain-core (Library and API documentation).

Day 5 · Agents and the Capstone

Module 14 · Agents

• Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. and Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations (ICLR 2023). Search summary.
• Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T. and Fritz, M. (2023). Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec ’23), 79–90. Indirect prompt injection; also cited in Module 13. Search summary.
• Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M. and Tramèr, F. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. Search summary.
• Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., Ozoani, E., Paverd, A., Tramèr, F. and Volhejn, V. (2025). Design Patterns for Securing LLM Agents against Prompt Injections. arXiv:2506.08837. Search summary.
• Perez, F. and Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques for Language Models. NeurIPS 2022 ML Safety Workshop. Search summary.
• OWASP Foundation (2025). OWASP Top 10 for LLM Applications 2025, LLM01:2025 Prompt Injection. Search summary, from secondary pages only; the OWASP site was not reachable and no link was checked.
• Schluntz, E. and Zhang, B. (2024). Building effective agents. Anthropic, 19 December 2024. Workflows against agents, and the workflow patterns of section 9. The page now carries a note that its list of tools has changed since 2024. Verified, page opened 2026-10-06.
• Hadfield, J., Zhang, B., Lien, K., Scholz, F., Fox, J. and Ford, D. (2025). How we built our multi-agent research system. Anthropic, 13 June 2025. The coordinator–subagent design, task descriptions for subagents, and the token costs quoted in section 9. Verified, page opened 2026-10-06.
• Rajasekaran, P., Dixon, E., Ryan, C. and Hadfield, J. (2025). Effective context engineering for AI agents. Anthropic, 29 September 2025. Context rot, compaction, structured notes and subagents as context isolation. Verified, page opened 2026-10-06.
• Aizawa, K. (2025). Writing effective tools for agents, with agents. Anthropic, 11 September 2025. Tool descriptions, fewer tools, high-signal results and actionable errors (section 3). Verified, page opened 2026-10-06.
• Anthropic (2026). Claude Certified Architect – Foundations: Exam Guide, version 1.0, effective July 2026, exam code CCAR-F. Linked from the certification page; free preparation courses are listed on the prep-courses page. Module 14 quotes task statements 1.1, 1.3, 1.4, 2.1 and 5.2. The certification itself is open only to organizations in the Claude Partner Network; the guide and the courses are public. Verified, guide and both pages opened 2026-10-06.
• Ng, A. (2024a). Four AI agent strategies that improve GPT-4 and GPT-3.5 performance. The Batch, 20 March 2024. The first of five letters on agentic design patterns (reflection, tool use, planning, multi-agent collaboration), with the HumanEval comparison quoted in section 1. The comparison is compiled from other teams’ papers, not a controlled experiment. Verified, page opened 2026-10-06.
• Ng, A. (2024b). Agentic design patterns part 3: tool use. The Batch, 3 April 2024. Verified, page opened 2026-10-06.
• Ng, A. (2024c). Welcoming diverse approaches keeps machine learning strong. The Batch, 12 June 2024. “Agentic” as an adjective with degrees. Verified, page opened 2026-10-06.
• Ng, A. (2025b). Improve agentic performance with evals and error analysis, part 2. The Batch, 22 October 2025. Reading traces, measuring each step, and removing scaffolding as models improve. Ng’s course Agentic AI (DeepLearning.AI, 2025) teaches the same patterns in plain Python. Verified, pages opened 2026-10-06.
• Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X. and Zhou, D. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. International Conference on Learning Representations (ICLR 2024). Self-correction without outside feedback often fails to help. Verified, arXiv abstract page opened 2026-10-06.
• Trivedy, V. (2026a). The anatomy of an agent harness. LangChain blog, 10 March 2026. “Agent = Model + Harness”, and a list of what a harness contains. Verified, page opened 2026-10-06.
• Trivedy, V. (2026b). Improving deep agents with harness engineering. LangChain blog, 17 February 2026. A harness-only change on Terminal-Bench 2.0 with the model held fixed; the authors’ own report. Verified, page opened 2026-10-06.
• Schluntz, E. (2025). Claude SWE-bench performance. Anthropic, 6 January 2025. An agent as a model plus “software scaffolding”. Verified, page opened 2026-10-06.
• Böckeler, B. (2026). Harness engineering for coding agent users. martinfowler.com, 2 April 2026. The harness a tool ships with, and the outer harness its users add. Verified, page opened 2026-10-06.
• EleutherAI. lm-evaluation-harness. An evaluation harness, the other sense of the word. Verified, repository.
• Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547. Introduces ARC. Verified, arXiv abstract page opened 2026-10-06.
• Chollet, F., Knoop, M., Kamradt, G. and Landers, B. (2024). ARC Prize 2024: Technical Report. arXiv:2412.04604. Table 2 gives the GPT-4o and Claude 3.5 Sonnet baselines and harness results in section 1’s worked example. Verified, arXiv abstract page opened 2026-10-06.
• ARC Prize. The leaderboard and testing policy; OpenAI o3 breakthrough high score on ARC-AGI-Pub (December 2024); ARC Prize 2025 results and analysis; OpenAI’s GPT-6 Astra on ARC-AGI-3 (3 September 2026); ARC Prize 2026: ARC-AGI-3 Milestone Prize #1. Leaderboard figures were read from the page’s data file (generated 2026-10-01); they change as systems are added and repriced. Verified, pages opened 2026-10-06.
• Greenblatt, R. (2024). Getting 50% (SoTA) on ARC-AGI with GPT-4o. Redwood Research blog. Berman, J. (2024). How I came in first on ARC-AGI-Pub using Sonnet 3.5 with evolutionary test-time compute. Poetiq (2025). ARC-AGI-2 SOTA at half the cost, 5 December 2025. The three harnesses in section 1’s worked example, described by their authors. Verified, pages opened 2026-10-06.
• Also used: Liu et al. (2024) (Module 13); the LangGraph documentation and langgraph source, the Claude Agent SDK and the Model Context Protocol (Library and API documentation); langchain-typesafe (TypeSafe and Jev: public sources).

Module 15 · Capstone

• Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N. and Narayanan, A. (2024). AI Agents That Matter. arXiv:2407.01502; reported as published in Transactions on Machine Learning Research (2025). Report cost beside accuracy. Search summary.
• Rajpurkar, P., Jia, R. and Liang, P. (2018). Know What You Don’t Know: Unanswerable Questions for SQuAD. Proceedings of the 56th Annual Meeting of the ACL (Volume 2: Short Papers), 784–789. Verified, ACL Anthology.
• Kamath, A., Jia, R. and Liang, P. (2020). Selective Question Answering under Domain Shift. Proceedings of the 58th Annual Meeting of the ACL, 5684–5696. Verified, ACL Anthology.
• Dror, R., Baumer, G., Shlomov, S. and Reichart, R. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. Proceedings of the 56th Annual Meeting of the ACL (Volume 1: Long Papers), 1383–1392. Verified, ACL Anthology.
• Liang, P., Bommasani, R., Lee, T., Tsipras, D. et al. (2023). Holistic Evaluation of Language Models. Transactions on Machine Learning Research. Search summary.
• Also used: Modules 8 and 11 to 14 of this workshop.

Library and API Documentation

The labs pin their package versions in _variables.yml. The versions below are the ones the briefings were checked against.

• Hugging Face Transformers and PEFT: the documentation pages listed under Modules 6 and 7. The briefings were checked against transformers 5.18.0 and peft 0.21.2. Versions verified, PyPI; documentation pages: search summary.
• LlamaIndex. Documentation, and the installed source of llama-index-core 0.14.25, which Module 13 was checked against. Version verified, PyPI; documentation site: search summary (not reachable from the build environment).
• LangChain. Documentation, and the installed source of langchain-core 1.6.6. Version verified, PyPI; documentation site: search summary.
• LangGraph. Persistence and Interrupts, and the installed source of langgraph 1.2.12. Module 14’s behavior was checked against the source and by running small graphs. Version verified, PyPI. Documentation pages: not read; search confirms they exist.
• OpenAI and Anthropic: the documentation pages listed under Module 8.
• Anthropic. Claude Agent SDK: the overview, agent-loop, hooks, subagents and sessions pages, for the optional mapping in Module 14, section 9. Verified, pages opened 2026-10-06; no SDK code was run.
• Model Context Protocol. Introduction and server concepts; LangChain’s MCP adapter, now installed as langchain[mcp]. Verified, pages opened 2026-10-06.

TypeSafe and Jev: Public Sources

TypeSafe’s documentation names its training method, RLCD (“Reinforcement learning for calibrated decisions”), states its aim and says that Jev “is trained with RLCD to return calibrated decisions”. TypeSafe has not published how RLCD works: we have found no paper, reward function, training dataset or reliability data from TypeSafe. Module 12 separates what TypeSafe states from our illustration, and lists the remaining third-party reports about RLCD as unverified; those third-party pages are not TypeSafe sources and are not listed here.

• TypeSafe AI. typesafe-sdk 0.7.2, the Python SDK for Jev (MIT license; import name typesafe_sdk). Used in Labs 12 to 14. Verified, PyPI.
• TypeSafe AI. typesafe-ai/skills, the file skills/typesafe-ai/SKILL.md: TypeSafe’s guidance on typed questions, probabilities and confidence, and the sentence “System One models are trained for calibrated decisions; validate their performance in the target domain.” Verified, GitHub repository.
• TypeSafe AI. system-one-adapter 0.2.1, a drop-in replacement for the SDK client backed by LLM APIs, for comparing the two. Verified, PyPI.
• TypeSafe AI. WorkflowEvals, the code behind TypeSafe’s published workflow evaluations. Verified, GitHub repository.
• langchain-typesafe 0.0.1a3, a LangChain integration for TypeSafe classifiers (a pre-release version). Used in Lab 14. Verified, PyPI.
• TypeSafe AI. Documentation: the AI primer (what RLCD is and what it optimizes for), System One (the training sentence, and what calibration does not guarantee), Models (Jev is trained with RLCD; price, rate limits and aliases), Confidence (the confidence formulas for choice and score answers), Jev 1.13 jaggedness and How to build with TypeSafe. Quoted in Module 12, section 5. Verified, pages read 2026-10-06, also as the site’s full-text export; the quotations await the workshop lead’s sign-off.
• TypeSafe AI. A blog post at typesafe.ai/blog/introducing-system-one-models-and-jev, which the documentation links to for Jev’s pricing. It may be the announcement that third parties date to September 2026. Not read.