Resources

What to read or watch next, curated

This page is the course’s learning library: a short, curated set of resources for going further, organized by the part of the course they extend. It is not the evidence ledger. That lives in sources/source-map.md and sources/claims.csv, which answer a different question (“what supports this factual statement?”) and list every cited source with its exact locator. Some papers appear in both places, because a primary source can also be good reading; where that is so, the entry says why it is recommended as reading.

Every link on this page was opened on 2026-09-15. Durations are given only where the resource’s own page states them. Vendor documentation changes without notice: for those entries, treat the description as of the access date, and reread the current page rather than memorizing its syntax.

Prefer video? Start here

A short recommended video route through this page, for a general reader. Each item’s full entry, with its caveats, is below.

  1. Karpathy, Intro to Large Language Models (2023). About an hour; general audience. The source of the course’s “code plus parameters” picture, and a compact way to see inference, training, fine-tuning, and tools as one system.
  2. Seitz, Large Language Models from scratch and Part 2. Two short animated videos; general audience. From autocomplete to transformers with the least possible machinery.
  3. 3Blue1Brown, Transformers and Attention (Deep Learning chapters 5–6). Visual, introductory. The animated counterparts of the course’s softmax, temperature, and attention pages.
  4. Karpathy, Deep Dive into LLMs like ChatGPT (2025). About three and a half hours; general audience. The comprehensive treatment, and the source of the course’s training-stack structure, classroom analogy, and parameters-versus-context memory analogy.
  5. 3Blue1Brown, Neural Networks chapters 1–5. Visual, introductory. For readers who then want gradient descent and backpropagation animated in full.

Foundations: neural networks and learning

3Blue1Brown, Neural Networks (chapters 1–5)

  • Creator: Grant Sanderson (3Blue1Brown)
  • Format: video series with written adaptations; five chapters from 2017 (“But what is a Neural Network?”, “Gradient descent, how neural networks learn”, “Analyzing our neural network”, “What is backpropagation really doing?”, “Backpropagation calculus”)
  • Level: introductory, strongly visual; no code
  • Useful for: seeing the multi-layer case of everything the neural-networks lesson works by hand on one unit: units, layers, gradient descent, and backpropagation, animated on a real digit-recognition network
  • Limitation: the examples are image classification, not language; the site’s own chapter numbering restarts for its later transformer lessons, so “chapter 5” is ambiguous across the two series
  • Link: https://www.3blue1brown.com/lessons/neural-networks (chapter 1; each lesson links to the next)

Michael Nielsen, Neural Networks and Deep Learning

  • Creator: Michael Nielsen
  • Format: free online book, six chapters (Determination Press, 2015; last updated December 2019)
  • Level: introductory to intermediate; assumes comfort with algebra and some calculus, and reads code
  • Useful for: a patient, complete derivation of backpropagation (chapter 2) and a visual argument for why networks can represent arbitrary functions (chapter 4); a strong free text for readers who want the mathematics of the neural-networks lesson in full
  • Limitation: the code is written in Python 2.7, which is no longer supported, and the book predates transformers entirely
  • Link: http://neuralnetworksanddeeplearning.com/

Andrej Karpathy, Neural Networks: Zero to Hero

  • Creator: Andrej Karpathy
  • Format: eight video lectures, building everything from scratch in code; the syllabus page states each lecture’s length (the first, on backpropagation, is 2h25m; the GPT lecture is 1h56m; the tokenizer lecture is 2h13m)
  • Level: intermediate; requires solid Python and introductory calculus, and is meant to be followed at a keyboard
  • Useful for: turning the course’s concepts into working code: a hand-written backpropagation engine, a character-level language model, a small GPT, and a byte pair encoding tokenizer
  • Limitation: long and hands-on; the page describes the course as ongoing, so its scope may change
  • Link: https://karpathy.ai/zero-to-hero.html

Transformers and language models

3Blue1Brown, Transformers, the tech behind LLMs and Attention in transformers, step-by-step

  • Creator: Grant Sanderson (3Blue1Brown); text adaptations by Justin Sun
  • Format: two video lessons with written versions (Deep Learning chapters 5 and 6, April 2024)
  • Level: introductory, strongly visual
  • Useful for: the animated counterparts of the course’s softmax, temperature, and attention pages: why softmax needs a normalizing sum, how temperature reshapes a distribution, and how query, key, and value vectors combine
  • Limitation: leans on the “what is this token looking for” reading of attention more freely than this course does, and does not cover the multinomial-logit bridge or the training objective
  • Links: https://www.3blue1brown.com/lessons/gpt, https://www.3blue1brown.com/lessons/attention

Steve Seitz, Large Language Models from scratch and Part 2, with Jay Mody’s GPT in 60 Lines of NumPy

  • Creator: Steve Seitz (Professor, Paul G. Allen School of Computer Science & Engineering, University of Washington), on his channel Graphics in 5 Minutes; Jay Mody for the code post
  • Format: two short animated videos (July 2022) that go from autocomplete and n-grams to neural networks, then word embeddings, transformers, stacked attention layers, and training data; plus a blog post (January 2023) that implements GPT-2 inference in about sixty lines of NumPy and loads OpenAI’s released weights to generate text. The University of Washington’s CSE 163, an undergraduate data-science course, uses exactly this pairing in its language-models lesson. [S-80] [S-81]
  • Level: introductory (videos); intermediate, reads Python (code)
  • Useful for: a short visual explanation of the whole mechanism, followed by seeing how compact the inference computation is once the parameters exist
  • Limitation: the videos are from 2022 and stop before post-training and agents; the code is inference only, with training discussed conceptually
  • Links: https://www.youtube.com/watch?v=lnA9DMvHtfI, https://www.youtube.com/watch?v=YDiSFS-yHwk, https://jaykmody.com/blog/gpt-from-scratch/

Jay Alammar, The Illustrated Transformer

  • Creator: Jay Alammar
  • Format: illustrated blog post (June 2018; positional-encoding section updated July 2020), with a companion post The Illustrated GPT-2 (August 2019) on the decoder-only case
  • Level: introductory to intermediate; light mathematics
  • Useful for: a diagram-by-diagram walk through the original encoder–decoder Transformer, including multi-head attention and the residual and normalization structure the attention deep dive describes in words
  • Limitation: describes the 2017 translation model, not a modern decoder-only language model; the GPT-2 post’s author lists his own simplifications (words and tokens used interchangeably, inference shown rather than training)
  • Links: https://jalammar.github.io/illustrated-transformer/, https://jalammar.github.io/illustrated-gpt2/

Andrej Karpathy, Intro to Large Language Models

  • Creator: Andrej Karpathy
  • Format: single video (YouTube, published November 2023; listed at 59:48), described by its author as “a 1 hour general-audience introduction to Large Language Models” [S-75]
  • Level: introductory, plain language, no equations
  • Useful for: the mental model the short story opens with: a runnable model as code plus a parameter checkpoint; then inference, training, fine-tuning into an assistant, tool use, and the surrounding system in one sitting; a good first video before the 2025 deep dive
  • Limitation: recorded in 2023, so product details and the “future” sections have aged; use it for the conceptual framing, not for current product facts
  • Link: https://www.youtube.com/watch?v=zjkBMFhNj_g

Andrej Karpathy, Deep Dive into LLMs like ChatGPT

  • Creator: Andrej Karpathy
  • Format: single long video (YouTube, published February 2025; listed at 3:31:24), described by its author as “a general audience deep dive” covering “the full training stack” [S-77]
  • Level: introductory, plain language, no equations
  • Useful for: the whole pipeline in one sitting, from pretraining data and tokenization through inference, post-training, hallucinations and tool use, reinforcement learning, and DeepSeek-R1, with real tokenizers and real generation loops on screen. This course borrows three things from it, with attribution: the order of the training stack, the classroom analogy for the training stages, and the parameters-versus-context memory analogy [S-76] [S-77]
  • Limitation: no mathematics, and long enough to watch in segments; its analogies are teaching devices, which the course labels as such
  • Link: https://www.youtube.com/watch?v=7xTGNNLPyMI

Vaswani et al., Attention Is All You Need

  • Creator: Ashish Vaswani and colleagues (Google Brain, Google Research, University of Toronto)
  • Format: paper, arXiv:1706.03762 (v7, 2023-08-02); Sections 3 and 4 are the parts the course uses
  • Level: technical
  • Useful for: the primary source and, unusually for a primary source, still a clear, short statement of the architecture: the attention equations, the block structure, and Table 1’s comparison of recurrent and self-attention layers. Recommended as reading, not only as evidence, because the course’s why-transformers and attention pages are built directly on it
  • Limitation: a machine-translation encoder–decoder model; the decoder-only form used by language models gets one paragraph
  • Link: https://arxiv.org/abs/1706.03762

Bengio, Ducharme, Vincent, and Jauvin, A Neural Probabilistic Language Model

  • Creator: Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin
  • Format: paper, Journal of Machine Learning Research 3 (2003), 1137–1155; Sections 1 and 2 carry the ideas
  • Level: technical, but the first two sections are readable without the rest
  • Useful for: a clear early statement of two ideas the course depends on, a learned vector per vocabulary entry and next-token maximum likelihood, in a setting simple enough to see them without a transformer around them
  • Limitation: the network is not a transformer and the vocabulary is words, not subword tokens
  • Link: https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

Hugging Face, LLM Course

  • Creator: Hugging Face
  • Format: free web course, twelve chapters, with runnable code using the Hugging Face libraries; originally an NLP course, since extended to large language models, fine-tuning, and reasoning models
  • Level: intermediate; the course states that it requires good Python and is best taken after an introductory deep-learning course
  • Useful for: working with real tokenizers, pretrained models, and fine-tuning code after the conceptual lessons here
  • Limitation: tied to one library ecosystem, and revised continuously, so chapter contents move
  • Link: https://huggingface.co/learn/llm-course/chapter1/1

Training and post-training

Ouyang et al., Training language models to follow instructions with human feedback

  • Creator: Long Ouyang and colleagues (OpenAI)
  • Format: paper, arXiv:2203.02155 (v1, 2022-03-04); the abstract and Section 3 describe the recipe
  • Level: technical
  • Useful for: the primary description of the supervised, then reward-model, then reinforcement-learning recipe; the paper reports that, in its evaluations, outputs from a much smaller post-trained model were preferred to outputs from the much larger pretrained GPT-3 model. Recommended as reading because Section 3 is a concrete, well-labeled account of what each stage does
  • Limitation: one provider’s 2022 recipe; current recipes vary, and the paper’s “RLHF” is one route, not post-training as a whole
  • Link: https://arxiv.org/abs/2203.02155

Rafailov et al., Direct Preference Optimization

  • Creator: Rafael Rafailov and colleagues (Stanford)
  • Format: paper, arXiv:2305.18290 (v3, 2024-07-29); the introduction carries the idea, the body the derivation
  • Level: technical; the derivation needs some comfort with optimization
  • Useful for: seeing why the separate reward model can be folded into a direct loss on preference pairs; a clear demonstration that preference-based post-training can be implemented without the classic separate reward-model and online reinforcement-learning pipeline
  • Limitation: its comparison results are for the tasks and scales tested; it is not a verdict that one route has replaced the other
  • Link: https://arxiv.org/abs/2305.18290

DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

  • Creator: DeepSeek-AI
  • Format: paper, arXiv:2501.12948 (v2, 2026-01-04); the abstract and introduction distinguish R1-Zero from R1
  • Level: technical
  • Useful for: a detailed public account of reinforcement learning against rule-based, checkable rewards producing long reasoning traces, and of why the pure-RL model and the released multi-stage model are different objects
  • Limitation: one lab’s recipe; the traces it shows are outputs, not transcripts of internal computation, and other providers publish less
  • Link: https://arxiv.org/abs/2501.12948

Agents and practical use

Anthropic, Building effective agents

  • Creator: Erik Schluntz and Barry Zhang (Anthropic)
  • Format: engineering article, published 2024-12-19, accessed 2026-09-15
  • Level: practitioner
  • Useful for: the workflow-versus-agent vocabulary this course uses, a short catalogue of composable patterns, and a clear case for the simplest system that works
  • Limitation: one provider’s guidance for developers; the page now carries its own note that much of the tooling landscape it describes has changed since December 2024 [S-73]
  • Link: https://www.anthropic.com/engineering/building-effective-agents

Anthropic, Effective context engineering for AI agents

  • Creator: Anthropic Applied AI team
  • Format: engineering article, published 2025-09-29, accessed 2026-09-15
  • Level: practitioner
  • Useful for: the working definition of context as a finite resource and the named practices (just-in-time loading, progressive disclosure, compaction, structured notes) that the practical guide uses
  • Limitation: one provider’s guidance about its own models, not a peer-reviewed result; the course pairs it with the empirical study cited in the tokens-and-context deep dive
  • Link: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Lilian Weng, LLM Powered Autonomous Agents

  • Creator: Lilian Weng
  • Format: long blog post, June 2023
  • Level: intermediate; survey style, with references
  • Useful for: an independent, vendor-neutral survey of the agent pattern (planning, memory, tool use) with case studies and a candid section on challenges; a useful counterweight to reading only provider guidance
  • Limitation: written in 2023, before most current agent tooling, and it uses “memory” more loosely than this course, which separates parameters, context, and external state
  • Link: https://lilianweng.github.io/posts/2023-06-23-agent/

Hugging Face, AI Agents Course

  • Creator: Hugging Face
  • Format: free web course, units 0–4 plus bonus units, with code in several agent frameworks (smolagents, LlamaIndex, LangGraph); the page estimates three to four hours of work per week
  • Level: intermediate; assumes basic Python and basic familiarity with language models
  • Useful for: building the tool-call loop from this course’s agents lesson yourself, in more than one framework
  • Limitation: framework-specific code that will date; the page describes itself as a living project
  • Link: https://huggingface.co/learn/agents-course/unit0/introduction

Current provider prompting guidance (Anthropic, OpenAI, Google)

Anthropic, Claude Code documentation (best practices, skills, permissions)

Deeper courses and reference

Stanford CS224N, Natural Language Processing with Deep Learning

  • Creator: Stanford University (Winter 2026 offering taught by Diyi Yang and Yejin Choi)
  • Format: university lecture course; slides and notes are public, and the course page links a complete public lecture-video playlist from 2024, while the current term’s videos are for enrolled students
  • Level: advanced undergraduate or graduate; assumes calculus, linear algebra, probability, and Python
  • Useful for: the full academic treatment behind this course’s conceptual path, from word vectors and recurrent networks to transformers, pretraining, and post-training, with problem sets
  • Limitation: substantial time commitment; the public videos lag the current syllabus by a year or more
  • Link: https://web.stanford.edu/class/cs224n/

Stanford CS336, Language Modeling from Scratch

  • Creator: Stanford University (Tatsunori Hashimoto and Percy Liang)
  • Format: university lecture course with implementation assignments; the Spring 2026 page links a public lecture playlist. The course walks through “every aspect of language model creation, including data collection and cleaning for pre-training, transformer model construction, model training, and evaluation before deployment”, with assignments on tokenizer and model construction, systems and GPU efficiency, scaling, converting raw web crawls into pretraining data, and supervised fine-tuning with reasoning-oriented reinforcement learning. [S-79]
  • Level: advanced and implementation-heavy; the course lists Python proficiency, deep-learning and systems experience, calculus, linear algebra, probability, and prior machine learning as prerequisites
  • Useful for: the complete engineering lifecycle behind this course’s conceptual one, including the data pipelines, GPUs, and parallelism that this course only names
  • Limitation: far more implementation than most readers of this site need; treat it as the reference for what the short story’s lifecycle looks like at full scale
  • Link: https://cs336.stanford.edu/