M15.8 CONNECT THE MECHANISM
Follow one token through a transformer
Follow "a bird sings" through GPT-2 small, from token IDs to the scores that pick the next word, with the shape of every tensor along the way. Then see why long prompts get expensive.
LESSON OVERVIEW13 min lesson
Lesson overview
Follow "a bird sings" through GPT-2 small, from token IDs to the scores that pick the next word, with the shape of every tensor along the way. Then see why long prompts get expensive.
What you’ll explore
- Trace a decoder-only transformer from token representations through contextual updates to vocabulary scores.
GO TO THE SOURCE
Original explanations, connected to the research.
Language Models are Unsupervised Multitask Learners (GPT-2; Radford et al., 2019)Attention Is All You Need (Vaswani et al., 2017)Dive into Deep Learning — transformer architecturenanoGPT, a minimal GPT-2 implementation (Andrej Karpathy)FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022)Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.