Project: build a tiny next-token model and checkpoint ledger
Teach a model 98 bird names and it will invent new ones, like "startbird" and "bullover". Build it twice, from tally marks and then from learned vectors, and measure which one is less surprised.
Lesson overview
Teach a model 98 bird names and it will invent new ones, like "startbird" and "bullover". Build it twice, from tally marks and then from learned vectors, and measure which one is less surprised.
What you’ll explore
- Build a count-based bigram model and a small neural next-token model, compare them by held-out perplexity across saved checkpoints, and sample from them with temperature.
Original explanations, connected to the research.
Dive into Deep Learning — Language ModelsA Neural Probabilistic Language Model (Bengio et al., 2003)Prepare a project review
Give someone the context to challenge your thinking.
Download the authored brief, deliverables, and review criteria with your notes. Use the worksheet for self or peer review. Project artifacts are ungraded here, and this review does not award mastery credit.
Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.