ClassroomThe classroom
Back to the lesson libraryMECHANISM · 14 MIN
M14.3 CONNECT THE MECHANISM

Represent text with counts and short contexts

In 1948 Claude Shannon generated fake English with nothing but a book and a rule. Build his kind of language model from tally marks, then measure how surprised it gets.

LESSON OVERVIEW14 min lesson

Lesson overview

In 1948 Claude Shannon generated fake English with nothing but a book and a rule. Build his kind of language model from tally marks, then measure how surprised it gets.

What you’ll explore

  • Calculate a conditional next-token distribution with and without add-one smoothing and explain its limits.
GO TO THE SOURCE

Original explanations, connected to the research.

A Mathematical Theory of Communication (Shannon, 1948)Speech and Language Processing, 3rd edition draft, chapter 3 "N-gram Language Models" (Jurafsky & Martin)A Neural Probabilistic Language Model (Bengio et al., 2003)Dive into Deep Learning — language models
Suggest a correction

A precise note can make an explanation better.

Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.

The file includes this note, the scene title, and lesson metadata. Your saved progress and quiz responses are excluded. Download before leaving or reloading to keep your note.