M27.3 CONNECT THE MECHANISM
Turn logits into a controlled generation procedure
A bot stuck repeating itself, JSON that won't parse, and a bug that won't reproduce with the same seed. All three are fixed after the model, where scores become tokens.
LESSON OVERVIEW18 min lesson
Lesson overview
A bot stuck repeating itself, JSON that won't parse, and a bug that won't reproduce with the same seed. All three are fixed after the model, where scores become tokens.
What you’ll explore
- Explain how serving-time decoding controls (repetition penalties, logit bias, stop sequences, constraint masks, beam search, and seeded sampling) change which tokens are chosen without changing the model, and why the same seed can still give different outputs.
GO TO THE SOURCE
Original explanations, connected to the research.
The Curious Case of Neural Text Degeneration (Holtzman et al., 2019)CTRL: A Conditional Transformer Language Model for Controllable Generation (Keskar et al., 2019), penalized samplingEfficient Guided Generation for Large Language Models (Willard & Louf, 2023), the Outlines methodHugging Face Transformers documentation, generation strategiesDefeating Nondeterminism in LLM Inference (Horace He and Thinking Machines Lab, 2025)Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.