B18 A SMALL STEP SIDEWAYS
When the computer cannot store the exact number
In Python, 0.1 + 0.2 isn't 0.3, and in 16-bit numbers 2048 + 1 is 2048. See where a computer's numbers run out, and the tricks that keep softmax and training finite.
LESSON OVERVIEW12 min lesson
Lesson overview
In Python, 0.1 + 0.2 isn't 0.3, and in 16-bit numbers 2048 + 1 is 2048. See where a computer's numbers run out, and the tricks that keep softmax and training finite.
What you’ll explore
- Explain rounding, overflow, and underflow in floating-point formats, compute a softmax stably by subtracting the largest score, and compare approximate results with a tolerance.
GO TO THE SOURCE
Original explanations, connected to the research.
Deep Learning — numerical computationSuggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.