ClassroomThe classroom
Back to the lesson libraryMECHANISM · 12 MIN
M10.5 CONNECT THE MECHANISM

Constrain a solution or change the update

One typo in one order can fling a model's weights into nonsense. Meet the three safeguards (penalties, weight decay, and clipping) and learn which problem each one solves.

LESSON OVERVIEW12 min lesson

Lesson overview

One typo in one order can fling a model's weights into nonsense. Meet the three safeguards (penalties, weight decay, and clipping) and learn which problem each one solves.

What you’ll explore

  • Penalties, decoupled weight decay, projection, and gradient clipping act at different points in learning; they should not be treated as interchangeable safeguards.
GO TO THE SOURCE

Original explanations, connected to the research.

Dive into Deep Learning — authors’ open textbookDecoupled Weight Decay RegularizationOn the difficulty of training recurrent neural networks (Pascanu, Mikolov & Bengio, 2013)
Suggest a correction

A precise note can make an explanation better.

Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.

The file includes this note, the scene title, and lesson metadata. Your saved progress and quiz responses are excluded. Download before leaving or reloading to keep your note.