ClassroomThe classroom
Back to the lesson libraryMECHANISM · 14 MIN
M09.5 CONNECT THE MECHANISM

Evaluate the procedure that chose the model

Try 50 settings, keep the best, and your score is probably inflated even when nothing improved. Learn to choose a model without fooling yourself, and to measure the whole choosing procedure.

LESSON OVERVIEW14 min lesson

Lesson overview

Try 50 settings, keep the best, and your score is probably inflated even when nothing improved. Learn to choose a model without fooling yourself, and to measure the whole choosing procedure.

What you’ll explore

  • Apply a stated validation-selection rule with quality and operating constraints while protecting final-test independence.
GO TO THE SOURCE

Original explanations, connected to the research.

scikit-learn — common evaluation and preprocessing pitfallsscikit-learn — nested versus non-nested cross-validationOn the State of the Art of Evaluation in Neural Language Models (Melis, Dyer & Blunsom, 2018)Are GANs Created Equal? A Large-Scale Study (Lucic et al., 2018)Liang et al. — Holistic Evaluation of Language Models
Suggest a correction

A precise note can make an explanation better.

Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.

The file includes this note, the scene title, and lesson metadata. Your saved progress and quiz responses are excluded. Download before leaving or reloading to keep your note.