M03.5 CONNECT THE MECHANISM
Protect the examples used to judge a model
An X-ray model scores 94%, then fails at a new hospital. It had learned to recognize the hospital, not the disease. Learn to split data so tests catch tricks like this.
LESSON OVERVIEW15 min lesson
Lesson overview
An X-ray model scores 94%, then fails at a new hospital. It had learned to recognize the hospital, not the disease. Learn to split data so tests catch tricks like this.
What you’ll explore
- Design train/validation/test boundaries for grouped or time-ordered data and identify leakage through preprocessing or repeated test use.
GO TO THE SOURCE
Original explanations, connected to the research.
scikit-learn — inconsistent preprocessing and data leakageDive into Deep Learning — environment and distribution shiftVariable generalization performance of a deep learning model to detect pneumonia in chest radiographs (Zech et al., 2018)Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.