M32.7 CONNECT THE MECHANISM
Choose expensive experiments using a surrogate model
Each training run costs six GPU-hours and you can afford twenty. Learn to pick the next run with expected improvement, cut losers early, and see why architecture search went from 2,000 GPU-days to four.
LESSON OVERVIEW13 min lesson
Lesson overview
Each training run costs six GPU-hours and you can afford twenty. Learn to pick the next run with expected improvement, cut losers early, and see why architecture search went from 2,000 GPU-days to four.
What you’ll explore
- Bayesian optimization models an expensive objective and selects evaluations with an acquisition rule; AutoML adds search spaces, resource allocation, and validation procedures around model selection.
GO TO THE SOURCE
Original explanations, connected to the research.
A Tutorial on Bayesian Optimization (Frazier, 2018)Practical Bayesian Optimization of Machine Learning Algorithms (Snoek, Larochelle & Adams, 2012)Non-stochastic Best Arm Identification and Hyperparameter Optimization (Jamieson & Talwalkar, 2016)Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (Li et al., 2018)Neural Architecture Search with Reinforcement Learning (Zoph & Le, 2017)DARTS: Differentiable Architecture Search (Liu, Simonyan & Yang, 2019)Random Search and Reproducibility for Neural Architecture Search (Li & Talwalkar, 2019)Dive into Deep Learning — hyperparameter optimizationSuggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.