M33.3 CONNECT THE MECHANISM
Release changes with evidence and a recovery path
A new prompt passes every test, then misleads real visitors. Learn the release ladder, how long a 5% canary must run, and how to roll back everything at once.
LESSON OVERVIEW14 min lesson
Lesson overview
A new prompt passes every test, then misleads real visitors. Learn the release ladder, how long a 5% canary must run, and how to roll back everything at once.
What you’ll explore
- Move a release through offline gates, shadow mode, a canary, and an A/B test; size a canary to catch a given rise in errors; and roll back the whole release, including its prompt and retrieval settings.
GO TO THE SOURCE
Original explanations, connected to the research.
The ML Test Score (Breck et al., 2017)Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Kohavi, Tang & Xu, 2020)SEC order on Knight Capital Americas LLC (Release No. 34-70694, 2013)Google SRE Workbook, "Canarying Releases"Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.