M15.7 CONNECT THE MECHANISM
Match information flow to the learning objective
BERT fills in blanks, GPT-2 continues text, and T5 turns every task into text-to-text. See how one choice, what each position may look at, splits transformers into three families.
LESSON OVERVIEW13 min lesson
Lesson overview
BERT fills in blanks, GPT-2 continues text, and T5 turns every task into text-to-text. See how one choice, what each position may look at, splits transformers into three families.
What you’ll explore
- Encoder-only, decoder-only, and encoder-decoder transformers share mechanisms but differ in permitted context, output organization, and common training objectives.
GO TO THE SOURCE
Original explanations, connected to the research.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)Language Models are Unsupervised Multitask Learners (GPT-2; Radford et al., 2019)Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5; Raffel et al., 2019)Attention Is All You Need (Vaswani et al., 2017)Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.