M14.5 CONNECT THE MECHANISM
From token IDs to learned vectors
Is "coffee" closer to "tea" or to "coffin"? See how a model learns lists of numbers that put tea next door, and why king − man + woman ≈ queen needs fine print.
LESSON OVERVIEW14 min lesson
Lesson overview
Is "coffee" closer to "tea" or to "coffin"? See how a model learns lists of numbers that put tea next door, and why king − man + woman ≈ queen needs fine print.
What you’ll explore
- Distinguish an integer token ID, a learned embedding, and a representation that changes with context.
GO TO THE SOURCE
Original explanations, connected to the research.
Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013)Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013), skip-gram with negative samplingLinguistic Regularities in Continuous Space Word Representations (Mikolov, Yih & Zweig, 2013)Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (Bolukbasi et al., 2016)Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (Nissim, van Noord & van der Goot, 2020)Deep Contextualized Word Representations (Peters et al., 2018), ELMoBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)PyTorch — embedding lookupSuggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.