M15.5 CONNECT THE MECHANISM
Give a transformer information about order
To plain self-attention, "dog bites man" and "man bites dog" are the same news. See the three tricks, waves, rotations, and distance penalties, that give transformers a sense of order.
LESSON OVERVIEW12 min lesson
Lesson overview
To plain self-attention, "dog bites man" and "man bites dog" are the same news. See the three tricks, waves, rotations, and distance penalties, that give transformers a sense of order.
What you’ll explore
- Explain why self-attention needs position information, and trace additive (learned or sinusoidal) positions, rotary position embeddings, and distance biases.
GO TO THE SOURCE
Original explanations, connected to the research.
Attention Is All You Need, section 3.5 (Vaswani et al., 2017)RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Press, Smith & Lewis, 2021)Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., 2023)Transformer Language Models without Positional Encodings Still Learn Positional Information (Haviv et al., 2022)LLaMA: Open and Efficient Foundation Language Models (Touvron et al., 2023)Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.