ClassroomThe classroom
Back to the lesson libraryMECHANISM · 15 MIN
M18.3 CONNECT THE MECHANISM

Follow a token through dispatch, experts, and combination

Six tokens, four experts, twelve assignments. Sort the work by expert, run each expert once, and get every result back to the right token.

LESSON OVERVIEW15 min lesson

Lesson overview

Six tokens, four experts, twelve assignments. Sort the work by expert, run each expert once, and get every result back to the right token.

What you’ll explore

  • Dispatch groups token representations by selected expert, executes each expert, and combines returned outputs in the original token order using routing weights.
GO TO THE SOURCE

Original explanations, connected to the research.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)Switch Transformers (Fedus, Zoph & Shazeer, 2021)MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (Gale et al., 2022)
Suggest a correction

A precise note can make an explanation better.

Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.

The file includes this note, the scene title, and lesson metadata. Your saved progress and quiz responses are excluded. Download before leaving or reloading to keep your note.