ClassroomThe classroom
Back to the lesson libraryMECHANISM · 15 MIN
M18.2 CONNECT THE MECHANISM

Score experts and choose a small set

One small matrix decides which experts see each token. Compute its scores, keep the top two, rescale them, and mix the outputs, by hand and in code.

LESSON OVERVIEW15 min lesson

Lesson overview

One small matrix decides which experts see each token. Compute its scores, keep the top two, rescale them, and mix the outputs, by hand and in code.

What you’ll explore

  • Trace how a router picks the top-k experts and weights their outputs, including whether the kept weights are renormalized.
GO TO THE SOURCE

Original explanations, connected to the research.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)Switch Transformers (Fedus, Zoph & Shazeer, 2021)Mixtral of Experts (Jiang et al., 2024)DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)
Suggest a correction

A precise note can make an explanation better.

Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.

The file includes this note, the scene title, and lesson metadata. Your saved progress and quiz responses are excluded. Download before leaving or reloading to keep your note.