M17.5 CONNECT THE MECHANISM
Save memory by sharing state or recreating it
Plain data parallelism keeps 256 identical copies of the optimizer. Remove the duplicates and the 7B model's states drop from 112 GB to under half a gigabyte per GPU.
LESSON OVERVIEW14 min lesson
Lesson overview
Plain data parallelism keeps 256 identical copies of the optimizer. Remove the duplicates and the 7B model's states drop from 112 GB to under half a gigabyte per GPU.
What you’ll explore
- State sharding, activation checkpointing, and offloading reduce different memory allocations and trade them for communication, recomputation, or transfer latency.
GO TO THE SOURCE
Original explanations, connected to the research.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., 2019)PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (Zhao et al., 2023)Training Deep Nets with Sublinear Memory Cost (Chen et al., 2016)Reducing Activation Recomputation in Large Transformer Models (Korthikanti et al., 2022)ZeRO-Offload: Democratizing Billion-Scale Model Training (Ren et al., 2021)Suggest a correction
A precise note can make an explanation better.
Choose the scene and describe what needs attention. Download a feedback file to share through a channel you already use. This page does not send feedback or connect you with a reviewer.