zoom_in Click to enlarge

DeepSpeed

▲ ▼

language

description DeepSpeed Overview

Large-model training is the central purpose of DeepSpeed, an open-source deep learning optimization library developed by Microsoft. It supports distributed execution so workloads can use multiple GPUs and machines. Its ZeRO approach reduces duplicated training state across devices, helping make better use of available memory when model training exceeds the capacity of a single GPU. Mixed-precision training and communication optimizations form part of its approach to handling demanding workloads, including large language models.

DeepSpeed also offers inference optimizations for running trained models. It suits teams with substantial deep learning workloads and the expertise to configure distributed infrastructure. Its value is practical memory and execution management, with results depending on hardware and model design.

help DeepSpeed FAQ

What is DeepSpeed used for when training large language models?

DeepSpeed is Microsoft's open-source optimization library for PyTorch training and inference. It is best known for ZeRO, which partitions optimizer states, gradients, and parameters across GPUs to reduce memory pressure.

What is ZeRO in DeepSpeed?

ZeRO stands for Zero Redundancy Optimizer. Its staged approach, commonly called ZeRO-1, ZeRO-2, and ZeRO-3, reduces duplicated model-training state across distributed GPUs.

How does DeepSpeed compare with PyTorch FSDP?

Both DeepSpeed ZeRO-3 and PyTorch Fully Sharded Data Parallel target memory-efficient distributed training. DeepSpeed is often chosen for its mature JSON configuration system, optimizer offload features, and long history in large Microsoft model-training work.

Can DeepSpeed run on a single GPU?

Yes, DeepSpeed can be used on a single GPU, especially for optimizer or CPU offload experiments. Its biggest advantage usually appears in multi-GPU or multi-node training where model states can be partitioned.

swap_horiz

Looking for DeepSpeed alternatives? Compare top competitors ranked & scored

arrow_forward