Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Expert Parallelism › Combining Expert Parallelism with Other Strategies

Michael BrenndoerferNovember 19, 20256 min read

Language AI HandbookAll chapters

Part 33 of 67Mixture of Experts

Chapter 246Expert Parallelism › Combining Expert Parallelism with Other Strategies

Reading section 9 of 12 · Combining Expert Parallelism with Other Strategies · 6 minSection 9 of 12 · 6 min

Section 9 of 12 · 6 min read

Expert Parallelism › Combining Expert Parallelism with Other StrategiesOne section · Sections in this chapter9 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In practice, expert parallelism is combined with other parallelism strategies to scale MoE models to hundreds of GPUs. No single parallelism strategy can address all the challenges of training massive models: expert parallelism handles the distribution of experts, but other strategies are needed to handle large batch sizes, oversized individual layers, and the sequential dependencies between layers. Understanding how to compose these strategies is needed for anyone building or deploying large-scale MoE systems.

The three main parallelism strategies in deep learning are data parallelism, tensor parallelism, and pipeline parallelism. Each addresses a different bottleneck: data parallelism increases throughput by processing more data simultaneously, tensor parallelism reduces per-layer memory by sharding weight matrices, and pipeline parallelism enables sequential computation on very deep networks. Expert parallelism is a fourth orthogonal strategy that specifically addresses the memory and compute challenges of MoE architectures. In large-scale training, all four strategies are often combined simultaneously, creating what practitioners call "4D parallelism."

Expert + Data Parallelism: Replicate the expert parallel setup across multiple groups of devices. Each group has a complete set of experts, and different groups process different batches. This combination multiplies throughput by the number of data parallel replicas while keeping the communication complexity of expert parallelism contained within each replica group. The all-to-all communication happens within replica groups, while gradient synchronization happens across replica groups. This separation is beneficial because it allows the two communication patterns to use different network topologies: the all-to-all benefits from low-latency, high-bandwidth links within a node, while gradient synchronization can tolerate higher latency on cross-node connections.
Expert + Tensor Parallelism: Shard individual experts across devices in addition to distributing experts. This handles cases where individual experts are too large for a single device. For example, if each expert has 7 billion parameters (as in Mixtral-scale models), tensor parallelism can split each expert across 2 or more devices, with expert parallelism then distributing these sharded experts across additional device groups. The combination introduces additional all-reduce operations within each sharded expert, but allows training models with extremely large experts that would not fit on any single device.
Expert + Pipeline Parallelism: Distribute different layers across different pipeline stages, with expert parallelism within each MoE layer. This approach addresses the sequential nature of transformer computation, where each layer depends on the output of the previous layer. Pipeline parallelism overlaps computation across stages by processing multiple microbatches simultaneously, while expert parallelism handles the within-layer distribution of experts. The challenge is that the all-to-all operations within each MoE layer create synchronization points that can reduce pipeline efficiency, particularly if the pipeline is not carefully designed to overlap these operations with other computation.

In[29]:

Code

import math


def calculate_device_requirements(
    num_experts: int,
    expert_params: int,
    model_params: int,  # Non-expert parameters
    memory_per_device_gb: float,
    bytes_per_param: int = 2,  # bf16
) -> dict:
    """
    Calculate minimum devices needed for different parallelism strategies.
    """
    total_expert_memory_gb = (
        num_experts * expert_params * bytes_per_param
    ) / 1e9
    total_model_memory_gb = (model_params * bytes_per_param) / 1e9
    total_memory_gb = total_expert_memory_gb + total_model_memory_gb

    # Account for optimizer states and activations (rough 3x multiplier)
    training_memory_gb = total_memory_gb * 3

    # Expert parallelism only
    min_ep_devices = max(
        1, math.ceil(total_expert_memory_gb / memory_per_device_gb)
    )

    # With tensor parallelism for non-expert params
    min_combined_devices = max(
        math.ceil(total_expert_memory_gb / memory_per_device_gb),
        math.ceil(total_model_memory_gb / memory_per_device_gb),
    )

    return {
        "total_expert_memory_gb": total_expert_memory_gb,
        "total_model_memory_gb": total_model_memory_gb,
        "training_memory_gb": training_memory_gb,
        "min_expert_parallel_devices": min_ep_devices,
        "min_combined_devices": min_combined_devices,
    }


# Mixtral-8x7B scale model
requirements = calculate_device_requirements(
    num_experts=8,
    expert_params=7_000_000_000,  # 7B params per expert
    model_params=7_000_000_000,  # Shared params (attention, embeddings)
    memory_per_device_gb=80,  # A100 80GB
)

Out[30]:

Console

Device requirements analysis (Mixtral-8x7B scale):
  Expert memory: 112.0 GB
  Other model memory: 14.0 GB
  Training memory (with optimizer): 378.0 GB
  Minimum devices (expert parallel): 2

Out[31]:

Visualization

Stacked bar chart comparing memory requirements across two scenarios: inference (weights only) and training (with optimizer). Blue segments represent expert parameters, orange segments shared parameters, and green segments training overhead. An A100 GPU capacity reference line shows that both scenarios far exceed single-device memory, requiring multi-GPU distribution. — Memory requirements for a Mixtral-8x7B scale model compared to A100 GPU capacity. Expert parameters (blue) and shared parameters (orange) consume significant memory, but training overheads (green), including optimizer states and activations, triple the requirement to over 350 GB, necessitating multi-device distribution.

The analysis shows that for a model of this scale, the expert parameters alone (112 GB) exceed the memory of a single 80GB device. We require at least 2 devices just to store the experts, and significantly more when training due to optimizer states and activations. The 3x multiplier for training memory accounts for the Adam optimizer (which maintains two additional tensors per parameter: the first and second moment estimates) and the activation checkpointing overhead. The total training memory requirement of over 350 GB means that even a fully populated 8-GPU server with 640 GB of total GPU memory may not be sufficient for training without careful memory management through techniques like gradient checkpointing, CPU offloading, or mixed precision training. In practice, teams training models at this scale often use 8, 16, or more devices, combining expert parallelism with data parallelism to achieve both memory capacity and training throughput.