Definition

Mixture of Experts (MoE) is a neural network design that splits certain layers into several parallel sub-networks called experts, with a router (also called a gating network) deciding which experts handle each input.

The key is sparse activation. Total parameter count can be very large, but only a small fraction of experts do any work for a given token. That’s why MoE models are often described with two numbers: total parameters and active parameters. The first governs how much the model can hold; the second governs the compute cost of each inference step.

The idea dates back decades. As LLMs have grown, MoE has become an important way to keep improving models within a limited compute budget.

How It Works

In Transformer-based LLMs, MoE usually replaces the feed-forward network in each layer, or in some layers:

  1. Routing scores. When a token enters an MoE layer, the router scores every expert based on the token’s representation.
  2. Expert selection. Only the few highest-scoring experts are picked (often called top-k routing). The rest sit this token out.
  3. Weighted combination. Each selected expert processes the token, and their outputs are summed, weighted by routing scores, to form the layer’s result.

Because different tokens go to different experts, the experts gradually specialize during training. This specialization is usually less intuitive than “the math expert” or “the language expert”; it tends to reflect preferences for certain patterns or token types.

MoE brings several engineering challenges:

Load balancing. If the router keeps favoring a few experts, the others go undertrained and compute is distributed unevenly. Training usually adds an auxiliary loss or other mechanisms to spread tokens more evenly.

Memory footprint. Only some experts are active per token, but all their parameters typically need to be loaded. Total memory requirements are still set by total parameter count.

Distributed communication. Large MoE models often place experts on different devices, so tokens travel between them. Communication overhead and scheduling strongly affect training and inference efficiency.

Training stability. The discrete routing choices make training more prone to instability and call for extra techniques to keep it under control.

Typical Applications

  • Large language models: growing total capacity while keeping per-token inference cost in check.
  • Multilingual and multitask models: different experts can adapt to different languages or task distributions.
  • High-throughput serving: offering a larger-capacity model for more requests within the same compute budget.
  • Transformer: MoE typically replaces its feed-forward layers.
  • Large language model: MoE is one of the main routes to scaling LLMs.
  • Dense models: models that use all parameters on every step, the counterpart to sparse MoE.
  • Quantization: commonly used to ease the memory pressure of MoE’s large total size.

Papers: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Switch Transformers