Sparse Mixture-of-Experts Architectures for Scaling Large Language Models: Routing, Specialization, and Computational Efficiency

Abstract

Scaling large language models by increasing the number of parameters improves representational capacity but also substantially increases training and inference cost when all model parameters are activated for every input token. Sparse Mixture-of-Experts architectures address this limitation by conditionally routing tokens through subsets of specialized neural-network components. This review examines the evolution of Mixture-of-Experts language modeling through 2024, beginning with classical adaptive and hierarchical expert models and progressing toward sparsely gated neural networks, GShard, Switch Transformers, expert-choice routing, GLaM, sparse upcycling, Mixtral, and DeepSeekMoE. Modern sparse models separate total parameter capacity from active computational cost by using routing mechanisms that select a limited number of experts for each token. Mixtral demonstrated the practical feasibility of sparse expert activation within competitive open-weight large language models, while DeepSeekMoE emphasized fine-grained expert segmentation and shared experts to encourage stronger specialization. The review analyzes token-choice and expert-choice routing, auxiliary load-balancing losses, expert capacity, communication overhead, expert collapse, distributed training, inference memory, routing instability, and specialization. Particular attention is given to the relationship between the nominal parameter count of an MoE model and the much smaller subset of parameters activated during individual forward passes. Challenges include load imbalance, cross-device communication, expert redundancy, routing interpretability, memory requirements, and increasingly complex deployment infrastructure. By 2024, Mixture-of-Experts modeling had emerged as an important route toward increasing model capacity without proportionally increasing per-token computation.