Selective Sinkhorn Routing: Efficiently Optimizing Sparse Mixture-of-Experts Routing

Sparse Mixture-of-Experts (SMoE) models offer scalability but often require auxiliary methods (like load-balancing loss) to manage expert utilization, which can complicate training or introduce objective misalignment. This research re-frames the token-to-expert assignment as an optimal transport problem, allowing for direct, balanced routing. The authors introduce Selective Sinkhorn Routing (SSR), a novel, efficient mechanism based on the Sinkhorn algorithm. SSR derives gating scores directly from the transport map, proving that optimal transport-based routing enhances SMoE performance, accuracy, and robustness against input corruption while reducing training overhead.