简而言之
混合专家（MoE）模型依赖专家并行（EP）来跨多块 GPU 扩展推理。在 SGLang 中，DeepEP 和 EPLB 在 EP 下提供了高性能服务，但每个 rank 所看到的工作负载仍可能不均衡，因为 token 在各专家之间的路由并不均匀。
本博客介绍 SGLang 中的两项调度时负载均衡功能：
- Waterfill，一种面向 DeepEP 的轻量级共享专家负载均衡方法。它通过 DeepEP 调度共享专家，并将其分配到负载较低的 rank 上。在两个 Hopper GPU 节点上、采用 DeepSeek-V3/R1 风格的服务工作负载时，Waterfill 在 MMLU、GPQA 和 GSM8K 上将总吞吐量提升了 +1.48% 至 +4.66%。在 DeepSeek V4 上，最佳实测点从 49,253 tok/s 提升至 51,677 tok/s（+4.92%）。
- LPLB，一种面向冗余专家副本的基于线性规划的负载均衡器。它针对冗余专家求解逐层的调度优化问题。在同样两个 Hopper GPU 节点上采用冗余 EPLB 放置时，LPLB 在 MMLU、GPQA 和 GSM8K 上将总吞吐量提升了 +0.84% 至 +7.34%。
Waterfill 的工作建立在两个 SGLang PR 之上：EP 下的共享专家融合和Waterfill 调度均衡。DeepSeek V4 支持在#25391中加入。LPLB 在#24515中引入。
引言
DeepSeek-V3/R1 和 DeepSeek V4 等大型 MoE 模型使用稀疏专家激活来提升模型容量，同时让每个 token 的计算量保持在可控范围内。在推理过程中，EP 将专家分布到各个 GPU 上，并将 token 路由到拥有所选专家的 rank。这降低了单 GPU 的显存压力，使大规模服务变得可行，但也引入了一个核心的系统问题：路由器并不会生成完全均衡的专家流量。
当某些专家接收到的 token 远多于其他专家时，EP 组需要等待最繁忙的 rank。这种不均衡同时影响计算和通信。EPLB 等静态放置方法可以改善专家的长期放置和冗余副本，但单个批次仍可能存在残余不均衡。调度时负载均衡通过在运行时决定哪个物理副本应处理每个 token 或每个共享专家请求，来解决这一剩余缺口。
在 SGLang 中，我们一直在研究两种针对 DeepEP MoE 推理的调度时方法：
- Waterfill：一种专注于共享专家路径的低开销算法。
- LPLB：一种基于线性规划的算法，专注于在冗余专家副本之间进行 token 路由。
这两种算法针对的是系统中同一大层面的问题：调度时的 MoE 负载均衡。它们做出了不同的权衡，并在不同的调度选择上运作。
背景：DeepEP MoE 推理中的负载不均衡
DeepEP 通过为专家并行提供优化的 token 分发与合并 kernel，加速 MoE 推理。在典型的 DeepSeek 风格 MoE 层中，每个 token 会被路由到由模型路由器选出的若干个路由专家。部分模型还包含一个共享专家，它会被应用于每一个 token。
从服务系统的角度来看，路由专家和共享专家会产生不同的负载模式：
- 路由专家是稀疏的。不同的 token 会选择不同的专家，因此它们的负载取决于路由器分布。
- 共享专家是稠密的。每个 token 都需要共享专家，因此共享专家的工作负载在整个批次中始终存在。
- 由 EPLB 风格放置方案引入的冗余专家，为某些逻辑专家提供了多个物理副本。这为调度时的负载均衡创造了机会，因为系统可以在不改变模型逻辑专家选择的前提下，选择由哪个物理副本来处理某个 token。
静态专家放置虽有帮助，但无法消除所有运行时的不均衡。一个批次中的实际 token 仍可能集中在部分专家或 rank 上。在 DeepEP 中，这会导致某些 rank 等待过载的对等节点。Waterfill 和 LPLB 都旨在减少这种分发时的不均衡，同时保持模型的语义不变。
Waterfill：面向共享专家分发的轻量级负载均衡
Waterfill 分发策略
Waterfill 是一种面向 DeepEP 下共享专家路径的轻量级负载均衡算法。
如果共享专家始终在每个 rank 上本地计算，那么无论某个 rank 是否已被路由专家过载，它都要承担共享专家的开销。过载的 rank 依然过载，而负载较轻的 rank 也无法帮助分担共享专家的工作。
Waterfill 改变了这一点，它将共享专家视为一个可调度的专家槽位。在路由专家选定之后，Waterfill 会估算每个 EP rank 上当前的路由负载，然后将共享专家的工作分配给负载较低的 rank。从概念上讲，它填补了 rank 负载分布中的低谷，类似于将水倒入高低不平的容器中。
对于每个 token，Waterfill 会为共享专家额外增加一个专家槽位。它不会总是把该槽位分配给 token 所在的本地 rank，而是根据当前负载分布选择一个 rank。这样路由专家的选择保持不变，因此模型仍然计算相同的逻辑路由专家和相同的共享专家。唯一改变的是由哪个物理 rank 执行共享专家的工作。
从高层来看，该算法是：
-
统计已经落在每个 EP rank 上的路由专家负载。
-
将该计数用作每个 rank 的负载分数。在动态模式下，SGLang 首先运行一次 EP 组集合通信，因此该分数可以使用全局路由负载向量加上每个 rank 当前的本地 batch size。
-
为每个参与的 token 增加一个共享专家槽位，并计算目标水位线：
这里 是 rank 的负载分数， 是要放置的共享专家槽位数量， 是 EP 组大小。
-
低于该水位线的 rank 有空闲余量：
-
对于每个 token，Waterfill 从候选 rank 中采样共享专家的目标 rank，概率与空闲余量成正比，并带有轻微的本地 rank 偏好。如果所有候选 rank 的空闲余量都为零，则回退到明显更轻的候选 rank，同时仍然保留本地 rank 偏好。
详细的推导过程以及 SGLang 静态/动态行为的确切细节，记录在 Waterfill 调度均衡 PR 中。
这里存在一个重要的通信权衡。如果每个 token 都能把它的共享专家工作发送到任意 EP rank，Waterfill 就会有更大的均衡自由度，但这也可能增加 all-to-all 通信量。对于 GPU MoE 服务而言，通信往往比额外的共享专家计算更昂贵。
因此，通信保守型候选集会把共享专家保留在 token 为路由专家已经访问过的那些 rank 上，并将源 rank 作为回退选项。SGLang 还支持 all-rank 模式，这给 Waterfill 带来更大的均衡自由度，但可能新增一个每 token 的调度目的地。
这是一种刻意的通信权衡，而非模型语义的改变。
通过将共享专家工作从已经负载较重的 rank 转移到负载较轻的 rank，Waterfill 均衡了各 rank 的工作量，并提升了端到端吞吐量。
图 1. Waterfill 将共享专家工作从过载的 rank 转移到负载较轻的 rank，同时保持路由专家的选择不变，在不改变模型语义的前提下缩短了最慢的 MoE 层路径。
共享专家融合作为实现机制
通过融合共享专家和路由专家，可以进一步加速 Waterfill。
在 EP 下，共享专家使用的执行路径与路由专家相互独立。在 Waterfill 选定非本地共享专家 rank 之后，这种设计就需要从已分发的路由专家布局中提取共享专家的 token，并单独启动一次共享专家计算，从而带来额外的布局转换和启动开销。
共享专家融合通过将共享专家表示为同一个 DeepEP MoE 布局中的另一个专家槽位，避免了这条路径。在 DeepSeek V3/R1 中，路由器仍然选择原本的路由 top-k 专家，而 TopK 输出会为共享专家额外增加一列。在 DeepEP 的物理专家 ID 布局中，每个 rank 会在其路由专家旁边预留一个额外的共享专家槽位。这使得路由专家和共享专家可以共用同一套 DeepEP 分发、分组 GEMM 和合并流程。
这就是 Waterfill 功能被拆分为两部分的原因：
融合本身并不是最终的负载均衡算法。它是让共享专家分发对 DeepEP 可见、从而可被 Waterfill 控制的必要机制。
LPLB：面向冗余专家副本的基于 LP 的负载均衡
LPLB 所解决的问题
EPLB 会为热点逻辑专家放置冗余副本，然后默认将每个热点专家的 token 均匀地分配到其各个物理副本上。只有当用于构建放置方案的离线分布与实时流量相匹配时，均匀分配才是最优的。而在实践中，情况往往并非如此：单个批次集中的专家与校准集不同，服务所用的数据集偏离了录制数据集，而且再平衡周期足够长，以至于放置方案在许多批次上实际上是静态的。当这种情况发生时，即使将热点专家的负载均匀分配，拥有其副本的那些 rank 相对于 EP 组中的其他 rank 仍然负载不均，整个组都要等待最繁忙的那个 rank。
LPLB 在分发时弥补了这一差距。对于每个 MoE 层、每个批次，它会查看实际的每个专家的 token 数量，并决定如何将每个复制专家的 token 拆分到其各个物理副本上，从而使每个 rank 的最大负载最小化。它不移动权重，也不改变路由器的逻辑 top-k 选择——它只是在某个逻辑专家的各个有效物理副本之间，决定每个副本接收多少流量。其结果是针对当前批次的 min–max 最优分配，而非 EPLB 在离线时固化的静态均匀拆分。
LP 形式化
LPLB 将此建模为每层求解的一个小型线性规划。其直觉直接映射到约束条件上：
- 目标——最小化峰值。 引入一个标量
M，表示所有 rank 上的最大负载，并将其最小化。将M压低会把最繁忙的 rank 拉向平均值，而这正是缩短由 EP 不均衡所产生的 grouped-GEMM 长尾的关键。 - Rank 负载约束。 对于每个 rank，（来自其冗余专家副本的负载）+（来自其单副本专家的负载）+（到峰值的松弛量）= M。每个 rank 上的单副本负载是固定输入——这些专家没有调度选择。每个 rank 得到这样一个等式；松弛量非负，因此
M被强制至少不小于每个 rank 的真实负载。 - 冗余专家守恒。 对于每个被复制的逻辑专家，分配给其各副本的负载之和必须等于该专家观测到的总负载：，其中 是放置在副本 上的负载， 是该专家观测到的总负载。这保证了 LPLB 只重新分配现有流量，绝不会凭空产生或丢弃 token。
决策变量是被复制专家的各副本负载，加上每个 rank 的松弛量以及 M。单副本专家不是变量——它们只贡献固定项——这使得 LP 规模很小：其规模随 冗余专家的数量和 rank 的数量而扩展，而非随完整的专家总数扩展。
约束矩阵被拆分为离线部分和在线部分。结构性块——副本到逻辑专家的映射、复制副本在各 rank 上的归属，以及 slack/−M 列——只取决于专家到 GPU 的放置方式，因此它们在启动时以及每次 EPLB 重新平衡后预先计算一次。只有右端项会随每个 batch 变化：观测到的冗余专家负载以及各 rank 的单副本负载。一个 Big-M 辅助列在求解过程中保持系统可行，并在目标函数中被施以重罚，从而驱使求解器将其压至零。
从全局计数到求解 LP
DP-attention 的一个微妙之处在于，不同的 EP rank 在同一步中运行不同的前向模式——prefill、decode 或 idle——因此没有任何单个 rank 能看到全局 token 分布。LPLB 通过一种刻意简单的集合通信设计来处理这一点：
- 每个 rank 统计其本地 token 在各逻辑专家上的数量。
- 所有 EP rank 都参与对这些计数的一次 all-reduce——idle 的 rank 贡献零——因此每个 rank 最终都得到完全相同的全局各专家分布。
- 随后每个 rank 都基于这些完全相同的输入独立求解同一个 LP，并得到相同的解，因此无需广播结果。
LP 本身通过一个基于 cuSOLVERDx/cuBLASDx 构建的融合内点法（IPM）kernel 在 GPU 上求解，并在启动时针对该层的矩阵形状预先编译，因此第一个真实请求无需承担 JIT 编译开销。整个逐批次路径——构建右端项、求解、提取每个副本的分配——被压缩为三次 CUDA kernel 启动，写入预分配的缓冲区，从而将启动开销和主机同步移出关键路径。
从 LP 求解到 Token 分发
LP 为每个复制的逻辑专家返回其负载应当如何在其各个物理副本之间分配。LPLB 将其归一化为该专家在有效物理副本上的概率分布（log2phy_prob）。在分发时，每个路由到某个复制逻辑专家的 token 从该分布中采样一个物理副本；单副本专家则像之前一样映射到其唯一的物理位置。这是对现有 dynamic 策略的直接替换，后者以均匀随机方式选取副本——LPLB 保持相同的概率化、逐 token 分发形态，但将均匀抽取替换为针对该批次计算出的负载最优分布。
图 2. LPLB 在不改变逻辑专家路由的情况下，将复制专家的流量转向负载较轻的 rank，因此当存在冗余副本时，同一个被选中的专家可以在负载较低的物理副本上完成计算。
LPLB 与 Waterfill 有何不同
Waterfill 和 LPLB 的最终目标相同——在 DeepEP 下压平每个 rank 的负载——但作用于不同的分发选择，且所用机制不同：
| Waterfill | LPLB | |
|---|---|---|
| Target | 共享（稠密）专家，应用于每一个 token | 该路由专家，EPLB 已复制 |
| 决策 | 哪个 rank 执行每个 token 的共享专家槽位 | 如何将每个复制专家的 token 拆分到其各个物理副本上 |
| 方法 | 基于当前 rank 负载的轻量级填谷启发式 | 在 GPU 上求解的逐层最小–最大线性规划 |
| 要求 | 共享专家融合 | 存在 EPLB 冗余副本 |
| 成本 | 近乎零开销 | 每层一次 all-reduce 加一次 LP 求解 |
它们是互补而非竞争关系：Waterfill 消除由稠密共享专家带来的不均衡，而 LPLB 消除稀疏路由副本之间的不均衡。由于 LPLB 只在同一逻辑专家的有效副本之间重新分配流量，且从不改变路由器的逻辑 top-k，因此它出于与 Waterfill 相同的原因保持了模型语义不变。
LPLB 何时帮助最大
LPLB 的收益取决于实时批次偏离 EPLB 校准所依据的分布的程度。当流量均衡良好且批次巨大（大规模、高度多样化的服务）时，留给 LP 消除的残余不均衡很少。当流量基本不变且范围狭窄（少量几乎相同的问题）时，静态 EPLB 已经捕捉到了该分布，均分已接近最优。LPLB 在两者之间给出最强的信号——中等规模的服务，聚焦于数量适中的相关主题，此时每个批次的不均衡方式超出了离线放置的预期，但其结构仍足够清晰，使得每批次的最优划分能够有意义地降低峰值 rank 负载。
评估
Waterfill 与 LPLB 在 DeepSeek V3/R1 上的表现
我们在相同的 DeepSeek-V3/R1 风格服务配置上评估了 Waterfill 和 LPLB。下表对应直接集成 SGLang 的运行结果 dsv3_ep16_three_dataset_lplb_matrix_20260605_101821，使用的 SGLang 提交为 a462e0f864103785fd3e64327104103f1356f220。
基准配置如下：
- 模型：DeepSeek-V3 FP8，用作 DeepSeek-V3/R1 风格的服务工作负载。
- 硬件：两个 Hopper GPU 节点，共 16 块 GPU。
- 并行与后端：TP16、DP16、EP16、DP attention、DeepEP normal 模式。
- 数据集：MMLU、GPQA 和 GSM8K 提示词池。
- 基准形态：
batch_size=1000、concurrency=256、request_rate=inf、max_tokens=1。
该表格将每组对比限制在相同的放置配置内。LPLB 仅在启用 EPLB 放置时才有意义，因为它是在同一逻辑专家的多个有效物理副本之间进行路由。
| 数据集 | 基线设置 | 基线 | Waterfill | Waterfill 增益 | LPLB | LPLB 增益 |
|---|---|---|---|---|---|---|
| MMLU | 无 EPLB | 28,968 tok/s | 29,697 tok/s | +2.52% | - | - |
| MMLU | Static EPLB, red0 | 30,392 tok/s | 31,424 tok/s | +3.40% | 29,938 tok/s | -1.50% |
| MMLU | Static EPLB, red16 | 30,638 tok/s | 31,483 tok/s | +2.76% | 31,104 tok/s | +1.52% |
| MMLU | 静态 EPLB，red32 | 30,714 tok/s | 31,169 tok/s | +1.48% | 31,547 tok/s | +2.72% |
| GPQA | 无 EPLB | 23,201 tok/s | 24,283 tok/s | +4.66% | - | - |
| GPQA | 静态 EPLB，red0 | 26,322 tok/s | 26,970 tok/s | +2.46% | 25,899 tok/s | -1.61% |
| GPQA | Static EPLB, red16 | 26,124 tok/s | 26,683 tok/s | +2.14% | 26,350 tok/s | +0.86% |
| GPQA | Static EPLB, red32 | 25,975 tok/s | 26,655 tok/s | +2.62% | 26,193 tok/s | +0.84% |
| GSM8K | No EPLB | 29,649 tok/s | 30,892 tok/s | +4.19% | - | - |
| GSM8K | Static EPLB，red0 | 33,058 tok/s | 34,529 tok/s | +4.45% | 32,744 tok/s | -0.95% |
| GSM8K | Static EPLB，red16 | 34,026 tok/s | 35,226 tok/s | +3.53% | 35,474 tok/s | +4.26% |
| GSM8K | Static EPLB，red32 | 33,988 tok/s | 35,070 tok/s | +3.19% | 36,482 tok/s | +7.34% |
图 3. 在 MMLU、GPQA 和 GSM8K 上，Waterfill 通过将共享专家的工作转移到负载较低的 EP rank，持续将总吞吐量提升至高于匹配基线的水平。
图 4. 当存在冗余专家副本（red16/red32）时，LPLB 能提升吞吐量，因为 LP 有物理副本可供选择。而在 red0 的情况下，没有冗余副本可供重新平衡，因此该算法无法改善分发，其 all-reduce/求解路径仅表现为额外开销。
这些结果表明，Waterfill 在保持模型质量的同时提升了吞吐量，因为它只改变物理共享专家的放置，而不改变逻辑专家计算。当冗余专家副本提供了有用的分发选择时，LPLB 表现最强，如 red16 和 red32 行所示。相比之下，当未提供冗余专家时，LPLB 没有空间来平衡负载，因此仅表现出算法开销。
Waterfill 在 DeepSeek V4 上的应用
DeepSeek V4 可以使用 HashTopK 路由路径，此时 Waterfill 还必须在 HashTopK 输出路径中追加并重新映射共享专家槽位。#25391 将 Waterfill 扩展到了该路径。共享专家平衡这一思路本身并不特定于 HashTopK；它也适用于非 HashTopK 的路由路径。
DeepSeek V4 Flash FP8 在两个 Hopper GPU 节点上，在 MMLU 风格的服务负载上展现出持续稳定的吞吐量提升。本次 V4 Flash 运行使用 14,042 条提示词的 MMLU 池，batch=512，concurrency=128，max_tokens=1，2 轮预热，4 轮测量。表中报告的是截尾平均总吞吐量。由于本次 V4 Flash 运行使用了更小的 batch/concurrency 配置，这些数字应被理解为 V4 专属验证，而非与上述 DeepSeek-V3/R1 风格矩阵的直接吞吐量对比。
| 配置 | 基线 | 注水 | 增益 |
|---|---|---|---|
| 无 EPLB | 45,951 tok/s | 47,876 tok/s | +4.19% |
| 静态 EPLB，red0 | 49,253 tok/s | 51,677 tok/s | +4.92% |
| 静态 EPLB，red16 | 50,006 tok/s | 51,655 tok/s | +3.30% |
| 静态 EPLB，red32 | 50,167 tok/s | 51,813 tok/s | +3.28% |
图 5. Waterfill 在 DeepSeek V4 Flash 上同样有效，在无 EPLB 和静态 EPLB 设置下均提升了吞吐量，增益从 +3.28% 到 +4.92%。
这些结果验证了 Waterfill 在 DeepSeek V4 Flash 上同样保持正向效果，与上述 DeepSeek-V3/R1 风格的工作负载一致。
精度验证
Waterfill 保持了模型语义，因为它不改变路由器的逻辑 top-k 决策。模型所选择的路由专家保持不变，共享专家仍是同一个共享专家。Waterfill 只改变由哪个物理 EP rank 执行共享专家槽位。
LPLB 出于相同的结构性原因保持了模型语义。它从不改变路由器的逻辑 top-k；它只选择由所选逻辑专家的哪个物理副本来执行每个 token。由于一个逻辑专家的所有副本都持有相同的权重，token 所获得的结果与由哪个副本处理无关。这与 EPLB 以及 dynamic 策略已经依赖的精度保证相同。
如何使用
启用 Waterfill
Waterfill 通过 DeepEP MoE 路径启用。一条代表性的启动命令为：
python3 -m sglang.launch_server \
--model-path /path/to/DeepSeek-V3 \
--tp 16 \
--dp-size 16 \
--nnodes 2 \
--node-rank ${NODE_RANK} \
--dist-init-addr ${HEAD_NODE_IP}:${PORT} \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code \
--moe-a2a-backend deepep \
--deepep-mode normal \
--enable-dp-attention \
--enable-deepep-waterfill \
--init-expert-location /path/to/expert_distribution.pt
关键标志位如下：
--moe-a2a-backend deepep：使用 DeepEP 进行 MoE all-to-all dispatch。--enable-deepep-waterfill：启用共享专家融合与 Waterfill 路径。--init-expert-location：可选地根据收集到的专家分布统计数据，初始化专家放置和 rank 负载元数据。
DeepSeek V4 支持使用 HashTopK 路径，该路径在 #25391 中添加。
启用 LPLB
LPLB 是通过 DeepEP MoE 路径上的 EP 调度算法选出的。由于 LPLB 会在冗余副本之间均衡 token，它要求 EPLB 的放置方案中确实包含冗余专家。一个具有代表性的双节点启动命令如下：
python3 -m sglang.launch_server \
--model-path /path/to/DeepSeek-R1 \
--tp 16 \
--dp-size 16 \
--ep-size 16 \
--nnodes 2 \
--node-rank ${NODE_RANK} \
--dist-init-addr ${HEAD_NODE_IP}:${PORT} \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code \
--moe-a2a-backend deepep \
--deepep-mode normal \
--enable-dp-attention \
--ep-num-redundant-experts 16 \
--ep-dispatch-algorithm lp \
--init-expert-location /path/to/expert_stats.pt
重要的标志如下：
--ep-dispatch-algorithm lp：选择 LPLB 线性规划调度器，以替代默认的static或均匀随机的dynamic策略。--ep-num-redundant-experts：为热点逻辑专家创建冗余物理副本。没有这些副本，LPLB 就无从均衡——这正是上表中red0各行显示 LPLB 没有带来增益的原因。--init-expert-location：加载从专家分布记录运行中收集到的静态 EPLB 放置方案（物理到逻辑的映射，包括冗余槽位）。此处的副本数量必须与--ep-num-redundant-experts保持一致。
致谢
本工作基于 SGLang DeepEP 和 MoE 服务栈构建，并通过 SGLang 项目中的社区协作开发完成。
我们感谢 SGLang 维护者和审阅者在相关 PR 中提供的讨论、审阅和集成支持：
- #20089：在 EP 下将共享专家融合进 MoE 分发
- #19290：为共享专家分发添加 Waterfill 负载均衡
- #25391：支持 DeepSeek V4 DeepEP Waterfill
- #24515：LPLB：面向 MoE 专家并行的线性规划负载均衡器
我们衷心感谢为本工作做出贡献的人员：
- NVIDIA 团队：Xuting Zhou、Fei Liang 和 Aichen Feng
- SGLang 团队：Cheng Wan
我们还要感谢 DeepSeek 在 deepseek-ai/LPLB 开源了他们的 LPLB 工作，其用于在冗余专家副本之间平衡 token 的线性规划公式，启发了本文所述的 SGLang LPLB 集成。
TL;DR
Mixture-of-Experts (MoE) models rely on Expert Parallelism (EP) to scale inference across multiple GPUs. In SGLang, DeepEP and EPLB provide high-performance serving under EP, but the workload seen by each rank can still be imbalanced because tokens are not routed uniformly across experts.
This blog introduces two dispatch-time load balancing features in SGLang:
- Waterfill, a lightweight shared-expert load balancing method for DeepEP. It dispatches the shared expert through DeepEP and assigns it to less-loaded ranks. On two Hopper GPU nodes with DeepSeek-V3/R1-style serving workloads, Waterfill improves total throughput by +1.48% to +4.66% across MMLU, GPQA, and GSM8K. On DeepSeek V4, the best measured point improved from 49,253 tok/s to 51,677 tok/s (+4.92%).
- LPLB, a linear-programming-based load balancer for redundant expert replicas. It solves a per-layer dispatch optimization problem over redundant experts. With redundant EPLB placement on the same two Hopper GPU nodes, LPLB improves total throughput by +0.84% to +7.34% across MMLU, GPQA, and GSM8K.
The Waterfill work is built on two SGLang PRs: shared expert fusion under EP and Waterfill dispatch balancing. DeepSeek V4 support is added in #25391. LPLB is introduced in #24515.
Introduction
Large MoE models such as DeepSeek-V3/R1 and DeepSeek V4 use sparse expert activation to increase model capacity while keeping per-token computation manageable. During inference, EP distributes experts across GPUs and routes tokens to the ranks that own the selected experts. This reduces per-GPU memory pressure and makes large-scale serving practical, but it also introduces a central systems problem: the router does not generate perfectly balanced expert traffic.
When some experts receive many more tokens than others, the EP group waits for the busiest ranks. This imbalance affects both computation and communication. Static placement methods such as EPLB can improve the long-term placement of experts and redundant replicas, but a single batch can still have residual imbalance. Dispatch-time load balancing addresses this remaining gap by deciding, at runtime, which physical replica should process each token or each shared-expert request.
In SGLang, we have been working on two dispatch-time approaches for DeepEP MoE inference:
- Waterfill: a low-overhead algorithm focused on the shared expert path.
- LPLB: an LP-based algorithm focused on token routing across redundant expert replicas.
The two algorithms target the same broad layer of the system: dispatch-time MoE load balancing. They make different tradeoffs and operate on different dispatch choices.
Background: Load Imbalance in DeepEP MoE Inference
DeepEP accelerates MoE inference by providing optimized token dispatch and combine kernels for expert parallelism. In a typical DeepSeek-style MoE layer, each token is routed to several routed experts selected by the model router. Some models also include a shared expert, which is applied to every token.
From a serving-system perspective, routed experts and shared experts create different load patterns:
- Routed experts are sparse. Different tokens choose different experts, so their load depends on the router distribution.
- Shared experts are dense. Every token needs the shared expert, so the shared-expert workload is present for the full batch.
- Redundant experts, introduced by EPLB-style placement, provide multiple physical replicas for some logical experts. They create an opportunity for dispatch-time balancing, because the system can choose which physical replica processes a token without changing the model's logical expert choice.
Static expert placement is helpful, but it cannot remove all runtime imbalance. The actual tokens in a batch may still concentrate on a subset of experts or ranks. In DeepEP, this can leave some ranks waiting for overloaded peers. Waterfill and LPLB both aim to reduce this dispatch-time imbalance while preserving the model's semantics.
Waterfill: Lightweight Load Balancing for Shared Expert Dispatch
Waterfill Dispatch Strategy
Waterfill is a lightweight load balancing algorithm for the shared expert path under DeepEP.
If the shared expert is always computed locally on every rank, then each rank pays the shared-expert cost regardless of whether it is already overloaded by routed experts. The overloaded ranks remain overloaded, and the less-loaded ranks cannot help absorb the shared-expert work.
Waterfill changes this by treating the shared expert as a dispatchable expert slot. After the routed experts are selected, Waterfill estimates the current routed load on each EP rank, then assigns the shared expert work to ranks with lower load. Conceptually, it fills the valleys in the rank-load distribution, similar to pouring water into uneven containers.
For each token, Waterfill adds one extra expert slot for the shared expert. Instead of always assigning that slot to the token's local rank, it selects a rank based on the current load distribution. This keeps the routed expert choices unchanged, so the model still computes the same logical routed experts and the same shared expert. The only thing that changes is which physical rank executes the shared expert work.
At a high level, the algorithm is:
-
Count the routed expert load already landing on each EP rank.
-
Use that count as a per-rank load score. In dynamic mode, SGLang first runs one EP-group collective, so the score can use the global routed-load vector plus each rank's current local batch size.
-
Add one shared-expert slot per participating token, and compute a target waterline:
Here is rank 's load score, is the number of shared-expert slots to place, and is the EP group size.
-
Ranks below this waterline have slack:
-
For each token, Waterfill samples the shared-expert target rank from candidate ranks with probability proportional to slack, with a small local-rank preference. If all candidates have zero slack, it falls back to the clearly lighter candidate rank, again keeping the local-rank preference.
The detailed derivation and the exact SGLang static/dynamic behavior are documented in the Waterfill dispatch balancing PR.
There is an important communication tradeoff. If every token could send its shared-expert work to any EP rank, Waterfill would have more balancing freedom, but it could also increase all-to-all traffic. For GPU MoE serving, communication is often more expensive than the extra shared-expert computation. The communication-conservative candidate set therefore keeps the shared expert on ranks that the token already visits for routed experts, with the source rank kept as a fallback. SGLang also supports an all-rank mode, which gives Waterfill more balancing freedom but can add a new per-token dispatch destination. This is a deliberate communication tradeoff rather than a change in model semantics.
By shifting shared-expert work away from already-heavy ranks and toward lighter ranks, Waterfill balances per-rank work and improves end-to-end throughput.
Figure 1. Waterfill moves shared-expert work from overloaded ranks to lighter ranks while keeping the routed expert choices unchanged, shortening the slowest MoE-layer path without changing model semantics.
Shared Expert Fusion as the Enabling Mechanism
Waterfill can be further accelerated by fusing shared experts and routed experts.
Under EP, shared experts used a separate execution path from the routed experts. After Waterfill chooses non-local shared-expert ranks, that design would need to extract shared-expert tokens from the dispatched routed-expert layout and launch a separate shared-expert computation, adding extra layout conversion and launch overhead.
Shared expert fusion avoids that path by representing the shared expert as another expert slot in the same DeepEP MoE layout. In DeepSeek V3/R1, the router still selects the original routed top-k experts, and the TopK output gets one additional column for the shared expert. In the DeepEP physical expert ID layout, each rank reserves one extra shared-expert slot next to its routed experts. This lets routed experts and the shared expert share the same DeepEP dispatch, grouped-GEMM, and combine flow.
This is why the Waterfill feature was split into two pieces:
- #20089 fuses the shared expert into the DeepEP MoE path with a fixed local assignment.
- #19290 adds Waterfill, which replaces the fixed assignment with load-aware shared-expert dispatch.
The fusion itself is not the final load balancing algorithm. It is the required mechanism that makes shared-expert dispatch visible to DeepEP and therefore controllable by Waterfill.
LPLB: LP-Based Load Balancing for Redundant Expert Replicas
The Problem LPLB Solves
EPLB places redundant replicas of hot logical experts and then, by default, splits each hot expert's tokens evenly across its physical copies. Even splitting is optimal only when the offline distribution used to build the placement matches the live traffic. In practice it often does not: a single batch concentrates on different experts than the calibration set, the served dataset drifts away from the recording dataset, and the rebalance period is long enough that placement is effectively static for many batches. When that happens, evenly dividing a hot expert's load still leaves the ranks that own its copies unevenly loaded relative to the rest of the EP group, and the whole group waits on the busiest rank.
LPLB closes this gap at dispatch time. For each MoE layer, on each batch, it looks at the actual per-expert token counts and decides how to split each replicated expert's tokens across its physical copies so that the maximum per-rank load is minimized. It does not move weights and it does not change the router's logical top-k choices — it only chooses, among the valid physical replicas of a logical expert, how much traffic each replica receives. The result is an optimal min–max assignment for the batch in front of it, rather than the static even split EPLB bakes in offline.
The LP Formulation
LPLB casts this as a small linear program solved per layer. The intuition maps directly onto the constraints:
- Objective — minimize the peak. Introduce a scalar
Mrepresenting the maximum load over all ranks, and minimize it. DrivingMdown pulls the busiest rank toward the average, which is exactly what shortens the grouped-GEMM tail that EP imbalance creates. - Rank-load constraints. For every rank, (load from its redundant-expert copies) + (load from its single-copy experts) + (slack to the peak) = M. The single-copy load on each rank is fixed input — those experts have no dispatch choice. Each rank gets one such equation; the slack is non-negative, so
Mis forced to be at least every rank's true load. - Redundant-expert conservation. For every replicated logical expert, the loads assigned to its copies must sum to that expert's total observed load: , where is the load placed on copy and is the expert's total observed load. This guarantees LPLB only redistributes existing traffic and never invents or drops tokens.
The decision variables are the per-copy loads of the replicated experts plus the per-rank slacks and M. Single-copy experts are not variables — they contribute only fixed terms — which keeps the LP small: its size scales with the number of redundant experts and the number of ranks, not the full expert count.
The constraint matrix is split into an offline part and an online part. The structural blocks — the copy-to-logical-expert mapping, the per-rank ownership of replicated copies, and the slack/−M columns — depend only on the expert-to-GPU placement, so they are pre-computed once at startup and after every EPLB rebalance. Only the right-hand side changes per batch: the observed redundant-expert loads and the per-rank single-copy loads. A Big-M auxiliary column keeps the system feasible during the solve and is penalized heavily in the objective so the solver drives it to zero.
From Global Counts to a Solved LP
A subtlety of DP-attention is that different EP ranks run different forward modes in the same step — prefill, decode, or idle — so no single rank sees the global token distribution. LPLB handles this with a deliberately simple collective design:
- Each rank counts its local tokens per logical expert.
- All EP ranks participate in one all-reduce of those counts — idle ranks contribute zeros — so every rank ends up with the identical global per-expert distribution.
- Every rank then solves the same LP independently from those identical inputs and obtains the same solution, so no broadcast of the result is needed.
The LP itself is solved on-GPU by a fused interior-point-method (IPM) kernel built on cuSOLVERDx/cuBLASDx, pre-compiled for the layer's matrix shape at startup so the first real request does not pay the JIT cost. The whole per-batch path — build the right-hand side, solve, and extract the per-copy split — collapses into three CUDA kernel launches that write into pre-allocated buffers, keeping launch overhead and host syncs off the critical path.
From LP Solution to Token Dispatch
The LP returns, for each replicated logical expert, how its load should be divided across its physical copies. LPLB normalizes this into a per-expert probability distribution over the valid physical copies (log2phy_prob). At dispatch, each token routed to a replicated logical expert samples a physical copy from that distribution; single-copy experts map to their one physical location as before. This is a drop-in replacement for the existing dynamic policy, which picks a copy uniformly at random — LPLB keeps the same probabilistic, per-token dispatch shape but replaces the uniform draw with the load-optimal distribution computed for the batch.
Figure 2. LPLB shifts replicated-expert traffic toward lighter ranks without changing logical expert routing, so the same selected expert can finish on a less-loaded physical replica when redundant copies are available.
How LPLB Differs from Waterfill
Waterfill and LPLB share the end goal — flatten per-rank load under DeepEP — but act on different dispatch choices, with different machinery:
| Waterfill | LPLB | |
|---|---|---|
| Target | The shared (dense) expert, applied to every token | The routed experts that EPLB has replicated |
| Decision | Which rank executes each token's shared-expert slot | How to split each replicated expert's tokens across its physical copies |
| Method | Lightweight valley-filling heuristic over current rank loads | Per-layer min–max linear program solved on-GPU |
| Requires | Shared expert fusion | EPLB redundant replicas to exist |
| Cost | Near-zero overhead | An all-reduce plus an LP solve per layer |
They are complementary rather than competing: Waterfill removes imbalance contributed by the dense shared expert, while LPLB removes imbalance among the sparse routed replicas. Because LPLB only redistributes traffic across valid replicas of the same logical expert and never alters the router's logical top-k, it preserves model semantics for the same reason Waterfill does.
When LPLB Helps Most
LPLB's benefit tracks how much the live batch deviates from the distribution EPLB was calibrated on. When traffic is well-balanced and batches are huge (large-scale, highly diverse serving), there is little residual imbalance left for the LP to remove. When traffic is essentially invariant and narrow (a handful of near-identical questions), static EPLB already captures the distribution and an even split is close to optimal. LPLB delivers the strongest signal in between — medium-scale serving focused on a moderate number of related topics, where each batch is imbalanced in a way the offline placement did not anticipate but is still structured enough that an optimal per-batch split meaningfully lowers the peak rank load.
Evaluation
Waterfill and LPLB on DeepSeek V3/R1
We evaluated Waterfill and LPLB on the same DeepSeek-V3/R1-style serving setup. The table below corresponds to the direct SGLang integration run dsv3_ep16_three_dataset_lplb_matrix_20260605_101821, using SGLang commit a462e0f864103785fd3e64327104103f1356f220.
The benchmark configuration was:
- Model: DeepSeek-V3 FP8, used as the DeepSeek-V3/R1-style serving workload.
- Hardware: two Hopper GPU nodes, 16 GPUs total.
- Parallelism and backend: TP16, DP16, EP16, DP attention, DeepEP normal mode.
- Datasets: MMLU, GPQA, and GSM8K prompt pools.
- Benchmark shape:
batch_size=1000,concurrency=256,request_rate=inf,max_tokens=1.
The table keeps each comparison within the same placement configuration. LPLB is only meaningful when EPLB placement is enabled, because it routes among valid physical replicas of the same logical expert.
| Dataset | Baseline setting | Baseline | Waterfill | Waterfill gain | LPLB | LPLB gain |
|---|---|---|---|---|---|---|
| MMLU | No EPLB | 28,968 tok/s | 29,697 tok/s | +2.52% | - | - |
| MMLU | Static EPLB, red0 | 30,392 tok/s | 31,424 tok/s | +3.40% | 29,938 tok/s | -1.50% |
| MMLU | Static EPLB, red16 | 30,638 tok/s | 31,483 tok/s | +2.76% | 31,104 tok/s | +1.52% |
| MMLU | Static EPLB, red32 | 30,714 tok/s | 31,169 tok/s | +1.48% | 31,547 tok/s | +2.72% |
| GPQA | No EPLB | 23,201 tok/s | 24,283 tok/s | +4.66% | - | - |
| GPQA | Static EPLB, red0 | 26,322 tok/s | 26,970 tok/s | +2.46% | 25,899 tok/s | -1.61% |
| GPQA | Static EPLB, red16 | 26,124 tok/s | 26,683 tok/s | +2.14% | 26,350 tok/s | +0.86% |
| GPQA | Static EPLB, red32 | 25,975 tok/s | 26,655 tok/s | +2.62% | 26,193 tok/s | +0.84% |
| GSM8K | No EPLB | 29,649 tok/s | 30,892 tok/s | +4.19% | - | - |
| GSM8K | Static EPLB, red0 | 33,058 tok/s | 34,529 tok/s | +4.45% | 32,744 tok/s | -0.95% |
| GSM8K | Static EPLB, red16 | 34,026 tok/s | 35,226 tok/s | +3.53% | 35,474 tok/s | +4.26% |
| GSM8K | Static EPLB, red32 | 33,988 tok/s | 35,070 tok/s | +3.19% | 36,482 tok/s | +7.34% |
Figure 3. Across MMLU, GPQA, and GSM8K, Waterfill consistently raises total throughput over the matched baseline by shifting shared-expert work toward less-loaded EP ranks.
Figure 4. LPLB improves throughput when redundant expert replicas exist (red16/red32) because the LP has physical copies to choose among. With red0, there are no redundant replicas to rebalance, so the algorithm cannot improve dispatch and its all-reduce/solve path appears only as overhead.
These results indicate that Waterfill improves throughput while preserving model quality, because it only changes physical shared-expert placement and does not change the logical expert computation. LPLB is strongest when redundant expert replicas provide useful dispatch choices, as shown by the red16 and red32 rows. In contrast, when no redundant experts are provided, LPLB has no room to balance the load and thus shows only the algorithm overhead.
Waterfill on DeepSeek V4
DeepSeek V4 can use a HashTopK routing path, where Waterfill must append and remap the shared-expert slot in the HashTopK output path as well. #25391 extends Waterfill to that path. The shared-expert balancing idea itself is not specific to HashTopK; it also applies to non-HashTopK routing paths.
DeepSeek V4 Flash FP8 on two Hopper GPU nodes showed consistent throughput improvement on the MMLU-style serving workload. This V4 Flash run uses the 14,042-prompt MMLU pool, batch=512, concurrency=128, max_tokens=1, 2 warmup rounds, and 4 measured rounds. The table reports trimmed mean total throughput. Because this V4 Flash run used a smaller batch/concurrency shape, these numbers should be read as V4-specific validation rather than a direct throughput comparison against the DeepSeek-V3/R1-style matrix above.
| Configuration | Baseline | Waterfill | Gain |
|---|---|---|---|
| No EPLB | 45,951 tok/s | 47,876 tok/s | +4.19% |
| Static EPLB, red0 | 49,253 tok/s | 51,677 tok/s | +4.92% |
| Static EPLB, red16 | 50,006 tok/s | 51,655 tok/s | +3.30% |
| Static EPLB, red32 | 50,167 tok/s | 51,813 tok/s | +3.28% |
Figure 5. Waterfill is also effective on DeepSeek V4 Flash, improving throughput across no-EPLB and static-EPLB settings with gains from +3.28% to +4.92%.
These results validate that Waterfill remains directionally positive on DeepSeek V4 Flash in addition to the DeepSeek-V3/R1-style workloads above.
Accuracy Validation
Waterfill preserves model semantics because it does not alter the router's logical top-k decisions. The routed experts selected by the model remain the same, and the shared expert remains the same shared expert. Waterfill only changes which physical EP rank executes the shared expert slot.
LPLB preserves model semantics for the same structural reason. It never alters the router's logical top-k; it only chooses which physical replica of a selected logical expert executes each token. Because all replicas of a logical expert hold identical weights, the result a token receives is independent of which replica processes it. This is the same accuracy guarantee EPLB and the dynamic policy already rely on.
How to Use
Enable Waterfill
Waterfill is enabled through the DeepEP MoE path. A representative launch command is:
python3 -m sglang.launch_server \
--model-path /path/to/DeepSeek-V3 \
--tp 16 \
--dp-size 16 \
--nnodes 2 \
--node-rank ${NODE_RANK} \
--dist-init-addr ${HEAD_NODE_IP}:${PORT} \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code \
--moe-a2a-backend deepep \
--deepep-mode normal \
--enable-dp-attention \
--enable-deepep-waterfill \
--init-expert-location /path/to/expert_distribution.pt
The important flags are:
--moe-a2a-backend deepep: use DeepEP for MoE all-to-all dispatch.--enable-deepep-waterfill: enable the shared expert fusion and Waterfill path.--init-expert-location: optionally initialize expert placement and rank-load metadata from collected expert distribution statistics.
DeepSeek V4 support uses the HashTopK path added in #25391.
Enable LPLB
LPLB is selected through the EP dispatch algorithm on the DeepEP MoE path. Because LPLB balances tokens across redundant replicas, it requires an EPLB placement that actually contains redundant experts. A representative two-node launch command is:
python3 -m sglang.launch_server \
--model-path /path/to/DeepSeek-R1 \
--tp 16 \
--dp-size 16 \
--ep-size 16 \
--nnodes 2 \
--node-rank ${NODE_RANK} \
--dist-init-addr ${HEAD_NODE_IP}:${PORT} \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code \
--moe-a2a-backend deepep \
--deepep-mode normal \
--enable-dp-attention \
--ep-num-redundant-experts 16 \
--ep-dispatch-algorithm lp \
--init-expert-location /path/to/expert_stats.pt
The important flags are:
--ep-dispatch-algorithm lp: select the LPLB linear-programming dispatcher in place of the defaultstaticor the uniform-randomdynamicpolicy.--ep-num-redundant-experts: create redundant physical replicas for hot logical experts. LPLB has nothing to balance without them — this is why thered0rows above show no LPLB gain.--init-expert-location: load the static EPLB placement (the physical-to-logical map, including the redundant slots) collected from an expert-distribution recording run. The replica count here must be consistent with--ep-num-redundant-experts.
Acknowledgment
This work builds on the SGLang DeepEP and MoE serving stack and was developed through community collaboration in the SGLang project.
We thank the SGLang maintainers and reviewers for discussions, reviews, and integration support across the related PRs:
- #20089: Fuse shared expert into MoE dispatch under EP
- #19290: Add Waterfill load balancing for shared expert dispatch
- #25391: Support DeepSeek V4 DeepEP Waterfill
- #24515: LPLB: linear-programming load balancer for MoE expert parallelism
We gratefully acknowledge the people who contributed to this work:
- NVIDIA team: Xuting Zhou, Fei Liang, and Aichen Feng
- SGLang team: Cheng Wan
We also thank DeepSeek for open-sourcing their LPLB work at deepseek-ai/LPLB, whose linear-programming formulation for balancing tokens across redundant expert replicas inspired the SGLang LPLB integration described here.