# IntBMoE：将块级条件引入专家组合的全参与式混合专家模型

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-18 08:00
- AIHOT 分数：40
- AIHOT 链接：https://aihot.news/items/cmub7i2yd0p8oroln72tla8cu
- 原文链接：https://arxiv.org/abs/2609.21346

## AI 摘要

IntBMoE 通过稠密专家组合与稀疏块执行，解耦 MoE 的参与度、计算量与显存占用，块来自小型学习码本，超网络将每层专家池合并为组合专家。图像分类实验显示其持续优于代表性稀疏与稠密 MoE 基线，语言建模与序列推荐实验验证了泛化能力。该架构已部署于高德生成式推荐系统，服务数亿用户，60ms 延迟预算下在线 A/B 测试 UVCTR 相对提升 2.4%。

## 正文

Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.