📱 35B parameters. Real-time inference. On a phone.

Meet Lookahead Expert Scheduling (LES), the inference system powering MiniCPM-V 4.7’s 35B-A3B MoE model on edge devices.

🚀 On an iPhone 18 Pro Max, the full system reaches 20.51 tokens per second, a 6.07× speedup over loading expert weights on demand.

How? Three coordinated optimizations:
- 🔮 Scheduling: Predict and prefetch the next layer’s experts while the current layer computes.
- 💾 Storage: Combine contiguous expert layouts with mixed-precision quantization to reduce loading overhead.
- ⚡ Compute: Use DSpark speculative decoding to verify multiple candidate tokens in one pass.

Predictions guide prefetching; the model’s actual router still determines which experts execute. The architecture, expert counts, and routing remain intact. Mixed-precision quantization reduces the weight file by 26.7%, with just a 0.16-point drop in the five-benchmark average.

LES shows how better inference systems can bring larger, more capable models to everyday devices. 🌍

👉 LES: https://github.com/openbmb/llama.cpp-omni/blob/main/docs/les/lookahead-expert-scheduling.md
🔗 GitHub: http://github.com/OpenBMB/MiniCPM-V

#MiniCPM #OnDeviceAI #MoE #EdgeAI