SGLang 将 SSD-LLaMA 的核心思路引入 MoE 推理：把无法容纳在 VRAM 和主机内存中的路由专家放到 NVMe SSD 上，只加载路由器选中的专家，并利用 Expert Pack 布局、直接 I/O、锁页暂存、异步 H2D 传输以及 GPU 缓存，将 SSD 容量转化为实用的后备存储层。
1. 引言：把 VRAM 问题转化为存储问题
DeepSeek-V4-Flash 和 Kimi-K3 的总参数容量远远超出单块消费级 GPU 的 VRAM。因此，常规部署需要多块 GPU，或者数百 GB、有时甚至数 TB 的主机内存。这种容量需求在前沿模型能力与本地硬件之间形成了一道巨大的壁垒。
SGLang 的 SSD 支撑 Expert Pack 路径采用了不同的方法。路由专家权重保留在 NVMe SSD 上。路由器对每个 token 只激活一小部分专家，因此运行时只移动那些尚未缓存到 GPU 的已选中专家。Expert Pack 将每一层/专家对的权重重组为可直接寻址的连续专家块。运行时通过直接 I/O 将该专家块读入对齐的锁页主机缓冲区，然后异步传输到 GPU 缓存。
这条路径改变的是模型权重的存储与交付方式，而非模型计算本身。它不会对选中的专家进行剪枝、替换、合并或跳过，也不会降低 Expert Top-K。其结果是提供了一种切实可行的方式，用 Intel Ultra5 230F CPU、32 GB 内存、TiPro9000 2 TB 磁盘以及 32 GB VRAM 的 RTX 5090 来运行 DeepSeek-V4-Flash 以及已验证的纯文本 Kimi-K3 路径。
MoE 计算是稀疏的，但模型容量并非如此
混合专家模型将前馈网络拆分为众多专家。路由器为某个 token 对各专家打分后，只有一小部分专家参与该 token 的计算。其余专家对该 token 处于闲置状态。
路由器的选择会随 token 和提示词而变化。因此，尽管任一时刻只有一小部分工作集处于活跃状态，完整的专家池仍必须保持可用。量化可以减小产物体积，但并不能消除存储专家池的需求。因此，MoE 推理具有两个不同的特性：
- 每个 token 的计算是稀疏的；
- 必须存储和交付的专家总容量非常庞大。
这正是 SSD 作为有用后备层级的原因。它提供的容量远超消费级 VRAM 或 RAM，而现代 PCIe 5.0 NVMe SSD 提供的顺序带宽足以让精心设计的交付路径变得可行。只有当布局、读取路径和缓存策略与专家级访问模式相匹配时，SSD 容量才能成为可执行的模型内存。
容量与成本的差异
容量与成本的对比让这一权衡一目了然。以下数字仅为容量下限，并非完整系统价格：
该图并不意味着 SSD 和 DRAM 具有相同的延迟，也不意味着仅购买一块 SSD 就足以运行该模型。它表明，将完整的专家池放入 VRAM 或 DRAM 会很快变得不切实际，而使用 SSD 提供容量、并用一个有界的 GPU 缓存承载活跃工作集，则可以大幅降低硬件门槛。
SGLang 的 SSD Expert Pack 方案
SSD-LLaMA 的核心思想是将 SSD、RAM 和 VRAM 作为一个由运行时控制的存储层级来管理。完整的专家池保留在高容量层级中，而有限的 VRAM 则保留那些观测到复用率最高的专家。
SGLang 的 Expert Pack 是在 SGLang MoE 运行时内对该思想中最重要的以专家为中心的部分的实现：
- Expert Pack 将层/专家对变为一个可独立寻址的连续专家块。
O_DIRECT和对齐的 pinned 缓冲区消除了额外的 page-cache 暂存拷贝。- 一个按字节预算管理的 LFU/LRU GPU 缓存会保留被复用的完整专家。
当前的 SGLang 路径并不声称复现了 SSD-LLaMA 论文中的每一个机制。SGLang 的实现是以 GPU 为中心的：pinned 主机内存是一个有界的传输暂存区，而不是持久的主机专家缓存，并且当前功能不需要论文中的 CPU 专家执行或无损失 CUDA 解压缩。明确保留这一区别，才能使功能边界精确清晰。
2. 为什么原生 GGUF 加载路径还不够
在原始的 GGUF 或多分片张量布局下，同一个专家的 gate、up 和 down 权重可能位于不同的文件区域。因此，一次 router 命中就可能触发多次小规模读取、张量名称查找和暂存操作。
显式的按需读取避免了推测性预取，但会把完整的 SSD 和 H2D 延迟暴露在当前 MoE 层的关键路径上。router 产生结果后，GPU 必须等待被选中的专家。预取可以隐藏部分延迟，但它有两个根本性局限：
- 正确的专家仍可能到达得太晚，因为路由结果只有在上一步计算完成后才能得知；
- 错误的预测会消耗 SSD 带宽、暂存空间和 GPU 缓存容量，而此后实际被选中的专家仍然必须被读取。
因此，SGLang 首先改变专家的物理布局，然后降低每一次不可避免的缓存未命中的代价。
为什么原生 GGUF 无法直接使用专家级 O_DIRECT
原生 GGUF 是一种面向张量的模型容器，而不是面向专家（expert）的直接 I/O 存储。它的元数据和张量载荷都是围绕单个张量来组织的，而单个专家的 gate、up 和 down 权重可能分散在文件的不同区域，或跨越多个分片。原始的加载路径通常使用解析器、mmap 或带缓冲的文件读取，因此应用程序看到的是可分页的、由页缓存支持的映射，而不是预先分配的、对齐的 DMA 目标地址。
O_DIRECT 要求以下所有条件都由调用方控制：
- 与存储和文件系统约定对齐的文件偏移量；
- 与该约定对齐的读取长度；
- 由用户提供的缓冲区，其地址同样对齐且适合该读取操作。
原生 GGUF 文件中的任意张量切片并不提供这种专家级别的约定。它的偏移量可能未对齐，它的长度可能不是所需块大小的整数倍，而且一个专家所需的三个张量并不保证构成一个连续区间。调用方可以发出带有填充的、各自对齐的多次读取，然后在另一个缓冲区中重建该专家，但这样就放弃了主要优势：它重新引入了多次读取和额外的组装工作，而原始的 mmap/页缓存路径仍然无法将页缓存页面本身用作 O_DIRECT 的目标地址。
Expert Pack 是一种离线转换，使直接 I/O 变得切实可行。它将一个层/专家对的所有角色放入一个经过填充和对齐的专家块中，在清单中记录其精确偏移量和长度，并提供一个固定缓冲区，其地址满足相同的约定。运行时随后即可读取和传输一个完整的专家，而无需让原生 GGUF 布局表现得像直接 I/O 布局。
3. Expert Pack：按专家组织权重
连续的专家布局
SGLang Expert Pack v1 将一个 (layer, expert) 对视为一个完整的专家。一个 DeepSeek 专家包含 gate、up 和 down 角色。清单记录每个角色的张量边界、格式、完整性信息以及 pack 偏移量。
逻辑布局如下：
运行时不扫描文件来查找张量名称。它从 pack 元数据中推导出专家偏移量：
expert_offset = data_start
+ (layer * num_experts + expert) * expert_stride
专家块内的角色偏移量也会被验证。因此，一次清单查找即可解析出完整的专家读取范围。运行时可以根据 read_splits 将该范围拆分为有限数量的并行任务。
该布局改变的是权重在 SSD 上的物理组织方式，而非其张量内容、量化格式、路由决策或模型数学。Kimi-K3 使用单独的 GGML Expert Pack 适配器。当前已验证的输入由 38 个 Q2_K GGUF 分片组成；其路由专家对 gate/up 使用 Q2_K，对 down 使用 Q3_K。
直接 I/O 需要对齐
直接 I/O 无法像普通 read() 那样使用任意文件偏移、长度和用户缓冲区。SGLang 运行时验证：
- 每个专家的 Expert Pack 偏移；
- 每个读取范围的起始位置和长度；
- 每个固定暂存缓冲区的地址。
当前实现检查 4096 字节对齐。如果 pack 或暂存缓冲区不满足该约定，初始化就会失败，而不是在推理期间静默回退到不受控制的路径。
4. 关键优化：移除 page cache -> pinned memory 拷贝
这是 Expert Pack 与传统文件读取路径之间最重要的差异之一。
传统缓冲 I/O
传统文件读取通常经过操作系统页缓存：
页缓存是由内核管理的文件缓存。它与 CUDA 可用于异步 H2D 传输的页锁定用户内存不是一回事。要发起异步 H2D 传输，应用程序通常需要准备一个固定（pinned）缓冲区。因此，文件数据必须先从页缓存复制到该固定缓冲区，GPU 传输才能开始。从应用程序的角度来看，这次从页缓存到固定缓冲区的交接是一次同步的 CPU 内存复制：主机侧的暂存步骤必须完成，H2D 操作才有一个有效的固定源缓冲区。它本身并不是一次 cudaMemcpyAsync 操作。
这既不是 SSD 读取，也不是 H2D 传输。相反，它是 CPU 在主机侧执行的一次额外的同步内存复制操作：它从页缓存支撑的内存中读取数据，并将其写入一个固定的暂存缓冲区。对于一个大型专家（expert）而言，这意味着要对整个专家大小的数据进行一次读取和一次写入，消耗主机内存带宽，并在 GPU 传输能够进行之前引入一次额外的内核态到用户态的暂存交接。
使用直接 I/O 的 Expert Pack
当 direct_io=True 时，SGLang 使用 O_DIRECT 打开 Expert Pack，并将读取目标设为一个预分配的、对齐的固定暂存缓冲区：
读取目标已经是 CUDA 所需的固定缓冲区，因此下面这个中间步骤被移除了：
page cache -> pinned memory
这并不是该复制操作的更快实现。该复制操作已从数据路径中移除。一个简化的成本模型是：
Traditional path:
T = T(SSD -> page cache)
+ T(page cache -> pinned)
+ T(pinned -> GPU)
+ T(sync)
Expert Pack direct I/O:
T = T(SSD -> pinned)
+ T(pinned -> GPU)
O_DIRECT 并不会让 SSD 的物理带宽增加。它只是从端到端路径中去掉了一次完整的主机内存遍历，从而可以带来以下好处：
- 少一次主机内存读写，减轻 CPU 和内存带宽压力；
- 少一次内核页缓存与用户态暂存之间的同步交接；
- 不会有庞大的 expert 数据负载污染页缓存，并与无关数据争抢空间；
- 一个已完成的 expert 块可以直接进入 H2D 路径，无需经过页缓存暂存拷贝；
- 同一个 expert 级别的约定可复用于每一对 layer/expert。
页缓存拷贝这一说法仅适用于 direct_io=True。当前的 SGLang Expert Pack 加载器以及 DeepSeek/Kimi 5090 启动器默认启用该选项。如果显式禁用了直接 I/O，该路径可能会再次经过页缓存以及一次额外的暂存拷贝。
5. GPU/VRAM 缓存：让工作集靠近计算
GPU 缓存是将重复的专家访问转化为本地 VRAM 命中的机制。Expert Pack 并不是单个张量片段的缓存：一个缓存条目包含完整的gate/up/down一个(layer, expert)对的完整数据。将完整的专家保持在一起很重要，因为被选中的专家需要其所有角色参与计算。只缓存一个角色仍会迫使其他角色被读取，无法消除缓存未命中的开销。
该缓存按字节预算而非按专家数量预算。由于不同模型和适配器每个专家的载荷大小不同，运行时根据可用显存预算推导出可用的槽位数量：
usable_vram = min(requested_cache, free_vram - reserve)
slot_count = floor(usable_vram / expert_payload_bytes)
预留量保护模型、CUDA 运行时、激活值以及其他非缓存分配所需的内存。如果由此得到的槽位数量无法容纳一个完整的 top-k 工作集，初始化就会失败。这使得缓存契约变得明确：缓存预算不允许占用当前 MoE 计算所需的内存。
在缓存命中时，运行时会复用已驻留的专家，不会读取 Expert Pack，也不会为该专家发起 H2D 传输。如果此前的传输仍处于挂起状态，CUDA event 会保护消费者，避免其观察到仅部分安装的槽位。在缓存未命中时，运行时会选择一个牺牲槽位，将完整的专家读入可复用的 pinned 暂存缓冲区，把专家复制到 GPU 槽位，并且只有在传输 event 就绪后才发布该槽位。
替换策略结合了频率和新近度。运行时记录每个 (layer, expert) 被选中的频率以及最后一次使用的时间。被频繁选中的专家比冷门专家更难被驱逐；在效用相近的条目中，较旧的条目是更好的淘汰对象。当前 top-k 请求的活跃专家受到保护，不会被驱逐，因此缓存无法驱逐它即将执行的工作集。
源 Expert Pack 是不可变的。因此驱逐一个 GPU 条目无需写回：专家始终可以从其记录的 SSD 偏移量重建。这使得 VRAM 缓存管理比脏数据缓存更简单，并使替换策略专注于复用价值而非持久性。
运行时暴露了使缓存行为可度量的计数器：
cache_hits和cache_misses；cache_evictions；pack_read_bytes；h2d_bytes；fallback_count和io_errors。
这些计数器将缓存问题与 I/O 问题区分开来。命中率低意味着 VRAM 预算或工作负载局部性不足；命中率良好但 pack_read_bytes 和 h2d_bytes 较高，则可能表明在某个特定阶段活跃集大于缓存。io_errors 报告观测到的 I/O 失败，而 fallback_count 是诊断遥测数据，其含义取决于经过插桩的回退路径；两者都不是缓存性能指标。
当前的执行顺序将专家缓存未命中置于其所服务的 MoE 计算的关键路径上。acquire() 将路由 ID 复制到 CPU，等待 SSD 读取 future，将专家级 H2D 传输入队，并让当前 CUDA 流等待这些传输事件。只有当这些事件就绪后，apply() 才会启动 MoE 内核。在交付阶段，不同缺失专家的读取和传输可能重叠，但当前路径并未将该交付与消费这些专家的 MoE 计算重叠。
因此，当前路径的简化逐步模型为：
T_step ~= T(miss delivery) + T(GPU compute)
在此，T(miss delivery) 包括路由 ID 准备、SSD 读取、暂存、H2D 提交，以及使所选专家可用所需的等待。在 GPU 缓存命中时，SSD 读取和 H2D 部分可以跳过。跨步流水线可能会改变这一模型，但这不属于此处描述的执行路径。实际结果取决于 SSD 带宽、访问分布、缓存命中率、暂存槽数量以及专家形状。
6. 实验设置
我们在单台消费级机器上，用 SGLang 的 SSD Expert Pack 路径评估了 DeepSeek-V4-Flash 和 Kimi-K3：一颗 Intel Ultra5 230F CPU、32 GB 内存、一块 TiPro9000 2 TB 硬盘，以及一块拥有 32 GB VRAM 的 RTX 5090。该工作负载代表的是端点推理，而非批量服务：十个固定请求逐个发送，只有前一个请求完成后才会发起下一个请求。
测试集包含五个 Alpaca 请求和五个 MMLU 请求。两组对比使用相同的提示词顺序、temperature 0、默认 EOS 处理方式，以及 200-token 的生成目标。结果部分报告了平均 prefill 和 decode 速率，以及 SGLang 缓存命中率和 SSD 流量。确切的版本、文件准备和服务器命令记录在结果之后。
Expert Pack 通过 --load-format expert_pack 显式选择；普通的 auto、safetensors 和 gguf 加载路径保持不变。DeepSeek-V4-Flash 和 Kimi-K3 的详细复现步骤见第 9 节。
7. 正确性边界
Expert Pack 是一种权重布局与交付优化，而不是近似推理算法。其正确性契约是：
- 路由器选中的每一个专家都会被执行；
- 专家 Top-K 保持不变；
- 被选中的专家不会被另一个常驻专家替换；
- 被选中的专家不会被剪枝、跳过或合并；
- 该 pack 与 manifest 会按照配置在结构、维度和密码学层面得到验证；
fallback_count和io_errors会被报告出来，而不是静默隐藏 I/O 失败。
验证表明，DeepSeek-V4-Flash 在多个提示词类别上生成的答案与 Ollama 语义等价。Kimi-K3 与 SGLang 的 200-token 参考输出一致；全部 92 个路由层均使用 io_errors=0 执行了 Top-16 专家。fallback_count 作为诊断遥测被保留，但当前路径并未为每一种假设的回退情况暴露可插桩的增量，因此零值不被用作独立的正确性证明。正确性改由路由/输出审计和结构化 pack 检查来确立。
8. 性能结果
所有 SGLang、Ollama 和 llama.cpp 的测量均使用第 9 节中记录的测试环境和版本。这些数据描述的是该硬件条件下的验证结果；token 数量和运行时特定的软件设置仍与各对比中所述一致。此前的汇总表已废弃；以下数据现为 token 速率对比的规范呈现。
经过验证的权重文件和生成的 Expert Pack 占用：
| 模型 | 原始权重文件 | 权重体积 | Expert Pack 体积 |
|---|---|---|---|
| DeepSeek-V4-Flash | 单个 Ollama MXFP4 GGUF blob | 155.10 GB | 147.18 GB |
| Kimi-K3 | 38 个 Q2_K GGUF 分片 | 1009.51 GB（约 1.01 TB） | 985.61 GB |
这些是经过校验的权重载荷的文件大小。Pack 索引、锁、清单及其他元数据另行计算。
DeepSeek-V4-Flash 对比 Baseline
该对比使用了十个共享请求：五个 Alpaca 和五个 MMLU。两个运行时每个请求最多生成 200 个 token。图表报告了每个数据集的平均 prefill 和 decode token 速率。
相对于 Baseline，SGLang 在 Alpaca 上把 prefill 提升了 2.28 倍，在 MMLU 上提升了 3.39 倍。Decode 则分别提升了 6.92 倍和 6.55 倍。
下面简要报告各数据集的底层均值。速率单位为 tokens 每秒，是每个数据集中五条记录的算术平均值。
| 数据集 | 基线 prefill | SGLang prefill | Prefill 提升 | 基线 decode | SGLang decode | Decode 提升 |
|---|---|---|---|---|---|---|
| Alpaca（n=5） | 1.108 | 2.532 | 2.28x | 0.288 | 1.992 | 6.92x |
| MMLU（n=5） | 1.223 | 4.141 | 3.39x | 0.282 | 1.846 | 6.55x |
Kimi-K3 对比 llama.cpp
该对比在两个运行时中使用同样的十个固定请求：五个 Alpaca 和五个 MMLU。两个客户端均使用 temperature 0 和默认 EOS 处理；每个请求恰好生成 200 个 completion token，因此解码对比现在在提示词集合、停止行为和输出长度上均已匹配。图表报告了每个数据集的平均 prefill 和 decode token 速率。
相对于 llama.cpp，SGLang 在 Alpaca 上把 prefill 提升了 6.96 倍，在 MMLU 上提升了 5.80 倍。Decode 分别提升了 3.30 倍和 3.52 倍。这十个请求的汇总使用了第 9 节中描述的十条保留请求记录。
专家缓存命中率与 SSD 流量
Token 速率应结合缓存遥测数据一起解读。以下由 Python 生成的柱状图报告了唯一键 VRAM 缓存命中率以及每生成一个 token 的平均 SSD 流量。cache_hits以及cache_misses计数唯一(layer, expert)键在acquire()每次更新内对键去重；它们并非按 token 统计的路由器边访问。缓存命中可避免该专家的 SSD 读取和 H2D 传输。SSD 流量指标是每请求总量在 prefill 和 decode 上的未加权平均值，按生成的 completion token 数归一化，因此并非仅 decode 的流量。两幅图均仅展示 SGLang 遥测数据，因此单一序列由周围文字标注，而非图例。pack_read_bytes在 prefill 和 decode 之间，按生成的 completion token 数归一化，因此并非仅 decode 的流量。两幅图均仅展示 SGLang 遥测数据，因此单一序列由周围文字标注，而非图例。
DeepSeek 使用了完整的十次请求运行：五次 Alpaca 和五次 MMLU 请求，每次生成 200 个 token 的补全。其唯一键 VRAM 缓存命中率在 Alpaca 上为 54.2%，在 MMLU 上为 46.4%。合并预填充和解码后的平均 SSD 流量，分别为每生成一个 token 1.66 和 2.04 十进制 GB。
Kimi 的数据使用了全部十次完成的请求：五次 Alpaca 和五次 MMLU，每次生成 200 个 token 的补全。未加权的每请求唯一键 VRAM 缓存命中率分别为 17.0% 和 22.7%。相应的平均 SSD 流量，合并预填充和解码后，分别为每生成一个 token 22.10 和 27.63 十进制 GB。
与 DeepSeek 的差异在预期之内：Kimi 的配置为 GPU 专家缓存预留了 5 GiB，而 DeepSeek 的运行预留了约 21 GiB，并且两个适配器具有不同的专家载荷大小和路由行为。
这一改进并非来自某一个孤立的更快的复制原语。它是以下因素的综合效果：
- Expert Pack 将分散的张量访问转变为可寻址的连续专家读取；
- 直接 I/O 消除了
page cache -> pinned memory复制； - 固定暂存区为专家级 H2D 提供了 CUDA 兼容的主机来源，无需页缓存暂存；
- GPU 缓存在命中时跳过 SSD 读取和 H2D 传输。
9. 详细复现
本节给出结果表格之后的完整复现流程。两个小节使用相同的十个固定请求。它们是端点风格的测试：客户端按顺序发送十个 HTTP 请求，在发送下一个请求之前等待每个响应，并使用并发数 1。不使用批量请求。
这些请求是五个 Alpaca 样本和五个 MMLU 样本，按此顺序：
| # | 样本 ID | 数据集 | 提示词 |
|---|---|---|---|
| 1 | alpaca-37246 | Alpaca | 总结电影《玩具总动员》 |
| 2 | mmlu-abstract_algebra-14 | MMLU | 回答这道多项选择题。选出正确选项并简要解释你的答案。 题目：求 n = 10 时 S_n 中元素的最大可能阶数。 A. 6 B. 12 C. 30 D. 105 |
| 3 | alpaca-50812 | Alpaca | 给定一个物品清单，建议一项有趣的活动。 输入：铅笔、纸、马克笔 |
| 4 | mmlu-moral_disputes-8315 | MMLU | 回答这道多项选择题。选出正确选项并简要解释你的答案。 问题：根据哈丁的观点，"棘轮效应"指的是 A. 人口过剩不会影响贫困人口的数量。 B. 人口过剩导致食物银行的建立，从而有助于遏制贫困率。 C. 世界饥饿与贫困促使人们承认免于饥饿的权利。 D. 利用世界食物银行来救济饥饿者会导致一系列不断升级的紧急状况。 |
| 5 | alpaca-9907 | Alpaca | 将这句话从西班牙语翻译成英语：El sol no brilla hoy. |
| 6 | mmlu-high_school_macroeconomics-3940 | MMLU | 回答这道多项选择题。选出正确选项并简要解释你的答案。 题目：政府借贷产生的挤出效应最准确的描述是 A. 紧缩性财政政策导致利率下降，从而使总需求向右移动。 B. 扩张性财政政策导致利率上升，从而使总需求向左移动。 C. 总统增加货币供应量从而降低实际利率并增加总需求的效应。 D. 听到中央银行行长表示他认为经济正处于衰退之中对经济产生的影响。 |
| 7 | alpaca-40699 | Alpaca | 举一个 AI 算法中可能存在的偏见的例子。 |
| 8 | mmlu-professional_law-10971 | MMLU | 回答这道多项选择题。选出正确选项并简要解释你的答案。 问题：房主拥有一处处于自然状态的地产，上面有一栋房子。该地产上没有任何类型的填土。邻居拥有东侧相邻的地产，修建了一条车道，其西侧边界沿着房主地产的边界。挖掘机将车道挖了五英尺深。土地开始沿着开挖线下沉，房主大约三英尺的土地塌落到车道中，使她这部分地产变得无法使用。房主要求邻居填土以支撑所造成的侵蚀。此事并未进行，侵蚀继续发生。房主提起诉讼，请求颁发禁令，强制邻居修建并维护一堵挡土墙。法院会判原告/房主胜诉吗？ A. 会，因为挖掘属于异常危险活动，邻居对因该违法行为造成的任何损害承担绝对责任。 B. 会，因为每位土地所有者都有权获得处于自然状态土壤的侧向支撑。 C. 不会，因为邻居没有进入相邻土地，并将所有挖掘都限制在自己的土地内。 D. 不会，侧向支撑权是一项普通法权利，但几乎在所有州都已被成文法废除，因此该权利已不再存在。 |
| 9 | alpaca-34440 | Alpaca | 为一家餐厅制作一道包含以下食材的菜单菜品。 输入：三文鱼、牛油果、菠菜 |
| 10 | mmlu-jurisprudence-6660 | MMLU | 回答这道多项选择题。选出正确选项并简要解释你的答案。 问题：以下哪一项是对伦理相对主义敌视人权的最有力反驳？ A. 功利主义 B. 社群主义。 C. 认知主义。 D. 实证主义。 |
每个请求使用 temperature 0、默认 EOS 处理和 200-token 目标长度。客户端记录每个请求的 prompt tokens、completion tokens、TTFT/prefill 耗时以及 decode 耗时。第 8 节中的表格报告的是每个数据集中五次请求的算术平均值。
在下面的 SGLang 命令中，model-meta 是用于初始化服务器的轻量级目录，其中存放模型配置和 tokenizer 元数据。它不是原始的 GGUF 权重文件。原始 GGUF 仍是源产物，而 Expert Pack 提供路由专家负载；加载器通过 source_path、pack_path 和 manifest 将它们连接起来。
9.1 DeepSeek-V4-Flash：SGLang 与 Ollama
版本与工作负载
SGLang 运行使用提交81c9f837f19ff8dfe1a9fcd1abfc6069dd28d2ec在分支support_deepseek-v4_and_kimi-k3_on_ssd。基线使用 Ollama0.33.1及其托管的 llama.cpp 运行器，提交为d222767c7。双方串行运行上述十个请求，一次一个请求，使用相同的采样设置。
启动 Ollama 基线服务器
在拉取模型并发送请求之前，先启动 Ollama 服务：
OLLAMA_HOST=127.0.0.1:11435 ollama serve >/tmp/deepseek-ollama.log 2>&1 &
准备
-
在基线服务运行的情况下，从 Ollama 模型页面 拉取经过验证的 DeepSeek-V4-Flash MXFP4 GGUF blob，并使用
ollama show --modelfile获取其本地文件路径。对应的模型卡是 Hugging Face 上的 DeepSeek-V4-Flash-0731：ollama pull frob/deepseek-v4-flash-0731 ollama show --modelfile frob/deepseek-v4-flash-0731记录的 blob SHA-256 为
947ac34c08c0e5c5752ac76398f934b3b6b4075cfe915ba43dd5ac754900a4cd，Ollama manifest 的 SHA-256 为882b1398c0ca4e7ec8ca0a501fd8c4372f780f690536a3ec17ffc75306569ed3。 -
安装 SGLang 检出并手动构建 DeepSeek Expert Pack。为
--model-config使用匹配的 DeepSeek 模型配置 JSON：cd /path/to/sglang-latest-deepseek-v4-kimi-k3-ssd python3 -m pip install -e 'python' python3 tools/expert_pack/prepare_deepseek_pack.py \ --gguf /path/to/deepseek-v4-flash-0731.gguf \ --model-config /path/to/deepseek-v4-flash-config.json \ --safety-margin-gib 16这会在源 GGUF 旁边创建或复用
DeepSeek-V4-Flash.expert-pack及其DeepSeek-V4-Flash.expert-pack.manifest.json。 -
验证生成的 Pack 并创建服务器所使用的元数据。此命令不会启动服务；由于 Pack 已在上一步构建完成，验证阶段会复用它而不是重新构建：
python3 examples/runtime/deepseek_v4/benchmark_deepseek_5090.py \ --gguf /path/to/deepseek-v4-flash-0731.gguf \ --validate-only生成的元数据文件存储在
${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/deepseek-v4-flash/<fingerprint>/model-meta/下：config.json、generation_config.json、tokenizer.json、tokenizer_config.json和metadata.json。这里的
<fingerprint>是从源 GGUF 状态和准备格式派生出的短哈希。它在本地生成，用于隔离不同源文件的产物；它不是一个固定的模型名称，也不是必须下载的目录。
启动 SGLang
以下是直接运行 Expert Pack 服务器的命令。哈希值在启动前从生成的清单中读取。
GGUF=/path/to/deepseek-v4-flash-0731.gguf
ARTIFACT_DIR=${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/deepseek-v4-flash/<fingerprint>
MODEL_META="$ARTIFACT_DIR/model-meta"
PACK_PATH="$(dirname "$GGUF")/DeepSeek-V4-Flash.expert-pack"
MANIFEST_PATH="$(dirname "$GGUF")/DeepSeek-V4-Flash.expert-pack.manifest.json"
STATS_PATH="$ARTIFACT_DIR/deepseek-v4-expert-pack.stats.json"
SOURCE_SHA256="$(jq -r '.source.sha256' "$MANIFEST_PATH")"
OLLAMA_MANIFEST_SHA256="$(jq -r '.model.model_identity_sha256 // .model.ollama_manifest_sha256' "$MANIFEST_PATH")"
CONFIG_SHA256="$(jq -r '.model.config_sha256' "$MANIFEST_PATH")"
python3 -m sglang.launch_server \
--model-path "$MODEL_META" \
--tokenizer-path "$MODEL_META" \
--trust-remote-code \
--load-format expert_pack \
--model-loader-extra-config "{\"pack_path\":\"$PACK_PATH\",\"manifest_path\":\"$MANIFEST_PATH\",\"source_path\":\"$GGUF\",\"source_sha256\":\"$SOURCE_SHA256\",\"ollama_manifest_sha256\":\"$OLLAMA_MANIFEST_SHA256\",\"config_sha256\":\"$CONFIG_SHA256\",\"cache_vram_mib\":21504,\"cache_vram_reserve_mib\":2048,\"stage_slots\":12,\"read_splits\":4,\"direct_io\":true,\"stats_flush_interval\":43,\"stats_path\":\"$STATS_PATH\"}" \
--attention-backend dsv4 \
--tp-size 1 --ep-size 1 \
--disable-cuda-graph --disable-flashinfer-autotune \
--disable-shared-experts-fusion --skip-server-warmup \
--max-running-requests 1 --mem-fraction-static 0.96 \
--watchdog-timeout 1800 --host 127.0.0.1 --port 30001
将同样的十行请求发送到正在运行的 Ollama /api/generate 端点。保留的客户端使用 num_predict=200、temperature=0、固定种子，并且一次只发一个请求；它会写入结果表所使用的逐请求 JSONL 记录和摘要。
9.2 Kimi-K3：SGLang 与 llama.cpp
版本与工作负载
SGLang 运行使用相同的提交 81c9f837f19ff8dfe1a9fcd1abfc6069dd28d2ec。基线使用 llama.cpp 提交 5fff128451d7603857597ee1fc18ac1dfb90f148。上述十个 Alpaca/MMLU 请求以串行方式逐个发送，temperature 为 0，使用默认 EOS 处理，并在两个运行时上均以 200-token 为目标。
准备
-
从 Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED 下载 38 个纯文本 Q2_K GGUF 分片：
hf download Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED \ --include "KIMI-K3-MXP4-DERISKED-Q2_K-*.gguf" \ --local-dir /path/to/kimi-k3 -
从 moonshotai/Kimi-K3 的已记录修订版本下载 tokenizer 和配置文件：
hf download moonshotai/Kimi-K3 \ config.json tokenizer_config.json generation_config.json \ tokenization_kimi.py encoding_k3.py tiktoken.model \ --revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 \ --local-dir /path/to/kimi-k3-tokenizer -
手动构建 Kimi Expert Pack。
--gguf指向第一个带编号的分片；脚本会在该目录中发现全部 38 个分片。Kimi 模型配置即所下载的config.json，其中包含text_config：cd /path/to/sglang-latest-deepseek-v4-kimi-k3-ssd python3 tools/expert_pack/prepare_kimi_pack.py \ --gguf /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \ --model-config /path/to/kimi-k3-tokenizer/config.json \ --safety-margin-gib 2这会在 GGUF 分片旁创建
KIMI-K3-MXP4-DERISKED-Q2_K.expert-major.pack。它是一个独立的 GGML Expert Pack；经过验证的路由专家在 gate/up 上使用 Q2_K，在 down 上使用 Q3_K。 -
创建 SGLang 所需的模型元数据和清单。此准备模式不会启动服务：
python3 examples/runtime/kimi_k3/benchmark_kimi_k3_5090.py \ --gguf /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \ --max-new-tokens 200 --direct-io --read-splits 1 --prepare-only它会创建
model-meta/以及kimi-k3-expert-pack.manifest.json在……之下${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/kimi-k3/<fingerprint>/。metadata 目录包含重写后的文本配置和复制过来的 tokenizer 文件。manifest 记录了分片清单、张量布局、Pack 索引、模型配置和 tokenizer 哈希值。一次常规的 benchmark 运行还会额外生成 stats JSON、report JSON 和服务器日志。
开始使用 SGLang
GGUF_DIR=/path/to/kimi-k3
PACK_PATH="$GGUF_DIR/KIMI-K3-MXP4-DERISKED-Q2_K.expert-major.pack"
ARTIFACT_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/kimi-k3/<fingerprint>"
MODEL_META="$ARTIFACT_DIR/model-meta"
MANIFEST_PATH="$ARTIFACT_DIR/kimi-k3-expert-pack.manifest.json"
STATS_PATH="$ARTIFACT_DIR/kimi-k3-expert-pack.stats.json"
python3 -m sglang.launch_server \
--model-path "$MODEL_META" \
--tokenizer-path "$MODEL_META" \
--trust-remote-code \
--load-format expert_pack \
--model-loader-extra-config "{\"pack_path\":\"$PACK_PATH\",\"manifest_path\":\"$MANIFEST_PATH\",\"cache_vram_mib\":5120,\"cache_vram_reserve_mib\":1536,\"stage_slots\":16,\"read_splits\":1,\"direct_io\":true,\"stats_flush_interval\":92,\"stats_path\":\"$STATS_PATH\",\"verify_pack_sha256\":false}" \
--tp-size 1 --ep-size 1 \
--disable-cuda-graph --disable-shared-experts-fusion \
--disable-radix-cache --mamba-radix-cache-strategy no_buffer \
--disable-overlap-schedule --skip-server-warmup \
--chunked-prefill-size 64 --watchdog-timeout 1800 \
--max-running-requests 1 --mem-fraction-static 0.98 \
--host 127.0.0.1 --port 30001
启动 llama.cpp 基线服务器
启动固定版本的 llama.cpp 构建，使用 CPU 专家执行：
/path/to/llama.cpp/build/bin/llama-server \
-m /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \
-ngl -1 --cpu-moe --host 127.0.0.1 --port 8081 \
-t 16 -tb 16 --threads-http 16 -np 1 -c 4096 \
--no-warmup --metrics \
--log-file /path/to/kimi-k3-llama-cpp/server.log
llama.cpp 客户端将同样的十个提示词逐个发送给 /completion，并配合使用 cache_prompt=false、temperature=0 和 n_predict=200。
10. 条件与限制
SSD 容量与准备时间
Expert Pack 需要额外的 SSD 容量。PR 记录显示，构建 DeepSeek-V4-Flash pack 预计需要 5-10 分钟，首次运行就绪大约需要 8-15 分钟。在保留的测量中，Kimi-K3 pack 的构建耗时 29 分 42 秒，首次就绪大约需要 35-45 分钟。
38 个 Kimi 源分片和生成的 Expert Pack 总共占用约 1.814 TiB；这些测量使用的是 TiPro9000 2 TB 磁盘，而实际部署建议使用 4 TB SSD 以留出余量。
直接 I/O
O_DIRECT 需要平台支持，并且要求文件偏移量、读取长度和用户缓冲区地址对齐。SGLang 在运行时初始化期间会以失败关闭（fail closed）方式处理。如果平台不支持直接 I/O，或者 pack 不满足对齐约定，那么结果就不应被描述为直接 I/O 性能。
缓存与工作负载
- 较小的 GPU 缓存会导致更多未命中，使 SSD 读取和 H2D 传输更频繁地出现在关键路径上。
- 提示词或工作负载分布的变化会改变热点专家，因此单个请求的热点集合并不能保证适配所有工作负载。
- 如果完整的专家池已经能放进 GPU 显存，那么 Expert Pack 就增加了一条不必要的数据路径，并不是合适的部署模式。
- 如果工作负载以 GPU 计算为主，那么省去主机拷贝所带来的收益可能会被计算时间掩盖。
- 如果 SSD 的随机读取行为、排队或热稳定性表现不佳，那么增加
read_splits和暂存槽位可能会带来排队和内存压力，而非提升吞吐量。
当前功能边界
该功能聚焦于路由专家的 SSD 投递与 GPU 缓存。它不提供 SSD KV-cache 卸载，也不改变请求调度。它是一条显式选择启用的 Expert Pack 路径，并非对每一种 SGLang 模型加载格式的全局替换。
11. 结论
SSD Expert Pack 并非简单地用更慢的磁盘替代 GPU 显存。它围绕 MoE 推理的稀疏访问模式重新设计了权重投递方式：
router selects a small set of experts
-> Expert Pack resolves their offsets
-> O_DIRECT reads into aligned pinned buffers
-> expert-level asynchronous H2D fills the GPU cache
-> the complete expert becomes available for computation
省去 page cache -> pinned memory 拷贝是一个容易被忽视的细节，但它是一项具体的端到端优化。传统路径先读入操作系统页缓存，然后将数据拷贝到可供 CUDA 使用的固定内存中。借助直接 I/O，Expert Pack 将预分配的固定缓冲区作为读取目标，从而消除了整个主机内存的搬运和同步交接。
这就是 SGLang 如何将 SSD-LLaMA 的核心思路应用到真实的 DeepSeek-V4-Flash 和 Kimi-K3 集成中：完整的专家池保留在高容量 SSD 上，一个有界的 GPU 缓存保留当前工作集，运行时只移动由路由器选中的专家。超大规模 MoE 模型不再需要堆叠足够的 VRAM 或 DRAM 来容纳完整模型，而是可以在一块消费级 GPU 搭配高速 NVMe SSD 上运行。
SGLang brings the core idea of SSD-LLaMA to MoE inference: keep routed experts that do not fit in VRAM and host RAM on an NVMe SSD, load only the experts selected by the router, and use Expert Pack layout, direct I/O, pinned staging, asynchronous H2D transfers, and a GPU cache to turn SSD capacity into a practical backing tier.
1. Introduction: turning a VRAM problem into a storage problem
The total parameter capacity of DeepSeek-V4-Flash and Kimi-K3 is far beyond the VRAM of a single consumer GPU. A conventional deployment therefore needs multiple GPUs or hundreds of gigabytes, and sometimes terabytes, of host memory. That capacity requirement creates a large barrier between frontier model capability and local hardware.
SGLang's SSD-backed Expert Pack path takes a different approach. Routed expert weights remain on an NVMe SSD. The router activates only a small subset of experts for each token, so the runtime moves only the selected experts that are not already cached to the GPU. Expert Pack reorganizes the weights of each layer/expert pair into a directly addressable contiguous expert block. The runtime reads that expert block into an aligned pinned host buffer using direct I/O, then transfers it to a GPU cache asynchronously.
This path changes how model weights are stored and delivered, not the model computation. It does not prune, replace, merge, or skip selected experts, and it does not reduce Expert Top-K. The result is a practical way to run DeepSeek-V4-Flash and the validated text-only Kimi-K3 path with an Intel Ultra5 230F CPU, 32 GB memory, a TiPro9000 2 TB disk, and an RTX 5090 with 32 GB VRAM.
MoE computation is sparse, but model capacity is not
Mixture-of-Experts models split the feed-forward network into many experts. After the router scores the experts for a token, only a small subset participates in that token's computation. The remaining experts are idle for that token.
The router's choice changes across tokens and prompts. The complete expert pool must therefore remain available even though only a small working set is active at any one time. Quantization reduces the artifact size, but it does not remove the need to store the expert pool. MoE inference consequently has two different properties:
- per-token computation is sparse;
- the total expert capacity that must be stored and delivered is very large.
This is why SSD is a useful backing tier. It provides much more capacity than consumer VRAM or RAM, and modern PCIe 5.0 NVMe SSDs provide enough sequential bandwidth to make a carefully designed delivery path viable. SSD capacity becomes executable model memory only when the layout, read path, and cache policy match expert-level access patterns.
The capacity-cost difference
A capacity-cost comparison makes the trade-off clear. The following figures are capacity-only lower bounds, not complete system prices:
The figure does not mean SSD and DRAM have the same latency, or that buying an SSD alone is sufficient to run the model. It shows that placing the complete expert pool in VRAM or DRAM quickly becomes impractical, while using SSD for capacity and a bounded GPU cache for the active working set can substantially lower the hardware barrier.
SGLang's SSD Expert Pack approach
The central SSD-LLaMA idea is to manage SSD, RAM, and VRAM as a runtime-controlled storage hierarchy. The complete expert pool stays in the high-capacity tier, while limited VRAM retains the experts with the highest observed reuse.
SGLang's Expert Pack is an implementation of the most important expert-centric parts of that idea inside the SGLang MoE runtime:
- Expert Pack makes a layer/expert pair an independently addressable contiguous expert block.
O_DIRECTand aligned pinned buffers remove the extra page-cache staging copy.- A byte-budgeted LFU/LRU GPU cache retains complete experts that are reused.
The current SGLang path does not claim to reproduce every mechanism in the SSD-LLaMA paper. SGLang's implementation is GPU-centric: pinned host memory is a bounded transfer staging area, not a persistent host expert cache, and the current feature does not require the paper's CPU expert execution or lossless CUDA decompression. Keeping this distinction explicit makes the feature boundary precise.
2. Why the native GGUF loading path is not enough
With the original GGUF or multi-shard tensor layout, the gate, up, and down weights of one expert may be located in different file regions. One router hit can therefore trigger several small reads, tensor-name lookups, and staging operations.
An explicit on-demand read avoids speculative prefetch, but exposes the full SSD and H2D latency on the critical path of the current MoE layer. After the router produces its result, the GPU must wait for the selected experts. Prefetching can hide part of that latency, but it has two fundamental limitations:
- the correct expert may still arrive too late because the routing result is only known after the previous computation;
- an incorrect prediction consumes SSD bandwidth, staging space, and GPU cache capacity, after which the actually selected expert must still be read.
SGLang therefore first changes the physical expert layout, then reduces the cost of every unavoidable cache miss.
Why native GGUF cannot directly use expert-level O_DIRECT
Native GGUF is a tensor-oriented model container, not an expert-oriented direct-I/O store. Its metadata and tensor payloads are organized around individual tensors, and a single expert's gate, up, and down weights may be separated across file regions or across multiple shards. The original loading path commonly uses a parser, mmap, or buffered file reads, so the application sees pageable page-cache-backed mappings rather than a preallocated aligned DMA destination.
O_DIRECT requires all of the following to be controlled by the caller:
- a file offset aligned to the storage and filesystem contract;
- a read length aligned to that contract;
- a user-provided buffer whose address is also aligned and suitable for the read.
An arbitrary tensor slice in a native GGUF file does not provide that expert-level contract. Its offset may not be aligned, its length may not be a multiple of the required block size, and the three tensors needed for one expert are not guaranteed to form one contiguous range. A caller could issue separate aligned reads with padding and then reconstruct the expert in another buffer, but that gives up the main benefit: it reintroduces multiple reads and extra assembly work, while the original mmap/page-cache path still cannot use the page-cache pages themselves as an O_DIRECT destination.
Expert Pack is the offline transformation that makes direct I/O practical. It places all roles of one layer/expert pair into one padded, aligned expert block, records its exact offset and length in the manifest, and provides a pinned buffer whose address satisfies the same contract. The runtime can then read and transfer a complete expert without asking the native GGUF layout to behave like a direct-I/O layout.
3. Expert Pack: organizing weights by expert
Contiguous expert layout
SGLang Expert Pack v1 treats one (layer, expert) pair as one complete expert. A DeepSeek expert contains the gate, up, and down roles. The manifest records each role's tensor boundaries, format, integrity information, and pack offset.
The logical layout is:
The runtime does not scan the file for tensor names. It derives the expert offset from the pack metadata:
expert_offset = data_start
+ (layer * num_experts + expert) * expert_stride
Role offsets inside the expert block are validated as well. One manifest lookup can therefore resolve the complete expert read range. The runtime may split that range into a bounded number of parallel tasks according to read_splits.
The layout changes the physical organization of weights on SSD, not their tensor contents, quantization formats, routing decisions, or model mathematics. Kimi-K3 uses a separate GGML Expert Pack adapter. The currently validated input consists of 38 Q2_K GGUF shards; its routed experts use Q2_K for gate/up and Q3_K for down.
Alignment is required for direct I/O
Direct I/O cannot use arbitrary file offsets, lengths, and user buffers in the same way as ordinary read(). The SGLang runtime validates:
- Expert Pack offsets for each expert;
- the start and length of every read range;
- the address of every pinned staging buffer.
The current implementation checks 4096-byte alignment. If the pack or staging buffers do not satisfy the contract, initialization fails instead of silently falling back to an uncontrolled path during inference.
4. The key optimization: removing the page cache -> pinned memory copy
This is one of the most important differences between Expert Pack and a conventional file-reading path.
Traditional buffered I/O
Traditional file reads normally go through the operating system page cache:
The page cache is a kernel-managed file cache. It is not the same thing as the page-locked user memory that CUDA can use for asynchronous H2D. To issue an asynchronous H2D transfer, the application normally prepares a pinned buffer. The file data therefore has to be copied from the page cache into that pinned buffer before the GPU transfer can start. From the application's perspective, this page-cache-to-pinned handoff is a synchronous CPU memory copy: the host-side staging step must complete before the H2D operation has a valid pinned source buffer. It is not itself a cudaMemcpyAsync operation.
This is neither SSD reads nor H2D transfers. Rather, it is an extra synchronous memory copy operation performed by the CPU on the host side: it reads the payload from page-cache-backed memory and writes it into a pinned staging buffer. For a large expert, this translates to a read and write of the full expert size, consuming host memory bandwidth and introducing an additional kernel-to-userland staging handoff before the GPU transfer can proceed.
Expert Pack with direct I/O
When direct_io=True, SGLang opens the Expert Pack with O_DIRECT and makes the read target a preallocated, aligned pinned staging buffer:
The read target is already the pinned buffer required by CUDA, so the intermediate step below is removed:
page cache -> pinned memory
This is not a faster implementation of that copy. The copy is removed from the data path. A simplified cost model is:
Traditional path:
T = T(SSD -> page cache)
+ T(page cache -> pinned)
+ T(pinned -> GPU)
+ T(sync)
Expert Pack direct I/O:
T = T(SSD -> pinned)
+ T(pinned -> GPU)
O_DIRECT does not make the SSD's physical bandwidth increase. It removes one full host-memory traversal from the end-to-end path, which can provide these benefits:
- one less host-memory read and write, reducing CPU and memory-bandwidth pressure;
- one less synchronization handoff between kernel page cache and user-space staging;
- no large expert payload polluting the page cache and competing with unrelated data;
- a completed expert block can enter the H2D path without a page-cache staging copy;
- the same expert-level contract is reused for every layer/expert pair.
The page-cache-copy claim applies only to direct_io=True. The current SGLang Expert Pack loader and the DeepSeek/Kimi 5090 launchers enable this option by default. If direct I/O is explicitly disabled, the path may go through the page cache and an additional staging copy again.
5. GPU/VRAM cache: keeping the working set close to compute
The GPU cache is the mechanism that turns repeated expert access into a local VRAM hit. Expert Pack is not a cache of individual tensor fragments: one cache entry contains the complete gate/up/down data for one (layer, expert) pair. Keeping the complete expert together matters because a selected expert needs all of its roles for computation. Caching only one role would still force the other roles to be read and would not remove the cache-miss cost.
The cache is byte-budgeted rather than expert-count-budgeted. Since different models and adapters have different per-expert payload sizes, the runtime derives the number of available slots from the usable VRAM budget:
usable_vram = min(requested_cache, free_vram - reserve)
slot_count = floor(usable_vram / expert_payload_bytes)
The reserve protects memory needed by the model, CUDA runtime, activations, and other non-cache allocations. Initialization fails if the resulting slot count cannot hold one complete top-k working set. This makes the cache contract explicit: a cache budget is not allowed to consume the memory required for the current MoE computation.
On a cache hit, the runtime reuses the resident expert and does not read the Expert Pack or issue an H2D for that expert. If a previous transfer is still pending, a CUDA event protects the consumer from observing a partially installed slot. On a cache miss, the runtime selects a victim slot, reads the complete expert into a reusable pinned staging buffer, copies the expert into the GPU slot, and publishes the slot only after the transfer event is ready.
Replacement combines frequency and recency. The runtime records how often each (layer, expert) is selected and when it was last used. A frequently selected expert is harder to evict than a cold expert; among similarly useful entries, an older entry is a better victim. Active experts for the current top-k request are protected from eviction, so the cache cannot evict the working set it is about to execute.
The source Expert Pack is immutable. Evicting a GPU entry therefore requires no write-back: the expert can always be reconstructed from its recorded SSD offset. This makes VRAM cache management simpler than a dirty data cache and keeps replacement focused on reuse value rather than persistence.
The runtime exposes counters that make cache behavior measurable:
cache_hitsandcache_misses;cache_evictions;pack_read_bytes;h2d_bytes;fallback_countandio_errors.
These counters distinguish a cache problem from an I/O problem. A low hit rate means the VRAM budget or workload locality is insufficient; high pack_read_bytes and h2d_bytes with a good hit rate may instead indicate that the active set is larger than the cache during a particular phase. io_errors reports observed I/O failures, while fallback_count is diagnostic telemetry whose meaning depends on an instrumented fallback path; neither is a cache-performance metric.
The current execution order places an expert-cache miss on the critical path for the MoE computation it feeds. acquire() copies routing IDs to CPU, waits for the SSD read futures, enqueues the expert-level H2D transfers, and makes the current CUDA stream wait for their transfer events. Only after those events are ready does apply() launch the MoE kernels. Reads and transfers for different missing experts may overlap during the delivery phase, but the current path does not overlap that delivery with the MoE computation that consumes the experts.
A simplified per-step model for the current path is therefore:
T_step ~= T(miss delivery) + T(GPU compute)
Here, T(miss delivery) includes route-ID preparation, SSD reads, staging, H2D submission, and the waits needed to make the selected experts available. On a GPU cache hit, the SSD-read and H2D portions can be skipped. A cross-step pipeline could change this model, but that is not part of the execution path described here. The actual result depends on SSD bandwidth, access distribution, cache hit rate, staging-slot count, and expert shapes.
6. Experimental setup
We evaluated DeepSeek-V4-Flash and Kimi-K3 with SGLang's SSD Expert Pack path on one consumer machine: an Intel Ultra5 230F CPU, 32 GB of memory, a TiPro9000 2 TB disk, and an RTX 5090 with 32 GB of VRAM. The workload represents endpoint inference rather than batched serving: the ten fixed requests are sent one at a time, and the next request starts only after the previous one finishes.
The test set contains five Alpaca requests and five MMLU requests. Both comparisons use the same prompt order, temperature 0, default EOS handling, and a 200-token generation target. The result section reports the mean prefill and decode rates, together with the SGLang cache hit rate and SSD traffic. Exact revisions, file preparation, and server commands are documented after the results.
Expert Pack is explicitly selected with --load-format expert_pack; ordinary auto, safetensors, and gguf loading paths are unchanged. The detailed DeepSeek-V4-Flash and Kimi-K3 reproduction procedures are in Section 9.
7. Correctness boundary
Expert Pack is a weight-layout and delivery optimization, not an approximate-inference algorithm. The correctness contract is:
- every expert selected by the router is executed;
- Expert Top-K is unchanged;
- a selected expert is not replaced by a different resident expert;
- selected experts are not pruned, skipped, or merged;
- the pack and manifest are structurally, dimensionally, and cryptographically validated as configured;
fallback_countandio_errorsare reported instead of silently hiding I/O failures.
Validation showed that DeepSeek-V4-Flash produced semantically equivalent answers to Ollama across multiple prompt categories. Kimi-K3 matched the 200-token SGLang reference output; all 92 routed layers executed Top-16 experts with io_errors=0. fallback_count is retained as diagnostic telemetry, but the current path does not expose an instrumented increment for every hypothetical fallback, so zero is not used as an independent correctness proof. Correctness is instead established by the route/output audit and the structural pack checks.
8. Performance results
All SGLang, Ollama, and llama.cpp measurements use the test environment and revisions documented in Section 9. The figures describe validation results under this hardware condition; token counts and runtime-specific software settings remain as stated in each comparison. The earlier summary tables are retired; the figures below are now the canonical presentation of the token-rate comparison.
The validated weight files and generated Expert Packs occupy:
| Model | Original weight files | Weight size | Expert Pack size |
|---|---|---|---|
| DeepSeek-V4-Flash | One Ollama MXFP4 GGUF blob | 155.10 GB | 147.18 GB |
| Kimi-K3 | 38 Q2_K GGUF shards | 1009.51 GB (about 1.01 TB) | 985.61 GB |
These are file sizes for the validated weight payloads. Pack indexes, locks, manifests, and other metadata are separate.
DeepSeek-V4-Flash vs. Baseline
The comparison uses ten shared requests: five Alpaca and five MMLU. Both runtimes generated up to 200 tokens per request. The chart reports mean prefill and decode token rates for each dataset.
Relative to Baseline, SGLang improves prefill by 2.28x on Alpaca and 3.39x on MMLU. Decode improves by 6.92x and 6.55x, respectively.
The underlying per-dataset means are compactly reported below. Rates are in tokens per second and are arithmetic means over the five records in each dataset.
| Dataset | Baseline prefill | SGLang prefill | Prefill gain | Baseline decode | SGLang decode | Decode gain |
|---|---|---|---|---|---|---|
| Alpaca (n=5) | 1.108 | 2.532 | 2.28x | 0.288 | 1.992 | 6.92x |
| MMLU (n=5) | 1.223 | 4.141 | 3.39x | 0.282 | 1.846 | 6.55x |
Kimi-K3 vs. llama.cpp
The comparison uses the same ten fixed requests in both runtimes: five Alpaca and five MMLU. Both clients use temperature 0 and default EOS handling; each request generated exactly 200 completion tokens, so the decode comparison is now matched for prompt set, stop behavior, and output length. The chart reports mean prefill and decode token rates for each dataset.
Relative to llama.cpp, SGLang improves prefill by 6.96x on Alpaca and 5.80x on MMLU. Decode improves by 3.30x and 3.52x, respectively. The ten-request aggregates use the ten retained request records described in Section 9.
Expert-cache hit rate and SSD traffic
Token rate should be read together with cache telemetry. The following Python-generated bar charts report the unique-key VRAM cache hit rate and mean SSD traffic per generated token. cache_hits and cache_misses count unique (layer, expert) keys after acquire() de-duplicates keys within each update; they are not per-token router-edge accesses. A cache hit avoids the SSD read and H2D transfer for that expert. The SSD traffic metric is the unweighted per-request mean of total pack_read_bytes across prefill and decode, normalized by generated completion tokens, and is therefore not decode-only traffic. Both figures show SGLang-only telemetry, so the single series is labeled by the surrounding text rather than a legend.
DeepSeek uses the complete ten-request run: five Alpaca and five MMLU requests, each with a 200-token completion. Its unique-key VRAM cache hit rate is 54.2% for Alpaca and 46.4% for MMLU. The mean SSD traffic, combining prefill and decode, is 1.66 and 2.04 decimal GB per generated token, respectively.
The Kimi figure uses all ten completed requests: five Alpaca and five MMLU, each with a 200-token completion. The unweighted per-request unique-key VRAM cache hit rates are 17.0% and 22.7%, respectively. The corresponding mean SSD traffic, combining prefill and decode, is 22.10 and 27.63 decimal GB per generated token. The difference from DeepSeek is expected: Kimi's configuration reserves 5 GiB for the GPU expert cache, while the DeepSeek run reserves about 21 GiB, and the two adapters have different expert payload sizes and routing behavior.
The improvement does not come from one isolated faster-copy primitive. It is the combined effect of:
- Expert Pack turning scattered tensor accesses into addressable contiguous expert reads;
- direct I/O removing the
page cache -> pinned memorycopy; - pinned staging provides a CUDA-compatible host source for expert-level H2D without page-cache staging;
- the GPU cache skipping SSD reads and H2D transfers on a hit.
9. Detailed reproduction
This section gives the complete reproduction flow after the result tables. The two subsections use the same ten fixed requests. They are endpoint-style tests: the client sends ten HTTP requests in order, waits for each response before sending the next request, and uses concurrency 1. No batch of requests is used.
The requests are five Alpaca samples and five MMLU samples, in this order:
| # | Sample ID | Dataset | Prompt |
|---|---|---|---|
| 1 | alpaca-37246 | Alpaca | Summarize the movie "Toy Story" |
| 2 | mmlu-abstract_algebra-14 | MMLU | Answer the multiple-choice question. Select the correct option and briefly explain your answer. Question: Find the maximum possible order for an element of S_n for n = 10. A. 6 B. 12 C. 30 D. 105 |
| 3 | alpaca-50812 | Alpaca | Given a list of items, suggest an interesting activity. Input: pencils, paper, markers |
| 4 | mmlu-moral_disputes-8315 | MMLU | Answer the multiple-choice question. Select the correct option and briefly explain your answer. Question: According to Hardin, the "ratchet effect" refers to the fact that A. overpopulation does not affect the number of people who are poor. B. overpopulation leads to creation of food banks that help curb poverty rates. C. world hunger and poverty leads to recognition of rights not to be hungry. D. the use of a world food bank to feed the hungry leads to an escalating series of emergency situations. |
| 5 | alpaca-9907 | Alpaca | Translate this phrase from Spanish to English: El sol no brilla hoy. |
| 6 | mmlu-high_school_macroeconomics-3940 | MMLU | Answer the multiple-choice question. Select the correct option and briefly explain your answer. Question: The crowding-out effect from government borrowing is best described as A. the rightward shift in AD in response to the decreasing interest rates from contractionary fiscal policy. B. the leftward shift in AD in response to the rising interest rates from expansionary fiscal policy. C. the effect of the President increasing the money supply which decreases real interest rates and increases AD. D. the effect on the economy of hearing the chairperson of the central bank say that he or she believes that the economy is in a recession. |
| 7 | alpaca-40699 | Alpaca | Give an example of a bias that could exist in an AI algorithm. |
| 8 | mmlu-professional_law-10971 | MMLU | Answer the multiple-choice question. Select the correct option and briefly explain your answer. Question: Homeowner owns a property in its natural condition with a house on it. There was no fill of any kind on the property. Neighbor, who owns the adjacent property to the East, built a driveway whose western boundary is along the border of homeowner's property. The excavator dug the driveway five feet deep. The land began to subside along the line of excavation and about three feet of homeowner's land fell off into the driveway, making that part of her property useless. Homeowner demanded that neighbor fill in the property to buttress the erosion created. That was not done and the erosion continued to occur. Homeowner sued and asked for an injunction compelling the neighbor to build and maintain a retaining wall. Will the court rule for the plaintiff/homeowner? A. Yes, because excavation is an abnormally dangerous activity and neighbor is absolutely liable for any damages caused by the violation. B. Yes, because every landowner has a right to the lateral support of the soil in its natural state. C. No, because the neighbor did not go onto the adjacent land and confined all excavation to his own land. D. No, the right to lateral support is a common law right that has been abrogated by statute in virtually all states so that the right no longer exists. |
| 9 | alpaca-34440 | Alpaca | Make a menu item for a restaurant that contains the following ingredients. Input: Salmon, avocado, spinach |
| 10 | mmlu-jurisprudence-6660 | MMLU | Answer the multiple-choice question. Select the correct option and briefly explain your answer. Question: Which of the following is the strongest argument against ethical relativism's hostility to human rights? A. Utilitarianism B. Communitarianism. C. Cognitivism. D. Positivism. |
Each request uses temperature 0, default EOS handling, and a 200-token target. The clients record prompt tokens, completion tokens, TTFT/prefill timing, and decode timing for every request. The tables in Section 8 report arithmetic means over the five requests in each dataset.
In the SGLang commands below, model-meta is the lightweight directory holding the model configuration and tokenizer metadata used to initialize the server. It is not the raw GGUF weight file. The raw GGUF remains the source artifact, while the Expert Pack supplies the routed expert payload; the loader connects them through source_path, pack_path, and the manifest.
9.1 DeepSeek-V4-Flash: SGLang and Ollama
Versions and workload
The SGLang run uses commit 81c9f837f19ff8dfe1a9fcd1abfc6069dd28d2ec on branch support_deepseek-v4_and_kimi-k3_on_ssd. The baseline uses Ollama 0.33.1 with its managed llama.cpp runner at commit d222767c7. Both sides run the ten requests above serially, one request at a time, with the same sampling settings.
Starting the Ollama baseline server
Start the Ollama service before pulling the model and sending requests:
OLLAMA_HOST=127.0.0.1:11435 ollama serve >/tmp/deepseek-ollama.log 2>&1 &
Preparation
-
With the baseline service running, pull the validated DeepSeek-V4-Flash MXFP4 GGUF blob from the Ollama model page and use
ollama show --modelfileto obtain its local file path. The corresponding model card is DeepSeek-V4-Flash-0731 on Hugging Face:ollama pull frob/deepseek-v4-flash-0731 ollama show --modelfile frob/deepseek-v4-flash-0731The recorded blob SHA-256 is
947ac34c08c0e5c5752ac76398f934b3b6b4075cfe915ba43dd5ac754900a4cdand the Ollama manifest SHA-256 is882b1398c0ca4e7ec8ca0a501fd8c4372f780f690536a3ec17ffc75306569ed3. -
Install the SGLang checkout and build the DeepSeek Expert Pack manually. Use a matching DeepSeek model configuration JSON for
--model-config:cd /path/to/sglang-latest-deepseek-v4-kimi-k3-ssd python3 -m pip install -e 'python' python3 tools/expert_pack/prepare_deepseek_pack.py \ --gguf /path/to/deepseek-v4-flash-0731.gguf \ --model-config /path/to/deepseek-v4-flash-config.json \ --safety-margin-gib 16This creates or reuses
DeepSeek-V4-Flash.expert-packand itsDeepSeek-V4-Flash.expert-pack.manifest.jsonbeside the source GGUF. -
Validate the generated Pack and create the metadata used by the server. This command does not start a service; because the Pack was built in the previous step, the validation phase reuses it instead of rebuilding it:
python3 examples/runtime/deepseek_v4/benchmark_deepseek_5090.py \ --gguf /path/to/deepseek-v4-flash-0731.gguf \ --validate-onlyThe generated metadata files are stored under
${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/deepseek-v4-flash/<fingerprint>/model-meta/:config.json,generation_config.json,tokenizer.json,tokenizer_config.json, andmetadata.json.Here
<fingerprint>is a short hash derived from the source GGUF state and preparation format. It is generated locally to isolate artifacts for different source files; it is not a fixed model name or a directory that must be downloaded.
Starting SGLang
The following is the direct Expert Pack server command. The hash values are read from the generated manifest before launch.
GGUF=/path/to/deepseek-v4-flash-0731.gguf
ARTIFACT_DIR=${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/deepseek-v4-flash/<fingerprint>
MODEL_META="$ARTIFACT_DIR/model-meta"
PACK_PATH="$(dirname "$GGUF")/DeepSeek-V4-Flash.expert-pack"
MANIFEST_PATH="$(dirname "$GGUF")/DeepSeek-V4-Flash.expert-pack.manifest.json"
STATS_PATH="$ARTIFACT_DIR/deepseek-v4-expert-pack.stats.json"
SOURCE_SHA256="$(jq -r '.source.sha256' "$MANIFEST_PATH")"
OLLAMA_MANIFEST_SHA256="$(jq -r '.model.model_identity_sha256 // .model.ollama_manifest_sha256' "$MANIFEST_PATH")"
CONFIG_SHA256="$(jq -r '.model.config_sha256' "$MANIFEST_PATH")"
python3 -m sglang.launch_server \
--model-path "$MODEL_META" \
--tokenizer-path "$MODEL_META" \
--trust-remote-code \
--load-format expert_pack \
--model-loader-extra-config "{\"pack_path\":\"$PACK_PATH\",\"manifest_path\":\"$MANIFEST_PATH\",\"source_path\":\"$GGUF\",\"source_sha256\":\"$SOURCE_SHA256\",\"ollama_manifest_sha256\":\"$OLLAMA_MANIFEST_SHA256\",\"config_sha256\":\"$CONFIG_SHA256\",\"cache_vram_mib\":21504,\"cache_vram_reserve_mib\":2048,\"stage_slots\":12,\"read_splits\":4,\"direct_io\":true,\"stats_flush_interval\":43,\"stats_path\":\"$STATS_PATH\"}" \
--attention-backend dsv4 \
--tp-size 1 --ep-size 1 \
--disable-cuda-graph --disable-flashinfer-autotune \
--disable-shared-experts-fusion --skip-server-warmup \
--max-running-requests 1 --mem-fraction-static 0.96 \
--watchdog-timeout 1800 --host 127.0.0.1 --port 30001
Send the same ten rows to the running Ollama /api/generate endpoint. The retained client uses num_predict=200, temperature=0, the fixed seed, and one request at a time; it writes the per-request JSONL records and summary used by the result table.
9.2 Kimi-K3: SGLang and llama.cpp
Versions and workload
The SGLang run uses the same commit 81c9f837f19ff8dfe1a9fcd1abfc6069dd28d2ec. The baseline uses llama.cpp commit 5fff128451d7603857597ee1fc18ac1dfb90f148. The ten Alpaca/MMLU requests above are sent serially, one at a time, with temperature 0, default EOS handling, and a 200-token target on both runtimes.
Preparation
-
Download the 38 text-only Q2_K GGUF shards from Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:
hf download Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED \ --include "KIMI-K3-MXP4-DERISKED-Q2_K-*.gguf" \ --local-dir /path/to/kimi-k3 -
Download the tokenizer and configuration files from moonshotai/Kimi-K3 at the recorded revision:
hf download moonshotai/Kimi-K3 \ config.json tokenizer_config.json generation_config.json \ tokenization_kimi.py encoding_k3.py tiktoken.model \ --revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 \ --local-dir /path/to/kimi-k3-tokenizer -
Build the Kimi Expert Pack manually.
--ggufpoints to the first numbered shard; the script discovers all 38 shards in that directory. The Kimi model configuration is the downloadedconfig.jsoncontainingtext_config:cd /path/to/sglang-latest-deepseek-v4-kimi-k3-ssd python3 tools/expert_pack/prepare_kimi_pack.py \ --gguf /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \ --model-config /path/to/kimi-k3-tokenizer/config.json \ --safety-margin-gib 2This creates
KIMI-K3-MXP4-DERISKED-Q2_K.expert-major.packbeside the GGUF shards. It is a separate GGML Expert Pack; the validated routed experts use Q2_K for gate/up and Q3_K for down. -
Create the model metadata and manifest needed by SGLang. This preparation mode does not start the service:
python3 examples/runtime/kimi_k3/benchmark_kimi_k3_5090.py \ --gguf /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \ --max-new-tokens 200 --direct-io --read-splits 1 --prepare-onlyIt creates
model-meta/andkimi-k3-expert-pack.manifest.jsonunder${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/kimi-k3/<fingerprint>/. The metadata directory contains the rewritten text configuration and copied tokenizer files. The manifest records the shard inventory, tensor layout, Pack index, model configuration, and tokenizer hashes. A normal benchmark run additionally creates the stats JSON, report JSON, and server log.
Starting SGLang
GGUF_DIR=/path/to/kimi-k3
PACK_PATH="$GGUF_DIR/KIMI-K3-MXP4-DERISKED-Q2_K.expert-major.pack"
ARTIFACT_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/kimi-k3/<fingerprint>"
MODEL_META="$ARTIFACT_DIR/model-meta"
MANIFEST_PATH="$ARTIFACT_DIR/kimi-k3-expert-pack.manifest.json"
STATS_PATH="$ARTIFACT_DIR/kimi-k3-expert-pack.stats.json"
python3 -m sglang.launch_server \
--model-path "$MODEL_META" \
--tokenizer-path "$MODEL_META" \
--trust-remote-code \
--load-format expert_pack \
--model-loader-extra-config "{\"pack_path\":\"$PACK_PATH\",\"manifest_path\":\"$MANIFEST_PATH\",\"cache_vram_mib\":5120,\"cache_vram_reserve_mib\":1536,\"stage_slots\":16,\"read_splits\":1,\"direct_io\":true,\"stats_flush_interval\":92,\"stats_path\":\"$STATS_PATH\",\"verify_pack_sha256\":false}" \
--tp-size 1 --ep-size 1 \
--disable-cuda-graph --disable-shared-experts-fusion \
--disable-radix-cache --mamba-radix-cache-strategy no_buffer \
--disable-overlap-schedule --skip-server-warmup \
--chunked-prefill-size 64 --watchdog-timeout 1800 \
--max-running-requests 1 --mem-fraction-static 0.98 \
--host 127.0.0.1 --port 30001
Starting the llama.cpp baseline server
Start the pinned llama.cpp build with CPU expert execution:
/path/to/llama.cpp/build/bin/llama-server \
-m /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \
-ngl -1 --cpu-moe --host 127.0.0.1 --port 8081 \
-t 16 -tb 16 --threads-http 16 -np 1 -c 4096 \
--no-warmup --metrics \
--log-file /path/to/kimi-k3-llama-cpp/server.log
The llama.cpp client sends the same ten prompts to /completion, one at a time, with cache_prompt=false, temperature=0, and n_predict=200.
10. Conditions and limitations
SSD capacity and preparation time
Expert Pack requires additional SSD capacity. The PR records an estimated 5-10 minutes to build the DeepSeek-V4-Flash pack and approximately 8-15 minutes for first-run readiness. Kimi-K3 pack construction took 29 minutes 42 seconds in the retained measurement, with approximately 35-45 minutes to first readiness. The 38 Kimi source shards and generated Expert Pack occupy about 1.814 TiB in total; the measurements use the TiPro9000 2 TB disk, while a 4 TB SSD is recommended for practical deployment headroom.
Direct I/O
O_DIRECT requires platform support and alignment of file offsets, read lengths, and user-buffer addresses. SGLang fails closed during runtime initialization. If the platform does not support direct I/O, or the pack does not satisfy the alignment contract, the result should not be described as direct-I/O performance.
Cache and workload
- A smaller GPU cache creates more misses, putting SSD reads and H2D transfers on the critical path more often.
- A change in prompt or workload distribution can change the hot experts, so one request's hot set is not guaranteed to fit every workload.
- If the complete expert pool already fits in GPU memory, Expert Pack adds an unnecessary data path and is not the right deployment mode.
- If the workload is dominated by GPU computation, the benefit of removing the host copy may be hidden by compute time.
- If SSD random-read behavior, queueing, or thermal stability is poor, increasing
read_splitsand staging slots may add queueing and memory pressure instead of throughput.
Current feature boundary
This feature focuses on routed-expert SSD delivery and GPU caching. It does not provide SSD KV-cache offload and does not change request scheduling. It is an explicit opt-in Expert Pack path, not a global replacement for every SGLang model-loading format.
11. Conclusion
SSD Expert Pack is not simply replacing GPU memory with a slower disk. It redesigns weight delivery around the sparse access pattern of MoE inference:
router selects a small set of experts
-> Expert Pack resolves their offsets
-> O_DIRECT reads into aligned pinned buffers
-> expert-level asynchronous H2D fills the GPU cache
-> the complete expert becomes available for computation
Removing the page cache -> pinned memory copy is an easy detail to miss, but it is a concrete end-to-end optimization. A traditional path reads into the OS page cache and then copies the payload into CUDA-usable pinned memory. With direct I/O, Expert Pack uses the preallocated pinned buffer as the read target and removes that full host-memory movement and synchronization handoff.
This is how SGLang applies the core SSD-LLaMA idea to real DeepSeek-V4-Flash and Kimi-K3 integrations: the complete expert pool remains on high-capacity SSD, a bounded GPU cache retains the current working set, and the runtime moves only the experts selected by the router. Very large MoE models no longer require stacking enough VRAM or DRAM to hold the complete model and can instead run on a consumer GPU paired with a high-speed NVMe SSD.