NVIDIA 发布开源 Snapshot：GPU 推理负载秒级恢复

NVIDIA Snapshot, a new open-source component of the NVIDIA DSX AI Factory Platform, makes initialized GPU workloads reusable across inference replicas—speeding startup and recovery.

When inference demand spikes or a replica fails, available GPUs don’t translate immediately into serving capacity. Initializing an LLM workload—a process known as a cold start—can take minutes, delaying scaling and recovery.

Snapshot is a checkpoint and restore system for GPU workloads on Kubernetes. It captures a fully initialized workload—including CPU and GPU memory—and restores that state on compatible infrastructure, bringing inference capacity online in seconds instead of minutes. Figure 1 shows the startup sequence. Snapshot skips repeated workload initialization; infrastructure provisioning and container image loading remain outside the measured restore path.

This post explains how Snapshot checkpoints and restores GPU workloads, how platform teams can integrate it, and how restore times compare with cold starts across three inference engines.

Snapshot for AI factory operations

Originally developed for NVIDIA Dynamo inference workers, Snapshot provides checkpoint and restore capabilities that other inference engines, schedulers, controllers, and platforms can integrate.

Snapshot provides two operations for GPU Pods:

• Checkpoint: Once a workload is fully initialized, capture its CPU and GPU memory as a portable artifact.
• Restore: Start a new Pod from that artifact on any node with the same GPU model and driver. The process continues from where it was checkpointed, so weight loading, compilation, and graph capture are skipped.

Both operations use Kubernetes custom resources. To checkpoint a Pod that is already running, create a PodSnapshot that names it. To start a Pod from a template and checkpoint it as soon as it is ready, create a SnapshotJob instead. To restore, add the annotation nvidia.com/restore-from to a new Pod, naming the PodSnapshot to restore from. The workload only needs a small handshake to pause before the checkpoint and resume after the restore. The repository has ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM.

Snapshot can checkpoint and restore compatible workloads that span multiple GPUs on a single node as a single checkpoint artifact across ranks. This supports fast startup for large mixture-of-experts (MoE) models using parallelism strategies such as tensor parallelism (TP), data and expert parallelism (DEP), and decode context parallelism (DCP). Workloads must meet Snapshot’s checkpoint/restore and runtime compatibility requirements. See the example configurations.

Benchmarks

Design partners have also evaluated Snapshot on their own workloads:

• Crusoe, an upstream contributor to Snapshot, reduced GLM-5.3 Pod readiness time from roughly 8 minutes to 31 seconds on NVIDIA HGX B300.
• Saturn Cloud, a token factory platform, benchmarked NVIDIA Snapshot for scale-from-zero inference, cutting vLLM startup from 2–8.5 minutes to 6–34 seconds on NVIDIA HGX B300.

The following benchmarks show how Snapshot accelerates inference startup across vLLM, SGLang, and NVIDIA TensorRT LLM (TRT-LLM)—with models from 600 million parameters to mixture-of-experts (MoE) models with 1 TB checkpoints on 8 GPUs. Across all tested configurations, Snapshot brings a replica back in under 45 seconds, 8x to 36x faster than a cold start.

Methodology

Each configuration was tested using two strategies on the same hardware and software stack:

• Cold start: The workload starts from scratch. The clock runs from process launch until engine initialization returns, before inference. The measurement is divided into four phases (Table 2):
  • Startup: Starting the engine and worker processes, importing Python libraries, and setting up communication between GPUs. Some just-in-time (JIT) kernel compilation also happens here.
  • Model load: Reading the model weights and constructing the model’s tensors, including any engine-specific conversion or repacking done while loading.
  • Initialization: Preparing the loaded model to serve: memory profiling, allocating the key-value (KV) cache and runtime buffers, and other kernel and runtime preparation.
  • Compilation, autotuning, and graph capture: Compiling kernels (for example, with torch.compile), autotuning them for the GPU, and capturing CUDA graphs.
• Snapshot: The workload is restored from its checkpoint. The clock runs from the moment Snapshot is asked to restore until the workload is serving again. The measurement is divided into four stages (Table 3):
  • Agent setup: The Snapshot agent on the node locates the checkpoint artifact and prepares the target Pod and GPUs.
  • Checkpoint/Restore in Userspace (CRIU) restore: CRIU, the standard Linux tool for checkpointing processes, reads the process image from storage into host memory. Speed is set mostly by the storage backend.
  • CUDA restore: Copying GPU state back into GPU memory.
  • Wake/remap: Adapting the process to the new node and waking the engine.

Both measurements exclude pulling the container image and setting up the container runtime. The KV cache release is triggered before every checkpoint, so it is not part of the artifact.

Table 1 also reports checkpoint time and size: how long it takes to capture a fully initialized replica and how much storage the stored artifact requires. This is a one-time cost per model and configuration.

Setup

• GPU: NVIDIA B200
• NVIDIA driver: 615.71.09
• Storage backend: VAST (NFS)
• Inference engines:
  • vLLM 0.31.0
  • SGLang 0.5.21
  • TensorRT LLM 1.3.0

Results

The benchmarks show four main findings.

• Restore times stayed under 43 seconds: Across all tested configurations, Snapshot restored replicas in 3.6 to 42.7 seconds—8x to 36x faster than cold start. The largest MoE models, with checkpoints of about 600 GB to 1 TB across eight GPUs, restored in under 43 seconds compared with cold-start times of 6 to 19 minutes.
• Restoring skips more than weight loading: Cold-start time includes loading weights, compiling and autotuning kernels, and capturing CUDA graphs. Snapshot restores the initialized workload, skipping these repeated steps.
• Checkpointing pays off with the first restored replica: In every tested configuration, checkpoint time plus restore time was shorter than a single cold start. Checkpoint creation took 15 seconds to 5.5 minutes, depending on the model and configuration.
• Moving GPU state dominates large restores: For the eight-GPU MoE configurations, restoring GPU state took 18 to 29 seconds. Restoring CPU process state with CRIU took 4.2 to 10.8 seconds.

Figures 2 to 4 compare cold-start and Snapshot restore times for vLLM, SGLang, and TensorRT LLM across the tested models and parallelism configurations. Table 1 provides the data for every configuration, including checkpoint time and size. All times are in seconds. In the figures and tables, TEP8 denotes tensor and expert parallelism across eight GPUs.

Table 2 breaks down the cold-start time, showing which initialization phases a restore skips.

Table 3 breaks down restore time into agent setup, CRIU restore, CUDA restore, and wake/remap.

Figure 5 shows the restore-time breakdown for vLLM.

Benchmarking tools and reproduction instructions are available in the repository.

Get started

To try Snapshot with a GPU workload on Kubernetes, start with the installation instructions and usage guides in the NVIDIA Snapshot repository.

Try it out with NVIDIA Dynamo to restore workers by following the user guide.

To discuss an integration or contribute feedback, open an issue or join the NVIDIA Dynamo Slack workspace and visit the #snapshot-dev channel.

Visit the NVIDIA booth #170 at KubeCon for Kubernetes deep dives and live demos.

About the Authors