NVIDIA AICR 集成 k8s-aibom，为 GPU 平台提供运行时 AI 资产清单

As teams deploy and scale AI workloads using NVIDIA Dynamo, Kubeflow, or Slurm, tracking what actually runs requires visibility beyond the platform configuration captured in a recipe. Application teams define and update these workloads independently of the platform recipe, so a reproducible deployment bundle alone cannot establish which workload artifacts are running on each node. Mutable image tags like :latest add another layer of complexity. Two nodes can end up running different code under identical tags, and a node that pulls next week may retrieve an image that was not covered by an earlier security review.

NVIDIA AI Cluster Runtime (AICR), an open source modular component of the NVIDIA DSX AI Factory Platform, now includes a qualified AI bill of materials (AIBOM) component to close that gap. An open source project created by Google, k8s-aibom tracks AI workloads running on Kubernetes. Two stock recipes select k8s-aibom by default: h100-gke-cos-inference and h100-gke-cos-inference-dynamo. Other GKE recipes can opt in during generation with --runtime-inventory enabled. Recipes selecting it can opt out with --runtime-inventory disabled.

AICR turns known-good GPU platform configurations into version-locked recipes and reproducible deployment bundles. Each recipe records the desired configuration for one hardware, cloud, OS, and intent combination: its components, version pins, values, constraints, deployment ordering, and validation expectations. AICR generates reproducible deployment artifacts from fixed recipe, catalog, and bundle inputs. Bundles include checksums; signing is optional through aicr bundle --attest.

Qualification with k8s-aibom is a specific claim, not a rubber stamp. AICR binds the exact upstream source, chart, CRDs, and digest-pinned controller image as one tested set, verifies provenance against Google’s expected build identity, reviews rendered RBAC and security defaults, and exercises installation, upgrade, health, and uninstall on the deployment paths it tested.

This blog will enable you to configure your Kubernetes runtime using AICR to provide a validated, performant configuration for your cluster.

Two ends of the same problem

AICR and k8s-aibom look at different parts of the AI infrastructure lifecycle. AICR helps platform teams generate and validate known-good configurations for GPU-accelerated Kubernetes. It works at the runtime deployment layer: drivers, operators, kernels, Kubernetes components, deployment configuration.

k8s-aibom looks higher up the stack and later in the lifecycle of the cluster. It watches Kubernetes workload and pod APIs and emits a CycloneDX 1.6 AIBOM for each recognized AI workload. Coverage is bounded: Only supported workload kinds carrying a recognized AI signal are inventoried. Coverage includes native Dynamo graphs, NIMService, LeaderWorkerSet, DisaggregatedSet, and CronJob resources. Tracked child workloads are combined into their owner’s AIBOM.

One provides control and reproducibility of the configuration for GPU-accelerated Kubernetes, while the other adds workload discovery and traceability in that cluster. Concretely, AIBOM helps you:

• Find unregistered or shadow AI workloads in namespaces selected for reporting
• Identify models, inference runtimes, agents, and RAG components
• Track external AI APIs and vector databases
• Detect changes or drift in AI dependencies
• Support security reviews, incident response, and compliance audits

Both projects arrived at this from opposite sides of the same problem, and recognized it as such. AICR records qualified deployment intent and evaluates it against a cluster at a point in time. k8s-aibom observes supported AI workloads and records what it found, including how confidently each attribute was determined. Neither view is sufficient for AI governance on its own, and the boundary between them is exactly where accountability is missing today.

Using AICR with k8s-aibom

These examples reflect AICR v1.2.0 and its catalog, which pins k8s-aibom 1.6.2. Use a CLI build matching this catalog to reproduce the examples.

Start with a Google Kubernetes Engine (GKE) cluster running Kubernetes 1.32 or later, with NVIDIA H100 GPU nodes using Container-Optimized OS and GKE-managed drivers and device plugins. The following workflow generates a recipe, renders it into a bundle, deploys it, and validates the result. Nothing is hand-assembled and nothing is tuned by hand.

With the same AICR version, catalog, and generation inputs, recipe resolution produces the same configuration. The last step checks the cluster against it. That is the half AICR brings to this pairing: a validated GPU platform quickly and repeatably. k8s-aibom then keeps inventorying supported AI workloads.

Generate the recipe

aicr recipe \
  --service gke \
  --accelerator h100 \
  --os cos \
  --intent inference \
  --output recipe.yaml

The new k8s-aibom component is selected as part of the recipe, not bolted on afterward. You can query any resolved value without generating a file:

aicr query \
  --service gke --accelerator h100 --os cos --intent inference \
  --selector components.k8s-aibom.version
# 1.6.2

Bundle and deploy

Create a deployment bundle with the default Helm deployer. Replace the example node selectors with labels present on your system and GPU node pools:

aicr bundle \
  --recipe recipe.yaml \
  --system-node-selector nodeGroup=system-pool \
  --accelerated-node-selector nodeGroup=gpu-worker \
  --output ./bundles

The bundle is one directory per component, ordered by the dependency graph, each carrying its values, checksums, and install script. AICR supports a number of deployers via --deployer flag, including helmfile, flux, and argocd.

Verify the generated bundle, then install it:

(cd ./bundles && aicr verify . && chmod +x deploy.sh && ./deploy.sh)

Once deployed, the k8s-aibom controller runs in the k8s-aibom-system namespace. AICR uses the main chart’s CRDs rather than the companion k8s-aibom-crds chart. Helm bundles include a CRD apply step; Flux and Argo CD handle updates through their controllers. Helmfile, Terraform, and Fleet require the documented manual CRD update when the transition calls for it. Check the installed storage version:

kubectl get crd aiboms.aibom.k8saibom.dev \
  -o jsonpath='{.spec.versions[?(@.storage==true)].name}'
# v1beta1

Validate

aicr snapshot --output snapshot.yaml
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml \
  --phase deployment --phase conformance

The k8s-aibom health check runs as part of the deployment phase. It requires the controller deployment to be available, the cluster-scoped AIBOMControllerConfig to report a current Ready=True, and both CRDs to report the storage version the pinned chart expects. A matching storage version does not prove the full CRD schema is current. During upgrades, also verify the target schema and check for Degraded=True with reason SchemaPredatesController; see health and readiness.

Opt a namespace in

Inventory is opt-in per namespace. The label controls which namespaces emit AIBOM records; it does not narrow what the controller reads. Informers still cache workload and pod specs cluster-wide, including container arguments and inline environment values. No sink and no secret access are configured by default, so nothing leaves the cluster without an explicit decision.

kubectl create namespace prod-jobs
kubectl label namespace prod-jobs aibom.k8saibom.dev/enabled=true

Run an inference workload

Supported workloads carrying a recognized AI signal are inventoried automatically when their namespace opts in. For a simple demonstration, here is a vLLM deployment serving Qwen. Note that it pulls vllm/vllm-openai:latest rather than a digest, which is not an uncommon pattern in some environments. The nvidia-h100-mega-80gb selector and toleration must match your GPU pool’s labels and taints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen
  namespace: prod-jobs
spec:
  replicas: 1
  selector:
    matchLabels: {app: vllm-qwen}
  template:
    metadata:
      labels: {app: vllm-qwen}
    spec:
      # Adjust to match your GPU node pool's taints and labels.
      tolerations:
        - key: dedicated
          operator: Equal
          value: gpu-workload
          effect: NoSchedule
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "Qwen/Qwen2.5-0.5B-Instruct",
                 "--max-model-len", "4096",
                 "--gpu-memory-utilization", "0.30"]
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "1"
          volumeMounts:
            - {name: shm, mountPath: /dev/shm}
      volumes:
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 2Gi}

Save this manifest as vllm-qwen.yaml, then create the workload:

kubectl apply -f vllm-qwen.yaml

See the inventory

After applying the workload, you can track the inventory:

kubectl -n prod-jobs get aiboms.aibom.k8saibom.dev

Example output:

NAME                        WORKLOAD     NAME        CATEGORY    RUNTIME   READY   AGE
apps-deployment-vllm-qwen   Deployment   vllm-qwen   inference   vllm      True    28m

The controller automatically classified the workload category and serving runtime. One boundary worth knowing: The record is built from the workload spec, so an AIBOM can exist before any pod is running. AIBOM can appear and report Ready=True while the pod is still pulling its image. An AIBOM tells you what was declared and, once a Pod exists, what it resolved to. It does not prove successful scheduling or execution.

Check the status conditions:

kubectl -n prod-jobs get aibom apps-deployment-vllm-qwen \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'
# Ready=True Synced=True SinkFailed=False Stale=False

Verify the AIBOM

The document is stored inline on the resource when it is small enough, with a published SHA-256 over the canonical bytes:

kubectl -n prod-jobs get aibom apps-deployment-vllm-qwen -o json > aibom.json
jq -r '.status.bomDocument.inline.data' aibom.json | base64 -d > bom.json

jq -r '.status.bomDocument.sha256' aibom.json
shasum -a 256 bom.json | awk '{print $1}'

Here’s where it all comes together:

jq -r '.components[] | "\(.type)\t\(.name)"' bom.json

The response should look something like this:

application              vllm
container                vllm/vllm-openai
machine-learning-model   Qwen/Qwen2.5-0.5B-Instruct

A conventional SBOM tells you which image is running. An AIBOM tells you which model is serving and which runtime is serving it. Attributes carry one of four confidence tiers—declared, inferred, unresolved, or verified—along with an evidence locator, so reviewers can see how each claim was reached.

Resource impact

Upstream measured k8s-aibom v1.6.1 on GKE with NVIDIA Dynamo and AICR’s configuration. The test used CPU-only nodes with workload pods remaining pending, so these figures describe controller inventory overhead:

• Approximately four API-server requests per minute during a 15-min steady-state observation window
• Average controller CPU usage of 2.2 millicores
• Controller memory working set of 28 to 34 MiB
• One AIBOM regeneration when the tested workload’s image tag changed

What comes next

The integration deliberately stops short of a few things, each its own decision rather than an oversight.

Signed runtime observations: The runtime AIBOM is written as a Kubernetes resource, but the observation itself is unsigned. Signing would make captured observations tamper-evident; it would not prove every workload declaration true or establish successful execution. The canonical bytes and published SHA-256 shown above make signing tractable.

Connecting verified provenance: k8s-aibom can verify Sigstore signatures on model claims against operator-configured trust roots, enforce signer-identity constraints, check Rekor inclusion, and assign qualifying claims the verified confidence tier. AICR leaves this verification disabled by default. AICR can also produce transparency-logged bundle attestations when signing is enabled; connecting that platform provenance to runtime AI inventory remains future integration work.

Intent and observed workloads: An AICR recipe describes deployment intent. A runtime AIBOM records observed workload specifications and available pod-status evidence. Correlating them requires shared artifact identities, freshness rules, and mapping between infrastructure and workload coverage. That comparison remains future integration work; an AIBOM does not establish successful execution or complete cluster conformance.

AICR is available at NVIDIA/aicr. k8s-aibom is available at GoogleCloudPlatform/k8s-aibom. Both are open source. If you run GPU workloads on Kubernetes, this integration adds an inventory of recognized AI workloads and evidence showing how each attribute was determined.