NVIDIA NeMo Gym 助力 Siemens Fuse EDA 智能体将基准综合得分从约 55% 提升至约 95%

AI agents offer remarkable productivity gains in complex chip design workflows, but their tendency to generate plausible yet incorrect artifacts makes reliability a major barrier to production adoption. In multistage engineering workflows, an undetected error early in setup can silently propagate through downstream stages and final artifact generation. Evaluating and refining these agents against verified final outputs, rather than isolated intermediate steps, is essential for building trustworthy automation.

NVIDIA NeMo Gym, an open library for evaluating and improving models and agents using environments, addresses this challenge. The NeMo Gym agentic evaluation environment is designed to diagnose failure modes and track error propagation across complex, multistage workflows. By establishing a domain-specific verifier to inspect final outputs against verified reference artifacts, NeMo Gym enables engineers to trace complete rollout trajectories and identify root causes behind subtle errors.

This post walks through how NeMo Gym was used to evaluate and improve the reliability of the Siemens Fuse EDA AI Agent when calling characterization tools. The Siemens Fuse team used NeMo Gym to scale up evaluations and verify the resulting Liberty (.lib) files against known references and tracing—both common and rare failures through each rollout. Those findings guided changes to the agent’s skill and Model Context Protocol (MCP) tool descriptions, increasing the measured benchmark composite score from approximately 55% to ~95% across a benchmark of representative standard-cell characterization tasks, while reducing turns and output tokens.

Siemens’ work provides a practical example for chip design teams and EDA developers on how to leverage the NVIDIA AI software stack and open libraries to improve agent reliability and chip design workflows.

Automating standard-cell characterization

Standard-cell characterization is the process of generating accurate timing, power, and noise models for logic cells across operating conditions. It is foundational to chip design and required whenever a semiconductor foundry issues a new process design kit (PDK). Each new PDK carries SPICE transistor models; design rules; and process, voltage, and temperature (PVT) definitions that connect the manufacturing process to design tools. When a foundry updates a PDK, existing characterization datasets may no longer accurately describe the updated devices. Timing and power models must then be regenerated and validated for every affected cell across combinations of input slew, output load, and PVT conditions.

The Solido Characterization Suite agent automates this process. The agent orchestrates tools from the broader Solido Characterization Suite through Siemens Fuse EDA AI Agent, which exposes MCP servers and skills to common coding harnesses including Claude Code, Codex, and Cursor.

MCP interfaces make these accelerated tools accessible to engineering agents, while skills guide their use in characterization workflows. Faster simulation helps agents complete engineering iterations sooner, while NVIDIA NeMo Gym helps developers measure and improve how reliably and efficiently agents orchestrate those tools.

The Solido Characterization Suite agent must perform tool setup, large-scale SPICE simulation, distributed compute, Liberty file generation, debugging, and final validation (Figure 1). Errors often produce plausible outputs instead of clean failures, leaving incorrect thresholds, missing corners, or incomplete cells hidden. Because these stages depended on one another, an early setup error could propagate into every step that followed. Defining a rigorous evaluation protocol was a key step in preparing the Solido Characterization Suite agent for production workflows.

Measuring and improving agent reliability with NeMo Gym

Part of the NVIDIA NeMo collection of open libraries for agent specialization and optimization, NeMo Gym integrates with the standard-cell characterization flow inside a controlled evaluation loop (Figure 2). It defines the task, tools, state, and verifier for each rollout so that attempts started from a clean state and produced comparable results. It also records thinking traces, input and output messages, and tool actions.

This enabled engineers to run the following optimization loop:

• Run baseline rollouts with a versioned task and verifier in a custom NeMo Gym environment
• Inspect trajectories, failed tool calls, and verifier diagnostics
• Identify ambiguous instructions, unproductive actions, and incorrect tool selection
• Refine the agent’s skills and MCP tool descriptions
• Repeat the same tasks under the same evaluation conditions
• Compare verified outcomes and efficiency per successful result

Reward design and artifact verification

As the team analyzed agent trajectories, end-to-end evaluation uncovered failures that intermediate checks missed. For example, final Liberty verification exposed missing cells and operating conditions, incorrect slew thresholds, and timing or power values outside tolerance. These findings show why engineering agents must be evaluated against final artifacts rather than individual tool calls.

To make the work repeatable, the team implemented verifiers that provide a final composite reward score based on the following metrics:

• Coverage: Whether the agent produced every expected cell and PVT corner.
• Validity: Whether the delivered Liberty files are well-formed and usable by the validation tools.
• Relative validation: Whether the output introduced failures beyond those already present in the reference.
• Accuracy: The fraction of expected timing and power values that falls within the defined tolerance.
• Integrity: Whether the characterization workflow produced the complete result rather than reusing, truncating, or reformatting existing values.

The reward score is defined as:

\(R=\mathbf{1}\left[\text{Coverage}\right]\times\mathbf{1}\left[\text{Validity}\right]\times\mathbf{1}\left[\text{Relative Validation}\right]\times\mathbf{1}\left[\text{Integrity}\right]\times\frac{X_{\text{tol}}}{X_{\text{total}}}\)

Here, \(X_{\mathrm{tol}}\) is the number of values in the output that are within tolerance, and \(X_{\mathrm{total}}\) is the total number of output values. Their ratio represents the fraction of output values within tolerance.

Each indicator \(\mathbf{1}[\cdot]\) equals \(1\) when its corresponding gate passes and \(0\) when it fails. The reward is zero if any gate fails.

Using this verifier, the team systematically evaluated and tuned the agent’s performance across multiple iterations.

Optimizing the agent

The team used the open source GF180MCU PDK to benchmark the Solido Characterization Suite agent on Liberty-file generation. The dataset includes 229 standard cells across 36 PVT corners, using Solido AFS for SPICE simulation.

A custom agentic environment was built using NeMo Gym. For each run, the characterization flow was restricted to two cells across 36 PVT corners, as this smaller subset was sufficient for tuning the agentic workflow. Each run took approximately 1.5 to 2 hours, and the team conducted approximately 25 rollouts using Claude Opus 4.8 running in Claude Code as the fixed model and agent harness.

Across successive evaluation rounds, the team refined the agent’s characterization skills and MCP tool descriptions. To improve tool selection, they simplified overlapping file and shell interfaces and defined tool inputs with explicit JSON schemas, making characterization tools easier for the agent to call.

Following these refinements, the mean composite verification score across evaluated rollouts increased from approximately 55% to 95%. The improved workflow also required fewer agent turns and output tokens. Agent turns fell from 155 to 53, a reduction of approximately 66%, while output tokens fell from approximately 81K to 29K, a reduction of approximately 64%.

Developing reliable agents in other domains using the same method

The same evaluation method applies to engineering agents in EDA, CAE, and other engineering domains:

• Evaluate the final engineering artifacts and not just whether individual tool calls succeeded.
• Write a domain-specific grader/verifier that detects missing, truncated, reformatted, or reused results to prevent agent cheating.
• Collect large numbers of rollouts to target improved reliability through optimized skills and MCP tool interfaces.

Get started with NeMo Gym

Use the NeMo Gym Quickstart guide to run an initial evaluation. Then apply the approach outlined in this post to one representative task: define a valid final artifact, build a domain-specific verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent evaluation conditions.

For more details about how Solido Fuse brings generative and agentic AI to Solido Characterization Suite agent, watch the Self-Verifying, Long-Running Agentic Characterization Workflow Demo.

Learn more about the collaboration between NVIDIA and Siemens.

Acknowledgments

We would like to acknowledge Gunjan Shah of NVIDIA, along with Tim Wheler, Lakshminarayanan Ramesh, and Carl Hofmeister of Siemens for their key contributions to developing the agent and establishing the evaluation methodology.