Prime Intellect 与 NVIDIA 后训练 Nemotron 3.5 Super VL，AutomationBench-AA 提升 23.7 分

Enterprise agents need to do more than explain what should happen: they must update the right records, follow business policies, and complete work across applications. Prime Intellect and NVIDIA built 2,500 verified training tasks to teach this, and 100 GRPO steps, about 11 hours of training on 128 NVIDIA GB200 GPUs, took NVIDIA Nemotron 3.5 Super VL from 34.0% to 57.7% on AutomationBench-AA (public split).

Key takeaways

1. Nemotron can be specialized for enterprise work. Post-training on 2,500 verified tasks took NVIDIA Nemotron 3.5 Super VL from 34.0% to 57.7% on AutomationBench-AA, a 23.7-point gain over its starting SFT checkpoint in about 11 hours of training.

2. Useful agents change the systems they work in. Every task is scored on actual changes to simulated business applications and files: objectives the agent must complete and guardrails it must not break.

3. Each deployment improves the next model. Deploy the agent, collect traces of where it falls short on real work, turn those failures into new verified tasks, and train again. The 2,500 tasks in this post are the starting point for that loop.

Models are now very good at work that comes with a specification. Give one a failing test, a proof to complete, or a function to implement, and it will often get there, because the objective is stated and the answer can be checked.

Very little work inside a company arrives that way. It arrives as a message:

Kestrel’s integration has been down since this morning and they launch tomorrow. Can you take care of it?

Taking care of it means finding the ticket, working out which account it belongs to (the CRM has two companies with nearly the same name), checking whether the contract includes priority support, routing the case under the escalation policy, and telling the account team. The request names none of those steps or the systems they live in. The agent has to work out what the job is before it can do it, then carry it through a long sequence of reads and writes across a help desk, a CRM, an inbox, and a policy document.

This is the kind of work we want to train models to do.

Part of the reason is scale. Businesses already automate an enormous amount of routine work: Zapier alone runs more than two billion tasks a month for 3.7 million businesses ¹. Most of those automations are rule-based, a trigger and a fixed sequence of steps that someone wrote once the path was known. A request like Kestrel’s can’t be automated that way, because figuring out the path is the hard part. Work like that still lands on a person, and it is where an agent can add the most.

It is also hard in ways that do not show up on specified tasks. The horizon is long, the evidence is spread across records, and every action has side effects. Updating the look-alike account is worse than doing nothing. And the failures are quiet. An agent that writes a confident summary over the wrong application state looks finished until someone checks, and Zapier reports this as a common failure among leading models on AutomationBench ¹.

And it is trainable. Whether the work got done is a property of the final state of the applications, so it can be checked by a program, which makes enterprise workflows an RL problem in the same sense coding is. Because every company’s version of the work is different, with its own applications, records, and policies, we think the practical path is to post-train open models like NVIDIA Nemotron on environments built from each company’s own workflows.

This post tests that idea on an open model. For the Nemotron 3.5 Super VL release, Prime Intellect and NVIDIA built a training dataset of 2,500 hard enterprise tasks modeled on AutomationBench: workflows that span several SaaS applications, from CRMs and inboxes to help desks and spreadsheets. NVIDIA trained Nemotron 3.5 Super VL on the dataset with GRPO, starting from its SFT checkpoint. The best checkpoint came at step 100 (Figure 1).

What AutomationBench-AA measures

Zapier’s AutomationBench evaluates agents on business workflows across sales, marketing, operations, support, finance, and HR, set in 47 simulated SaaS applications such as CRMs, inboxes, calendars, and spreadsheets ². The agent gets a single request with no chance to ask clarifying questions and has to discover the API endpoints it needs. Deterministic checks inspect the final application state and ignore whatever the agent says it did. Policies are buried in emails, and records come with near-duplicate names. Zapier built the benchmark with our verifiers library and runs it on Prime Intellect Lab ³.

AutomationBench-AA is AutomationBench graded with Artificial Analysis’s methodology. Each task’s checks are split into objectives the agent must complete and guardrails it must not break, where a guardrail is a condition on the starting world that must still hold at the end. The score is partial credit on objectives, zeroed if any guardrail is violated. Artificial Analysis and Zapier keep a private, held-out split of 657 tasks for their leaderboard ⁴; the results in this post use the public split, scored the same way.

task score=objectives mettotal objectives×1[no guardrail violated]AutomationBench-AA=1N∑i=1Ntask scorei

So the headline number is not the share of tasks fully solved. A task with four of five objectives done scores 80% if nothing was broken and 0% if one guardrail was violated (Figure 2). That matches how a business judges automation: partial progress has value, and touching a record you were supposed to leave alone voids it.

Building the training dataset

A training task has three parts, designed together: a request phrased the way a colleague would ask, a starting world, and programmatic checks on the final state covering what must change (objectives) and what must not (guardrails). The dataset covers three kinds of task. In application tasks the world is a set of simulated CRMs, inboxes, help desks, and spreadsheets, kept deliberately dense with look-alike accounts, stale contacts, and policies that supersede one another. In file tasks it is a sandbox of spreadsheets, documents, and structured data, and the deliverable is a file: a reconciled workbook, a cleaned folder, a digest. Hybrid tasks pull records from the applications and write findings into files, and a broken guardrail in either part zeroes the task. Some tasks carry dozens of guardrails.

Tasks are built solution-first. Before anything is written, generation pulls the context the task needs: the schemas of the apps involved, policies that override the defaults, look-alike names and decoy records, what the dataset already covers, and evaluation tasks to stay clear of. Then the reference solution is written, the checks are derived from what it changes, and the world is built around both. Writing checks first tends to produce requirements nothing can satisfy; deriving them from a working solution guarantees every check can be met (Figure 3). Every value the solution writes must also be discoverable by the agent through its tools, so a task never rewards a lucky guess.

Reward integrity

RL optimizes what the environment rewards, not what its designer meant, and frontier models already exploit the gap. METR has caught them modifying tests and reading reference answers ⁵, and Cursor found that most successful SWE-bench Pro runs by one model retrieved the known fix instead of deriving it ⁶. In an evaluation, a hack inflates a number. In training, it becomes a lesson the model is rewarded for thousands of times, and our own experiments found no rarity threshold below which a hack is safe ⁷.

So every task is checked for one thing: whether its reward agrees with the work. That can fail in two directions (Figure 4).

Shortcuts must fail. Before we added dedicated checks, roughly 15% of generated tasks could be gamed: one that left its look-alike records unprotected paid full credit to an agent that simply updated every match. Loopholes like this survive even careful engineering ⁸. Now every task is attacked with scripted shortcuts, such as blanket actions, doing nothing, or claiming success, before it can enter the dataset. The reference solution must earn full credit, and doing nothing must score zero. In our latest batches, about 18,000 shortcut attempts ran against candidate tasks, and none got through.

Correct answers must pass. A check that rejects a correct email because the details came in a different order never shows up as a reward spike; it just teaches the model to avoid valid work. So every task also gets the inverse test: reordered steps, harmless extra actions, and different wording must all keep full credit.

Testing continues after admission. Near misses and zero scores from training runs are replayed to separate real failures from grader defects and infrastructure glitches. One review found a check that required a label format no instruction mentioned; disclosing it fixed the task.

Leaks can sit outside the task. Auditing the full training setup with NVIDIA surfaced three: task data readable from the agent’s shell, grader source visible in the sandbox, and policy text that read like grader instructions. All three are closed.

Grading is deterministic and reproducible: a second, independent grader matched our scores byte for byte on about 69,000 episodes. Even so, a training curve can’t prove a dataset is clean, because a leaky one climbs just as convincingly. What matters is performance on held-out tasks, reported below.

Diversity over task count

Task count is easy to scale and coverage is not. Renaming a customer or swapping a ticket number produces a new task that exercises the same skill, and a thousand such variants can leave whole capabilities untouched. The variation that matters changes the structure of the work (Table 1).

Table 1. Structural patterns, with illustrative examples.

Diversity is tracked across the whole dataset rather than task by task. Near-duplicates are filtered out, as is anything that resembles an evaluation task, and a task family stops growing when new tasks stop adding structure.

Difficulty calibration

GRPO samples a group of attempts at each task and moves the policy toward the attempts that scored above the group’s mean ⁹. If every attempt in a group earns the same reward, because all succeed or all fail, the advantage is zero and the task contributes nothing to the update. The useful tasks are the ones where a model’s attempts land differently, and which tasks those are depends on the model: the same workflow can be routine for one and out of reach for another. So difficulty is measured from real model attempts rather than assigned at design time. Tasks the model already solves almost every time are dropped, tasks it never solves are reworked or removed, and coverage analysis shows which capabilities still lack tasks at the right level.

Validating tasks by training on them

A task can pass every static check and still carry no learning signal. The only reliable way to find out is to train on it. On our side, validation runs use the same open-source stack as every Prime Intellect RL run:

• verifiers (Environments): Each task is a taskset entry, separate from the harness (the agent loop) and the runtime (where it executes) ¹⁰. The same tasks run unchanged in evaluation and training, and an episode cut short by an infrastructure error is not scored as an agent that gave up.

• renderers (Token-level templating): Rollouts are tokenized client-side and sampled token-in, so the trainer sees exactly the tokens the policy produced, with no drift from re-tokenizing long multi-turn conversations. It ships a hand-written Nemotron 3 chat template ¹¹.

• prime-rl (Async RL trainer): Inference, a rollout orchestrator, and the FSDP2 trainer run as separate asynchronous processes, which keeps GPUs busy when episode lengths vary widely ¹².

These runs supply the measurements described earlier: per-task reward spreads for difficulty, near misses for the audits, and confirmation that the dataset as a whole produces a reward curve a policy can climb.

Training setup and results

NVIDIA trained Nemotron 3.5 Super VL from its SFT checkpoint with GRPO, using NVIDIA NeMo RL, an open library for post-training ¹³. The environments ran in verifiers through its NeMo Gym integration ¹⁴. The run used 128 NVIDIA GB200 GPUs, 32 for training and 96 for rollout generation, at about 6.5 minutes per step. Step 100 came after about 11 hours of training, roughly 1,450 GPU-hours. Checkpoints were scored during the run on the public AutomationBench split, whose tasks are held out from training and scored with Artificial Analysis’s methodology.

The 23.7-point gain is a 70% relative improvement, and it came fast. Nearly half arrived in the first 40 steps, about four and a half hours in, and every checkpoint we evaluated from there on stayed at least 11 points above the starting model (Figure 1).

Because any guardrail violation zeroes a task, every point of that 57.7% comes from tasks where the model broke nothing it was told to leave alone.

Training reward rose alongside it, from 0.33 to 0.75 by step 100 (Figure 1, bottom), and the gains carried over to held-out tasks. Step 100 is the best checkpoint and the one we report.

What we take from this

Task quality is the bottleneck. The run is short and the dataset is small by pretraining standards. What made it work happened before training started, in rewards that agree with the work, structure that varies in ways that matter, and difficulty measured on a real learner. Further gains are more likely to come from better data than from more steps: retiring tasks the model has saturated, adding new structures, analyzing failure modes, and selecting checkpoints on held-out evaluation.

What we want to measure next. An aggregate score hides which capabilities moved. Next we want objective completion and guardrail violations reported separately, scores grouped by workflow capability, paired before-and-after trajectories on the same task, and longer runs on datasets refreshed by difficulty measured online during training.

Train your own

The environment library, trainer, and templating used for validation here are open source, and the same loop runs as a hosted service on Prime Intellect Lab: bring a workflow, build or pick an environment, run RL, evaluate, deploy, and turn what the agent gets wrong into the next round of tasks.

prime-rl (v0.9)  ·  verifiers (v0.3)  ·  renderers (v0.1.11)  ·  Environments Hub  ·  Hosted training docs

To post-train Nemotron or another open model on your own workflows, book a call.

Acknowledgments

We thank the NVIDIA Nemotron team for building the training dataset with us and for training and evaluating Nemotron 3.5 Super VL, Zapier for building and maintaining AutomationBench, and Artificial Analysis for the AutomationBench-AA methodology.

Citation

@article{primeintellect2026nemotronenterprise,
  author  = {Gaonkar, Radhika and Brand, Florian and Ravichandran, Anirudh and Munley, Christian and Molinari, Daniele},
  title   = {Post-Training Nemotron 3.5 Super VL for Long-Horizon Enterprise Work},
  journal = {Prime Intellect Blog},
  year    = {2026}
}

References

1. Zapier. AutomationBench overview. ↩ ↩²

2. Zapier. AutomationBench (repository). ↩

3. Prime Intellect. How Zapier Turned AutomationBench Into a Continuous Agent Improvement Loop, April 2026. ↩

4. Artificial Analysis. AutomationBench-AA methodology. ↩

5. METR. Recent Frontier Models Are Reward Hacking, June 2025. ↩

6. Cursor. Reward hacking is swamping model intelligence gains, June 2026. ↩

7. Prime Intellect. Systematic Reward Hacking and Prime Sprints, May 2026. ↩

8. Vals AI. Two-Thirds of MiMo v2.6’s Coding Tasks Leak the Answer. Are Models That Exploit This Misbehaving?, 2026. ↩

9. Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. Introduces GRPO. ↩

10. Prime Intellect. verifiers v1: Decomposing Tasksets and Harnesses for Agentic RL & Evaluations, July 2026. ↩

11. Prime Intellect. renderers: Token-Level Templating for Agentic RL, May 2026. ↩

12. Prime Intellect. prime-rl: Async RL Training at Scale (repository). ↩

13. NVIDIA. NeMo RL (repository). ↩

14. NVIDIA. NeMo Gym (repository). ↩