斯坦福 HAI 发布 MedHELM、PhysicianBench 与 HealthAdminBench，测出医疗 AI Agent 实际完成率不足一半

Measuring What Medical AI Can Actually Do

Date

October 19, 2026

Topics

Over three years, we’ve built progressively harder, more realistic benchmarks for medical AI. Here’s what we learned.

Large language models now score near-perfect on medical licensing exams, fueling enthusiasm about AI’s potential in healthcare. But acing a multiple-choice test and working in a health system are very different things. A licensing exam asks for the right answer. A health system asks you to open a chart, find the echocardiogram from two years ago, reconcile it with this morning’s labs, decide what to do, place the orders, and write the note. One tests knowledge. The other tests whether you can do many tasks across time, documents, and software systems. If we want to know whether AI is clinically useful, we have to evaluate it on the tasks actually handled.

That distinction is what drove our three years of work building progressively harder, more realistic benchmarks for medical AI. We started with MedHELM, a systematic effort to map the full landscape of clinical tasks and measure how today’s best models perform across them. Then we went deeper with PhysicianBench, which tests whether AI agents can complete real clinical tasks end-to-end inside a functioning electronic health record environment, and HealthAdminBench, which explores administration and workflow.

These benchmarks tell the same story: Today’s best AI agents complete less than half the work. Agents lose track of information as they move across steps, documents, and systems.

In this post, we explain how we built these tests, what we found, and what it will take to close the gap between what AI says it can do and what it actually does.

MedHELM: Mapping the Whole Job

In 2023, in MedAlign, 15 clinicians wrote 983 questions of the kind they would pose to a patient’s record. The best model at the time, GPT-4, produced errors in 35% of its responses, which led us to build MedHELM in early 2024.

MedHELM was our attempt to systematically conduct a task-based evaluation. We started with the jobs or tasks to be done, and then designed the tests around them. With input from 29 clinicians, we built a taxonomy with five categories, 22 subcategories, and 121 tasks reflecting commonly done work in delivering medical care. The categories included clinical decision support, clinical note generation, patient communication and education, medical research assistance, and administration and workflow (Figure 1).

Figure 1: The MedHELM taxonomy includes 5 main categories, 22 subcategories and 121 tasks. Figure from Bedi, S., Cui, H., Fuentes, M. et al. Nat Med 32, 943–951 (2026).

We assembled 37 benchmark datasets to cover every subcategory and evaluated the performance of nine frontier models against those benchmarks. We used an LLM jury for scoring against clinician-defined criteria, ensuring that the jury’s agreement with clinicians matched the level of clinicians’ agreement with each other. We examine the “win rate,” the proportion of pairwise comparisons in which each model achieved superior performance across all benchmarks (possible range: 0–1). The results show the highest and lowest win rates of LLM models across the top five categories:

Clinical note generation: 0.74 to 0.85
Patient communication and education: 0.76 to 0.89
Medical research assistance: 0.65 to 0.75
Clinical decision support: 0.63 to 0.77
Administration and workflow: 0.53 to 0.63

Models consistently struggled on clinical decision support and administration and workflow tasks, exactly where health systems actually want the most help to address clinician time and administrative cost. Administrative spending in the U.S. alone exceeds $1 trillion a year. The categories with the largest potential return scored the lowest.

Tasks Are Workflows

MedHELM benchmarks grade single responses: A prompt goes in, text comes out, and the text is scored. Real tasks in the two weak categories rarely look like that.

A prior authorization is a good example. Someone reads the order in the EHR (electronic health record), checks the payer’s criteria, pulls supporting documentation, fills out a portal form, attaches files, submits, and records the reference number in the chart. A specialist consult has the same shape: retrieve data across encounters, reason about it, place orders, and document. Grading the text a model would produce at step 3 tells you nothing about whether step 7 happens.

We need a different kind of evaluation for these important categories. Give an agent an environment, a goal, and tools. Let it act. Then check what changed in the environment. That is the design behind PhysicianBench, which explores clinical decision support and execution, and HealthAdminBench, which explores administration and workflow.

The machine learning risk prediction chasm represents the wide gap between a model that makes a good prediction and one that changes what happens to a patient. These benchmarks highlight the generative AI chasm representing the wide gap between what a model says and what a model can actually do.

PhysicianBench and MedAgentBench: Can AI Agents Do the Clinical Work?

MedAgentBench established 300 patient-specific clinically derived tasks across 10 categories, written by human physicians. The tasks are based on realistic profiles of 100 patients in a virtual electronic health record compute environment. When this was released in early 2025, even the best frontier models achieved <70% proper task completion. By 2026, agentic advances are now seeing frontier models with over 90% task completion.

PhysicianBench pushes the frontiers further with 100 long-horizon, multi-step tasks adapted from real medical consultations between primary care physicians and subspecialists, across 21 specialties. A separate panel of physicians independently reviewed each task. The tasks are attempted against synthetically reconstructed patient records in an EHR sandbox. An average multi-step task takes 27 tool calls.

Collectively, evaluating the task results in 670 checks in total, covering data retrieval, clinical reasoning, action execution, and documentation. Grading is execution-grounded: Task-specific scripts check whether the right orders exist in the record, with LLM judgments for the reasoning and the note.

Across 12 models, the best (GPT-5.5) completed 46% of tasks on a single attempt. The best open-source model, DeepSeek V4-Pro, reached 19%. Consistency is worse: GPT-5.5 properly executed only 28% of tasks in all three of its runs.

Failure analysis gives us even more insight, as there are still limitations in clinical reasoning that account for about half of the failed checkpoints. The errors are mostly incomplete reasoning and near misses, while outright wrong conclusions are rare. In addition, there is a failure that no past benchmark could catch. Among action-execution failures, 62% for GPT-5.5 and 73% for Claude Opus 4.6 were cases where the agent wrote the correct action in its note and never placed the order in the EHR. The AI generated beautifully written words into a good plan, but never actually got the work done.

HealthAdminBench: Can AI Agents Do the Non-Clinical Work?

HealthAdminBench targets the bulk of healthcare system effort and staffing: the billing, authorization, and administrative work that happens behind the scenes of every patient encounter. The number of behind-the-scenes administrative staff is rapidly growing and may already outnumber frontline clinicians 10:1. We spent more than 100 hours observing administrators at Stanford Health Care to build four simulated web environments, an EHR, two payer portals, and a fax system. AI agents are able to operate them the way a person would, through screenshots, clicks, and typing.

There are 135 tasks across three workflows (60 prior authorizations, 60 appeals and denials, 15 durable medical equipment orders) at three difficulty levels, designed with practicing administrators. Each task decomposes into subtasks, and 1,698 verifiers grade them: 1,177 deterministic checks and 521 LLM-judge rubrics. All patient data is synthetic, allowing us to release the environments and run a leaderboard for public use.

The best agent tested at the time, Claude Opus 4.6 in its native computer-use mode, completed 36.3% of tasks end-to-end. The best open-weight model, Kimi K2.5, completed 15.6%.

A key distinction is that GPT-5.4 in computer-use mode passed 82.8% of individual subtasks but completed only 26.7% of whole tasks. Competence on the parts does not add up to reliability in a whole workflow. Filling out a form, then sending it to the wrong address doesn’t count. Patients aren’t going to grant partial credit for completing the documentation while prescribing the wrong medicine.

Here, the failures are coordination failures. Agents gather information in the EHR that they will need much later in a payer portal, fail to write it down, and lose it. They miss file uploads and downloads even when the task requires them. They rarely go back to fix an earlier step, and run out of budget instead.

Reading the Benches Together

These benchmarks make complementary bets. PhysicianBench prioritizes fidelity around clinician-facing processes based on real clinical records. HealthAdminBench prioritizes replication of human workflows for extensive administrative processes in healthcare systems.

A health system governing AI adoption needs both kinds of applications, with evidence for reproducibility and transparency marked by release of (synthetic) data, public code, and leaderboards.

The headline performance of 46% task completion in PhysicianBench is not directly comparable with 36% task completion in HealthAdminBench. The environments, model sets, interfaces, and grading schemes differ. The defensible claim is directional: Two independently built, execution-grounded benchmarks, one clinical and one administrative, both find today’s best agents finishing less than half of the work. In both, the dominant failures come from failing to carry information across encounters, documents, and systems, which points to agent architecture and memory (i.e., better harnesses) as the fix, at least as much as bigger models.

Both benchmarks trace back to the categories in which MedHELM found the weakest performance. MedHELM (www.medhelm.org) is community-owned and released under an Apache 2.0 license, and the point of an open framework is that others can contribute category-specific, workflow-level benchmarks. In fact, we are contributing both HealthAdminBench and PhysicianBench. If your health system has a workflow that matters and a benchmark to test it, we invite you to contribute it to MedHELM.

Link copied to clipboard!

Contributor(s)

Nigam Shah, Jonathan Chen

Nigam Shah

Professor of Medicine (Biomedical Informatics) and of Biomedical Data Science, Stanford University; Chief Data Scientist, Stanford Health Care; Faculty Affiliate, Stanford HAI

Holistic Evaluation of Large Language Models for Medical Applications

Nigam Shah, Mike Pfeffer, Percy Liang Feb 28 news Medical and AI experts build a benchmark for evaluation of LLMs grounded in real-world healthcare needs.