Anthropic's new research shows that Claude can solve real bioinformatics problems human experts miss.

23 “human-difficult” problems that their expert panel could not solve, and their top model, Claude Mythos Preview, solved 29.6% of that set.

The problem is that older science tests mostly check clean questions, not messy biology data work on real datasets.

BioMysteryBench tries to fix that by hiding objective answers inside real datasets and grading only the final answer.

It gives Claude standard biology tools and database access on 99 tasks, while up to 5 experts try them too.

On the 76 problems at least 1 expert solved, the best model got about 83%, and on 23 expert-stumping problems it got about 30%.

The post also found that wins on the hard problems were much less repeatable across 5 tries, so many successes were shaky rather than dependable.

Anthropic’s own examples suggest Claude is strongest when it behaves less like an oracle and more like an unusually fast research collaborator: it layers methods, cross-checks evidence, and uses broad background knowledge to narrow the search space.