Epoch Brief：AI 能否自动化 Epoch 自身工作，六款模型实测未过关

Welcome to the Epoch Brief. Here’s what we’ve been working on:

• Can AI automate our own jobs? We gave six AI models 11 realistic tasks drawn from work at Epoch, from designing graphics to drafting short research articles, to see whether they could automate Epoch. The verdict: not yet.

• AI is getting very good at a difficult video game. In six months, the best models went from struggling to beat the strategy video game Slay the Spire to winning over 90% of runs at base difficulty.

• Open-weight models trail on cybersecurity. Today’s best open-weight model scores about the same as the top closed-weight models from 6-7 months ago.

• Gemini is the closest it’s been to the frontier since February. Gemini 4 Argon scores 166 on the Epoch Capabilities Index, tying Anthropic’s Claude Fable 5.1. Claude Opus 5.5 and OpenAI’s GPT-6 Astra share the top spot.

• Four new benchmark reviews. We graded one as Verified, and three as Flawed. Four others lacked sufficient information for a verdict.

• New open roles. We’re still growing, with openings across research, design, product, and media.

Subscribe now

Can AI automate work at Epoch?

Many of us want to know if and when AI will automate our jobs. To explore this question, researcher Kelly Hong and head of benchmarks Greg Burnham tested how well AI models can do the work we do at Epoch.

In Can AI automate Epoch?, they gave six leading models 11 realistic tasks drawn from work at Epoch. The tasks spanned five areas: graphic design, generating short research reports, building interactive data tools, AI data center research, and research design. Each model received about the same context a new hire would, with its own workspace and Google account. They got one attempt per task with no human help.

Claude Fable 5.1 and GPT-6 Astra came out roughly tied for the lead, but no model could reliably meet the quality standards we set for Epoch work. Frontier closed-weight models performed best on well-defined tasks like coding and data analysis. Models fell short on open-ended tasks that required broader judgment, like designing informative experiments and interpreting results. Open-weight models struggled even on well-defined tasks.

In the words of the researchers: current models “cannot yet replace workers, at least not at Epoch.” While models aren’t quite ready to replace knowledge workers, AI capabilities continue to trend upward in related areas.

How good is AI getting?

AI is rapidly improving at the video game Slay the Spire

One way to test AI’s ability to handle long, complex tasks is through video games.

Our new benchmark, SpireClimb, tests this on Slay the Spire, a deck-building strategy video game. To succeed, an AI has to work through a huge decision tree, click the right things on the screen, and play for long stretches without help. Because SpireClimb requires computer interactions, a model that does well may be better positioned to take on computer-based work.

So how well do AI models do? In AI systems are improving rapidly at Slay the Spire and its sequel, researcher Benjamin Ou and Simon Chen found that in just six months, frontier models went from struggling to beat the game at all to winning over 90% of runs at base difficulty, with GPT-6 Astra winning 98% of runs. At the game’s hardest difficulty, GPT-6.1 Sol wins 20% of runs, and Astra wins 17%, while the other models tested win almost none of their runs.

Because Slay the Spire came out nearly a decade ago, the models know a lot about the game, and this could affect their scores. To check this, the researchers also had models play the game’s sequel, Slay the Spire 2, which launched this year. At the sequel’s hardest difficulty, Astra wins 35% of runs. That’s higher than its rate in the original, but the sequel’s hardest setting is easier. Still, models know less about the newer game, and the top models did reasonably well, suggesting their scores on the original didn’t come solely from prior knowledge.

Top human players still do better, winning an estimated 85% of runs at the original’s hardest difficulty and about 95% in the sequel.

Open-weight models trail the closed-weight cyber frontier by 6–7 months

In a recent Data Insight, Jaeho Lee and Alexander Barry compare open-weight and closed-weight models using the Cyber ECI, a cybersecurity-focused version of the Epoch Capabilities Index.

The top open-weight model is Z.ai’s GLM-5.3 with a score of 160. Its score is above GPT-5.4 (157, March), and sits right below GPT-5.5 (162, April) and Claude Mythos Preview (165, April).

This means the top open-weight model sits roughly where the frontier closed models were about six to seven months ago.

Meanwhile, the closed frontier continues to move ahead on cyber, with Claude Opus 5.5 and GPT-6 Astra both scoring 171. Some model scores come from only a few tests, so treat them with caution.

Help our research reach more people.

Share

Gemini closes in on the frontier

Google DeepMind publicly released Gemini 4 Argon on October 13, following a limited release to partners on September 30. The model currently scores 166 on the Epoch Capabilities Index, tying Claude Fable 5.1 and just behind Claude Opus 5.5 and GPT-6 Astra (both 169) and GPT-6.1 Sol (168).

This is Google’s closest approach to the frontier since Gemini 3.1 Pro in February, and is a big jump from Gemini 3.8 Flash in early September (157).

Gemini 4 Argon performed best on benchmarks in knowledge, agentic tasks, and image understanding, and worse in math and software engineering, currently ranking 8th on our math index and 6th on our software engineering index.

Reviewing benchmarks

We recently launched Benchmark Reviews, an initiative led by senior researcher Michelle Campeau, where we audit external AI benchmarks and assess whether their scores accurately measure what they are supposed to. If a benchmark has one or more substantive flaws that we believe users need to be aware of to accurately interpret results, we label it “Flawed.” If a benchmark meets our standards, we label it “Verified.”

Our latest batch includes 4 new reviews: 1 Verified and 3 Flawed. 4 more benchmarks lacked sufficient information to assign a verdict. This continues our project to improve overall benchmarking quality by providing independent audits. Detailed reviews for all benchmarks are available on our website.

We don’t review our own benchmarks. Instead, we’ve contracted with Generality Labs to independently audit them. So far, they’ve audited Chess Puzzles and EBR-bench. You can find those audits and our responses on our site.

Careers

Epoch is growing quickly, and we need exceptional people to help us scale our impact. We’re hiring for:

• Researcher / Senior Researcher: improve society’s understanding of AI capabilities and their trajectories

• Designer / Senior Designer: turn complex AI research into dashboards and data visualizations, specializing in either product and web design or data visualization

• Design Lead / Head of Design: lead UI/UX and data visualization, and manage our design team

• Product Manager, Website: decide what gets built on the platform where audiences meet our research

• Senior Writer & Editor: a founding writer and editor for our new media team, pitching, writing, and editing pieces for general audiences

• Social Video Producer: launch and produce short-form AI video for TikTok and Instagram

• Special Projects Associate (expression of interest): provide specialized operations support for our research teams

Not seeing a fit? Submit a general Expression of Interest. Applications are rolling, so apply soon!

Thanks for reading! Subscribe to stay up to date on our research.

Subscribe now