Pokemon champion in 18 hours instead of 96, a rocket launch in Factorio, the credits rolling in Fallout 3. GPT-6 Astra beats games that stumped every model before it. But then it spent hours farming potatoes in Minecraft after a Creeper blew up its stash.
Six blaze rods, three ender pearls. According to Vals AI, no AI system had ever gotten this far in Minecraft as GPT-6 Astra. In the Nether it had built a semi-automatic blaze farm, killed more than half a dozen Endermen in a warped forest, and gathered the ingredients for eyes of ender, which let you find the portal to the game's final battle. Thousands watched the stream as Astra put its loot in a chest.
Then came the Creeper. The explosion wiped out the chest and the bed. Everything was gone, and by the time Astra noticed, it had started to rain too.
Astra jotted down a note, partly in caps: "ALWAYS CARRY CRITICAL ITEMS with keepInventory; don't store in unguarded chest ever again." Then it spent several hours farming almost nothing but potatoes.
Faster than any model before it
Before that, GPT-6 Astra had spent well over 100 hours in the game. Vals AI finally called off the run after 141 hours. Compare that to the early days of the first Minecraft AI agents, and the size of the jump becomes clear. In 2022, OpenAI's VPT reached a diamond pickaxe in just 2.5 percent of runs, nowhere near a portal. In one Vals AI comparison, VPT needed 10 minutes to get there.
The cleanest comparison to more recent models comes from a different project, "GPT Plays Pokemon". According to operator Clad3815, Astra earned the champion title in Pokemon FireRed after 18 hours and 12 minutes. GPT-5.6 Sol needed 96 hours and 35 minutes. GPT-5.5 still hadn't finished the game after more than 218 hours.
A community run of Factorio: Space Age tells a similar story. Controlled through a custom Lua mod with an MCP interface, Astra produced blue science packs after about two hours and launched its first rocket after roughly ten. In the same setup, GPT-5.6 Luna and Fable 5.1 never got past power supply and oil prospecting.
Then there are completions with no direct comparison numbers. Portal all the way to the credits, Fallout 2 in 22 hours, a colony in RimWorld that Astra carried through several raids and eventually off the planet, plus the main story of Fallout 3 after about 59 hours.
Even before the community runs, ARC-AGI-3 had already signaled that this was a jump, not just another step. The benchmark drops models into unfamiliar, abstract game environments where they have to figure out the rules through their own actions. Astra hit 62.7 percent over the standard interface, and about 99.9 percent with OpenAI's own harness. GPT-5.6 Sol scored 7.78 percent, and Claude Opus 5 a little over 30 percent.
Check first, then repeat
ARC Prize, the organization behind the benchmark, explained how that leap happened: Astra translates unfamiliar game mechanics into compact symbolic descriptions and tracks objects, coordinates, rules, and planned actions in a shorthand it develops itself. Observations become rules, and rules become plans.
The idea of turning experience into reusable rules isn't new. Back in 2023, the research project Voyager, which involved Nvidia and Caltech, had GPT-4 propose tasks in Minecraft, write JavaScript code for them, and revise that code based on error messages. Working programs went into a skill library.
But Voyager never saw the game. It got structured data and controlled the game through the Mineflayer programming interface. Vals AI's Minecraft run, by contrast, runs through general computer use, meaning screen, mouse, and keyboard, with no specialized game hookup, according to the company. What researchers once built around the model, Astra now does on its own. In under three hours it had built a Nether portal, fought zombies, trapped a skeleton, and found its way back to base.
The Fallout 3 run shows a similar structure. The user imjustnewatai, who posted the full run on YouTube, played the opening in Vault 101 himself and then handed the save file to OpenAI's agent tool Codex, with Astra set to maximum reasoning. From there the agent picked its own route and actions. Galaxy News Radio, the search for the father, the G.E.C.K., the escape from Raven Rock, Project Purity. It played through normal game inputs in cycles of pausing, observing, and acting. Saving, companions, and fast travel were allowed. After about 59 hours, the credits rolled.
The pattern ARC Prize described shows up most clearly in the hardest stretch of the run. In Vault 87 the character died several times, ammo ran out, and Astra had to backtrack to heal. The status messages from this phase all follow the same shape. An action has no effect, and instead of repeating it, Astra looks for the cause and records the result.
A bed that was supposed to heal the character didn't respond at first. Astra let the game run for a moment, tried again, and reported: "The bed worked after letting the game run briefly. Health is now fully restored to 260/260." Shots at an enemy kept missing. Astra figured there was an obstacle in the line of fire, moved closer, and noted: "Moving closer cleared the obstruction that was spoiling the earlier shots." At a keypad, it waited until the game confirmed the button press before moving on.
Looking back, Astra boiled these episodes down to a single principle. Check whether an input worked before repeating it. That sounds trivial, but it's the skill that keeps the agent from looping the same useless action over and over. It can also be the starting point for behavior that later becomes a problem.
Never an unguarded chest again
The same run also shows where this tips over. After the deaths in Vault 87, status messages piled up in which Astra saved its progress, checked the recording, and vetted every step against the visible environment before moving on: "I'm checking each short move against the visible geometry so I can leave without losing the saved progress." That slowed the run down a lot, but it didn't grind it to a halt.
In Minecraft it did. Right after the Creeper explosion, Astra wrote itself a rule. Always carry important items on your person, never store them in an unguarded chest again. In its notes over the following hours, it distrusted every tall green object ("GREEN tall thing ahead was SUGARCANE, NOT creeper!") and was hard on itself. You can make mistakes and lose items in the process, it noted. And it didn't want to waste another night chasing "dark pink pixels," meaning pigs, an easy food source in Minecraft that Astra had apparently hunted at night without success. The only safe thing was the potato field, and according to Vals AI, that's where it stayed for several hours.
So a bad experience becomes a permanent rule, and the rule sticks even when it no longer helps. In ARC-AGI-3, this tendency helps the model work out the game mechanics from just a few tries. In an open world with random events, it can clearly lead the model to overcorrect and lose sight of its actual goal. Still, the jump over older models is stark. And it shows that the team behind the ARC-AGI benchmarks has once again built a test that served as the canary in the coal mine of AI progress, much like the early wins on ARC-AGI-1 marked the start of the reasoning models.