The LongCat team just released LARYBench, a benchmark built to test whether an AI model truly learns action from video, instead of only looking good when attached to a robot policy later.

It evaluates latent actions, meaning the hidden motion signals a model extracts from video, across 1.2M+ clips, 620K+ image pairs, 595K trajectories, 151 action classes, and 11 robot platforms.

A latent action representation tries to store the change between frames as something like reach, pick, place, move left, or close gripper, rather than memorizing raw pixels.

The key point is that robot training data is scarce, while human and robot videos are abundant, so the whole field wants a way to turn cheap video into useful action knowledge.

The paper argues that older evaluations mixed too many things together, because a robot succeeding on a task depends on the policy, training recipe, environment, and controller, so you could not tell whether the action representation itself was actually good.

LARYBench splits the problem into 2 cleaner tests, where one asks whether the representation knows what happened and the other asks whether it preserves enough detail for how to move.

The biggest result is that general self-supervised vision models beat specialized embodied LAMs, with V-JEPA 2 reaching 76.62% average action classification accuracy, while DINOv3 gives the best overall control regression score at 0.19 MSE, far ahead of embodied models clustered around 0.87 to 0.97.

The deeper point is that strong visual representations already contain a surprising amount of action knowledge, and the paper also shows that latent feature spaces map to robot control better than pixel reconstruction spaces, which helps explain why some robotics systems may be building on the wrong intermediate representation.

🧵 1.