Its getting hard to benchmark frontier agent performance on longer tasks. Repeated measurement is very expensive and there are differences between using models in harnesses versus via APIs.

I suspect benchmarks understate progress, they are built for models, not harnessed agents