TypeSafe AI· @typesafeai · X·· 2 小时前AI 评分35
AI 导读
谁来评测那些评测者？ 一个好的评测是： - 短暂的。它只截取某一时刻的状态与能力。 - 有依据的。它直接衡量你关心的任务和数据上的表现。 - 具体的。它针对的是期望结果本身，而不是其某种代理指标。 记好了！
正文 · 原文
who benchmarks the benchmarkers?
a good eval is:
- ephemeral. it snapshots a moment in time and capabilities.
- grounded. it directly measures performance on tasks and data you care about.
- specific. it operates on the desired results and not some proxy of them.
take notes!
Typesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑 http://parallel.ai/blog/are-all-system-one-models-created-equal在 X 查看被引用的帖子
来源：TypeSafe AI · x.com