OpenAI 首席研究官 Mark Chen 呼吁前沿实验室共享关键指标并公开自家困惑度数据
OpenAI 首席研究官 Mark Chen 表示各实验室自写安全案例、自设护栏导致 AI 竞赛动态难以摆脱,呼吁前沿实验室披露追踪进展的关键指标,并提出困惑度作为候选标准。
AI race dynamics are hard to escape when each lab writes its own safety cases and sets its own guardrails for developing more capable models.
We want frontier labs to operate under *common* limits, backed by *shared* safety cases. But the problem is we don't have a good, future-proof yardstick for AI progress that stays useful as models improve.
So we're calling on frontier AI labs to disclose the key metrics they use to track progress, and we propose perplexity as one candidate. We're taking a first step by publishing our own perplexity results over the past five years.
This is scary for us because we've long treated these numbers as sensitive IP. And we know there's still work (perhaps by 3P evaluators) to measure perplexities over the same datasets. But we hope this encourages a norm of sharing key metrics that we can build common safety cases on.
在 X 查看被引用的帖子We decided to provide more transparency into pretraining progress at OpenAI for the sake of moving toward shared safety bars. In “Perplexity and Intelligence” I argue for common limits based on perplexity and other measurements, with progress beyond them gated on our ability to understand AI behavior.
来源:Mark Chen · x.com