Dan Hendrycks 估算 GPT-4 以来预训练计算效率每年提升约 3–5 倍
Dan Hendrycks 引用 OpenAI 发布的《Perplexity and Intelligence》一文及其公开的基座模型困惑度曲线,估算自 GPT-4 以来预训练计算效率平均每年提升约 3–5 倍,即达到相同 loss 所需算力每年缩减至三分之一到五分之一。他指出大部分增益来自最近一年,这可能意味着今年的提升达到约 100 倍以上。
Averaged since GPT-4, this suggests a ~3–5×/yr gain in pretraining compute efficiency, meaning it takes a third to a fifth the compute each year to hit the same loss.
But most of this came in the last year, which could suggest a ~100×+ gain this year.
在 X 查看被引用的帖子We decided to provide more transparency into pretraining progress at OpenAI for the sake of moving toward shared safety bars. In “Perplexity and Intelligence” I argue for common limits based on perplexity and other measurements, with progress beyond them gated on our ability to understand AI behavior.
来源:Dan Hendrycks · x.com