Rohan Paul· @rohanpaul_ai · X·· 2 小时前AI 评分47
AI 导读
Sam Altman 谈本地推理。 “从全球资源分配的角度看，让每个人、让世界上有限 DRAM 供应中的一大批都放在自己的电脑里、却只有 5% 的时间在用，这相当糟糕” 按每 token 算，数据中心轻松胜出。batch size 为 1 的本地模型按任何利用率指标看都是浪费，但它也是整个技术栈中唯一没人能对你弃用的部分。https://x.com/morqon/status/2115838856524120128/video/1
正文 · 原文
Sam Altman on local inference.
"from a global allocation perspective, it's quite terrible to have everybody, a bunch of the world's limited supply of DRAM sitting in their own computer and being used 5% of the time"
per token, the datacenter wins easily. A local model at batch size 1 is wasteful by any utilization metric, and it’s also the only part of the stack nobody can deprecate on you. https://x.com/morqon/status/2115838856524120128/video/1
来源：Rohan Paul · x.com