新的未经删节的信息在这场版权诉讼中，《纽约时报》三年前对 OpenAI 和 Microsoft 提起的诉讼披露了一项承认，即 AI 抓取无异于盗窃，且 AI 产品对出版物构成重大威胁。
根据诉讼内容，微软一位高管私下将两家公司的 AI 训练做法描述为“盗窃”，而 OpenAI 自己的领导层也表示，其 AI 模型对那些作品被用于训练的出版商和记者构成了“生存威胁”。
解封的材料还详细说明了这些公司据称是如何在未被察觉的情况下绕过付费墙获取并使用这些内容、通过大规模抓取构建训练数据集，以及故意从训练数据中删除版权声明的。
值得注意的是，这些新信息大多来自《泰晤士报》自己的简报，而非仍处于封存状态的原始证据附件。以下引述均脱离了其原始语境。
这份未经涂删的法庭文件是这起已持续三年的诉讼的最新升级。在该案中，《纽约时报》最初指控这些公司通过用其内容训练生成式 AI 模型而违反了版权法。
AI 公司能否合法使用受版权保护的材料来训练 AI，这个问题尚无明确答案，但法官们在很大程度上倾向于支持 AI 公司的论点，即训练构成“合理使用”。这一法律规则允许人们在某些情况下未经许可使用受版权保护的作品，例如戏仿、新闻报道或批评。本月早些时候，特朗普政府提交了一份意见书，为 OpenAI 未经许可使用受版权保护材料训练其大语言模型进行辩护。
然而，新披露的若干承认内容与 OpenAI 的合理使用抗辩相矛盾，尤其是该规则要求使用不得替代或损害原作品的市场。
例如，微软自己的数据显示，其 Copilot“答案引擎”导致《纽约时报》域名的点击率相比传统 Bing 搜索下降了多达 93%。微软应用科学总监 Brent Hecht 于 2024 年 1 月撰写的一份微软内部演示文稿将这一下降描述为“末日循环”，称其将“同时损害我们模型的性能和整个网络”。
“一个终端产品威胁到其核心供应商的经济根基，这是极不寻常的，但这正是我们为自家大语言模型业务在其‘内容供应链’方面所创造的局面，”文件中引用的微软文档如此写道。
微软 CEO Satya Nadella 今年早些时候也在一次证词陈述中表示，“任何付费墙后的内容，都应获得任何想要使用它的人的授权……用于 grounding 或训练”，并明确表示，如果他“得知 OpenAI 抓取并训练了付费墙后的信息”，他会“行使[微软的权利]要求 OpenAI 重新训练其模型。”
其他承认则不利于合理使用测试的不同支柱：OpenAI 的 ChatGPT 负责人 Nick Turley 在内部沟通中写道，出版商面临来自聊天机器人等产品的“生存威胁”，这些产品“在很大程度上具有替代性”，并且“随着它们变得更好，将越来越具有替代性”。
OpenAI 总裁 Greg Brockman 将这些模型描述为“非常擅长新闻”。Nadella 今年早些时候在宣誓后同意，与聊天机器人对话“已经替代了……在 AI 平台上直接把信息呈现在你面前，而不是需要前往底层来源。”
这类措辞说明，这项技术可能如何直接与原始作品竞争，而不是对其进行改造。
一份微软文件指出，生成式 AI 存在“真实风险”，可能“严重扰乱那些生成了基础模型训练所用数据的人们的就业。”
复制规模之大令人震惊。这些文件首次披露，仅 OpenAI 的中期训练数据集就包含超过 91,692 份由 NYT、Daily News 和 Center for Investigative Reporting 发表的作品副本。一个源自 Common Crawl 的数据集仅来自 nytimes.com 的文档就超过 200 万份。
在 2023 年 1 月的一份内部备忘录中，Hecht 将其称为“一场规模空前的惊人盗窃”，以及“人类历史上最大的劳动成果盗窃”。
该文件以新的细节阐述了 OpenAI 和 Microsoft 是如何获取原告内容的，包括从 Bing Index 中抓取这些内容。
“OpenAI 将整个 GPT-3 训练数据集交付给了 Microsoft，Microsoft 用其评估如何在自己的商业产品中部署 OpenAI 的模型，”该文件写道。“Microsoft 同样通过名为 Project Taxi 和 Project Mango 的计划向 OpenAI 提供了训练数据。”
据称，这些公司将 Project Mango 的数据汇编成了一个训练数据集，其中包含至少 160,903 件来自这些新闻出版商的独特作品的副本。
为了最大限度地利用其抓取成果，OpenAI 员工据称想出了一项在不被察觉的情况下绕过付费墙的计划。文件显示，当 OpenAI 研究员 Nick Ryder 向 Brockman 提到一项“绕过纽约时报付费墙的黑客手段”时，Brockman 回复道：“啊，不错。”
OpenAI 员工还被指构建了 WebText 和 WebText2 等训练数据集，这些数据集过度依赖抓取来的新闻内容。他们还据称从 Common Crawl 中提取了数百万篇文章，Common Crawl 是一个免费、开放的网页抓取数据存储库。调查结果还描述了在训练数据进入模型之前，有人刻意去除其中的版权声明，因为研究人员“不希望模型向用户输出”“版权声明”。
OpenAI 和 Microsoft 未回应置评请求。
New unredacted information in the copyright lawsuit The New York Times brought against OpenAI and Microsoft three years ago reveals an admission that AI scraping was tantamount to theft, and that AI products pose a major threat to publications.
Per the lawsuit, a top Microsoft executive privately described the companies’ AI training practices as “theft,” and OpenAI’s own leadership said its AI models posed an “existential threat” to the publishers and journalists whose work trained them.
The unsealed material also details how the companies allegedly obtained and used that content by bypassing paywalls undetected, building training datasets via mass scraping, and deliberately stripping copyright notices from training data.
It’s worth noting that much of the new information comes from The Times’ own brief, not the underlying exhibits, which remain sealed. The quotes below are presented without their original context.
The unredacted filing is the latest escalation in the three-year-old lawsuit, in which The New York Times initially alleged the firms violated copyright law by training generative AI models on its content.
The question of whether AI firms can legally use copyrighted material to train AI has no clear answer, but judges have been largely favorable to AI companies’ arguments that training constitutes “fair use.” This legal rule lets people use copyrighted work without permission in certain cases, like parody, news reporting, or criticism. Earlier this month, the Trump administration contributed a brief in defense of OpenAI’s unlicensed use of copyrighted material to train its LLMs.
Several of the new admissions, however, run counter to OpenAI’s fair use defense, particularly the rule’s requirement that use doesn’t substitute or harm the market for the original work.
For example, Microsoft’s own data shows its Copilot “answer engine” caused click-through rates for The New York Times’ domain to drop as much as 93% compared to traditional Bing search. An internal Microsoft presentation written by Microsoft’s Director of Applied Science, Brent Hecht, in January 2024 describes the decline as a “doom loop” that would “hurt the performance of our models and the entire web at the same time.”
“It is highly unusual that an end-product threatens the economic foundations of its essential suppliers, but that is the situation we have created for our LLM business with respect to its ‘content supply chain,’” reads the Microsoft document, as quoted in the filing.
Microsoft CEO Satya Nadella also testified in a deposition earlier this year that “anything that is paywalled should be licensed by anyone who wants to use it…for grounding or training,” and made clear that, if he “had been made aware that OpenAI had scraped and trained on information that was behind a paywall,” he would have “invoked [Microsoft’s right to] require OpenAI to retrain its models.”
Other admissions cut against different pillars of the fair-use test: OpenAI’s Head of ChatGPT, Nick Turley, wrote in internal communication that publishers face an “existential threat” from products like the chatbot, which are “largely substitutive” and “will get more and more substitutive as they get better.”
OpenAI President Greg Brockman described the models as “excellent at news.” Nadella agreed under oath earlier this year that conversing with chatbots “has substituted … giving you the information right there on the website on the AI platform versus needing to go to the underlying source.”
That kind of language speaks to how the technology could directly compete with, rather than transform, the original work.
A Microsoft document states that there is a “real risk” that generative AI could “significantly disrupt the employment of the very people who generated the data on which the foundation model was trained.”
The sheer scale of the copying is striking. The documents reveal for the first time that OpenAI’s mid-training datasets alone contain more than 91,692 copies of works published by the NYT, Daily News, and Center for Investigative Reporting. A Common Crawl-derived dataset included more than 2 million documents from nytimes.com alone.
In a January 2023 internal memo, Hecht called it “an astonishing theft of unprecedented proportions” and “the largest theft of labor in human history.”
The filing lays out in new detail how OpenAI and Microsoft went about acquiring the plaintiffs’ content, including scraping it from the Bing Index.
“OpenAI delivered the entire GPT-3 training dataset to Microsoft, which Microsoft used to evaluate how to implement OpenAI’s models within its own commercial products,” the filing reads. “Microsoft similarly provided training data to OpenAI through initiatives called Project Taxi and Project Mango.”
The companies allegedly assembled the Project Mango data into a training dataset that contains copies of at least 160,903 unique works from the news publishers.
In order to get the most out of their scraping, OpenAI employees allegedly came up with a plan to circumvent paywalls without detection. The filings show that when OpenAI researcher Nick Ryder told Brockman about a “hack to get around nytimes paywall,” Brockman replied: “ah nice.”
OpenAI employees also allegedly built training datasets like WebText and WebText2 that disproportionately relied on scraped news content. They also allegedly pulled millions of articles from Common Crawl, a free, open repository of web crawl data. The findings also describe deliberate efforts to strip copyright notices from training data before it reached the model, since researchers “wouldn’t want model outputting” “copyright notices” to users.
OpenAI and Microsoft did not return requests for comment.