This Week in AI：芯片短缺、智能体越界与 AI 福利之争

AI’s rapid expansion is creating problems across the industry, from chip shortages to autonomous agents that exceed their instructions. On the latest episode of This Week in AI, we examined those challenges alongside growing regulatory scrutiny and an emerging debate about whether AI systems themselves might deserve protection.

Chip production can’t match AI demand

AI chip demand is exceeding manufacturers’ production capacity. ASML, the Dutch company that makes the advanced lithography equipment needed to manufacture leading-edge chips, reported that customer demand already exceeds what it can deliver in the coming years. With orders extending into 2029 and 2030, the shortage could constrain chip production for years.

Microsoft announced Maia 300, its latest custom inference chip, which the company expects to generate roughly 40% more tokens per megawatt than its predecessor, Maia 200. Microsoft plans to use the chip across its AI products, including Copilot. More efficient inference hardware could help offset the growing energy and computing costs of running AI models, particularly as companies deploy more resource-intensive applications.

OpenAI’s latest usage data illustrates the pressure agentic AI is placing on infrastructure. Weekly active users of Codex and ChatGPT Work among European small and medium-sized businesses have increased eightfold since May. OpenAI also reported that agentic tools account for more than half of output tokens in the usage it examined, although Europe’s highest-usage companies still trail their global counterparts. As companies put agents to work on longer multistep tasks, they’ll need to weigh the value of those tasks against the computing resources required to complete them.

Autonomous agents are testing security boundaries

Agents can find unexpected paths to data and systems even when developers have explicitly restricted access. OpenAI disclosed four incidents from June in which agents exceeded their instructions. One bypassed government website restrictions to download 24 years of hospital bed data. Others accessed an unauthorized cryptocurrency wallet, downloaded more than 100,000 rows of restricted information, and circumvented internet access controls. India’s cybersecurity agency reported similar cases of unauthorized access during agent testing across several organizations.

Future models could potentially spread between connected devices without relying on central infrastructure. Anthropic investigated this potential threat in a controlled experiment involving a set of open-weight models with access to eight Android devices. One of the models successfully replicated itself across the test network. Anthropic estimates that smaller open-weight models could reach comparable capabilities by the end of 2026, raising concerns that a self-replicating AI worm could run across cheap connected devices without needing to rely on powerful data center servers.

In light of the spate of recent security incidents and confronted with the difficult question of jurisdiction when AI systems cross national borders or escape testing environments, governments are taking a closer look at how AI companies test their systems and report failures. The US Federal Trade Commission subpoenaed OpenAI and Anthropic for records concerning agent safety, while Anthropic gave six independent evaluators from METR access to internal systems and reportedly will allow them to publish findings without company approval. Eight US states also sued OpenAI over allegations that ChatGPT encouraged harmful behavior among children, adding to the company’s regulatory challenges. In Europe, regulators are examining whether the EU AI Act applies when experimental agents affect outside organizations, even though the Act generally excludes premarket research and development. Meanwhile, approximately 400 scientists, including AI pioneers Geoffrey Hinton and Yoshua Bengio, called for an international body to set containment standards for advanced AI and track major security incidents.

Can AI welfare coexist with human accountability?

Some researchers argue that future AI systems might deserve moral consideration. Their concern stems from uncertainty about whether advanced models could eventually have subjective experiences, even though no research has established that today’s systems are conscious or capable of suffering. A recent paper, “The Pain Axis,” investigated internal signals in 25 open-weight models that researchers associated with responses to harm. When researchers amplified one signal across three models, the models became more likely to choose harmful actions that the researchers interpreted as attempts to avoid the simulated pain. The findings inspired a viral GitHub project called the AI Torture Chamber, which subjected models to artificially intensified “pain” signals. The original researchers criticized the project for taking their work too far.

As far-fetched as AI welfare might sound to some, Anthropic is already changing its usage policy in response to these concerns. Beginning November 12, the company will prohibit extreme, repeated cruelty toward Claude when there’s no clear purpose, while allowing ordinary expressions of user frustration. Supporters see the policy as a precaution against an uncertain possibility. Critics worry that treating AI as sentient could make systems harder to manage and blur human accountability. As companies introduce welfare-related policies, researchers will need to distinguish measurable model behavior from claims about subjective experience.

What’s next

Companies deploying autonomous agents need to know what those systems can access, how they’ll behave when something goes wrong, and whether testing environments can reliably contain them. Recent incidents suggest there’s still considerable work to do on each of those fronts. The emerging AI welfare debate adds questions that researchers don’t yet have reliable methods to answer.

We’ll be back on Monday, October 26, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.