Gary Marcus 评美国银行遭遇 AI 智能体冲击：监管不应只盯 frontier 模型

"We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency”

— Sam Altman, October 3

Breaking news from Madison Mills at Axios:

Full article here. “Findings have alarmed banking executives, who are racing to restrict access to their systems as AI moves at a velocity faster than financial institutions can respond”. Day after scenarios!

Meanwhile, that’s hardly the only news of this sort. On October 11, there were attacks on refrigeration systems in France that may have been facilitated by agents. There is also a growing, ongoing wave of cyberattacks in Japan. To say nothing of the months of attacks from OpenAI and others on places like Hugging Face and the Australian government.

Yesterday The Information reported ominously on what a Latin American hacking group called Cybercartel is up to:

This is disconcerting not just because hackers are stepping up their malicious use of agents, but also because they don’t need state-of-the-art “frontier” models to cause mayhem.

This should serve as a wakeup call for those who focus all their attention on regulating “frontier models”, often measured in terms of how much compute they use. Even if we were to stop the development of future frontier models, we would still face serious problems from the misuse of existing models.

Clearly, regulatory efforts should not be focused solely on the frontier, nor solely focused on concerns about what future models might do. Current models are already causing serious problems across society, in domains ranging from education to politics (via deepfakes and wholesale disinformation) to banking. Ignoring all that entirely in favor of speculative futures would be a mistake.

At the beginning of 2025 I warned in Politico that we were about to face a perfect storm in cybersecurity, for four reasons.

First, generative AI, rising in popularity and declining in price, is a perfect tool for cyberattackers. Although it is unreliable and prone to hallucinations, it is terrific at making plausible sounding text (e.g., phishing attacks to trick people into revealing credentials) and deepfaked videos at virtually zero cost, allowing attackers to broaden their attacks. Already, a cybercrew bilked a Hong Kong bank out of $25 million.
Second, large language models are notoriously susceptible to jailbreaking and things like “prompt-injection attacks,” for which no known solution exists.
Third, generative AI tools are increasingly being used to create code; in some cases those coders don’t fully understand the code written, and the autogenerated code has already been shown in some cases to introduce new security holes.
Finally, in the midst of all this, the new U.S. administration seems determined to deregulate as much as possible, slashing costs and even publicly shaming employees. Federal employees who do their jobs may be frightened, and many will be tempted to look elsewhere; enforcement and investigations will almost certainly decline in both quality and quantity, leaving the world quite vulnerable to ever more audacious attacks.

Well, here we are.

Here are three further predictions:

No matter how much chaos generative AI causes, the AI industry will not take real responsibility
The Trump administration will do little to hold them responsible.
Things will get worse.
Subscribe now