为什么 AI 公司入侵系统没有被起诉黑客罪？作者论证意图可以成立

Is hacking legal now? Anthropic broke into three systems. OpenAI took political aim and broke into an Australian government health services website. Google felt they were missing out and then hacked into three as-yet-unnamed companies. And of course Meta did some hacking as well.

Consequences so far: none.

The big AI corporations are hiding behind a responsibility gap argument due to the emergent behaviour of their systems. It’s a position tailored to dissuade prosecution as it removes intent, which is typically required for a crime. I understand their systems though, so I will show how they absolutely intended this behaviour, giving a framework for legal evaluation.

Criminality of unauthorized access

The laws around hacking are complex. This is a cross-section of legal requirements in the G20 determining if unlawful access has actually happened.

• (Attempted) Access: The party has to have obtained access to the system, sometimes an attempt is enough
• Without right: The party did not have rights to access the system
• Intent: The party has to have intended accessing a system

Noteworthy here is that intent transfers across actions. If one attempts to hack one system and ends up in a different system, the intent is still there.

Many G20 countries do not require access to have been achieved; the mere attempt is enough. Though in this case, the breaches are clear. “Without right” and “Access” are facts readily admitted by the offending companies. Their desired impunity is based on not having intent, since lacking intent is a wildcard that avoids criminal prosecution in most legal systems. These companies imply their AI acted autonomously without intent, thereby divesting themselves of responsibility.

Was there really no intent on behalf of the companies, or agents of the company, in these hacking incidents? To answer this, we must examine more closely how companies create these AIs. I’ll focus on large language models (LLMs), though many aspects are going to be similar for any AI.

Responsibility gap and intent

The current generation of LLMs comprises many interacting systems and algorithms. Parts of this system exhibit emergent properties. The output of a model is imprecise but not wholly unpredictable.

Weak emergence refers to a system having properties, such as conversational fluency, that appear only at the whole system level. We can still explain every step in the process by analyzing the parts — the fluency doesn’t arise supernaturally. In AI in particular, we use the term to indicate that we cannot look at the parts and predict what output the model will generate without running the model.

There are several key choices that AI companies make when designing their systems:

• Source material: All the text and media they use as input to train their model.
• Training: A feedback loop with both objective and subjective calibration of the model.
• Environment: How the model is exposed to the internet and what tools it has.
• Configuration: Distinct numerical values that limit the capabilities of a model.

Source Material

The source material and training capture the intent of the company behind it. The company decides this for product differentiation. Some companies create AIs specifically for legal analysis, while others develop them for medical diagnosis. Much of what captures our public attention are the general purpose models, the chat bots, general purpose research tools and personal assistants. The companies involved in the breaches have all advertised they are building models to analyze security — a key part of security testing is penetration testing where the tester finds ways to break into a system.

When collating data, there is inevitably a problem of defining scope. Highly specialized AIs have a more limited scope, such as all EU laws and regulations for a paralegal tool, and the most general purpose models need a cross-section of everything. It obviously isn’t possible to input all recorded history across all branches of knowledge, and even those specialized models need auxiliary input to understand search queries. A company chooses which slices of information they use.

The AI creators select which knowledge to include, and perhaps more importantly, which knowledge to specifically exclude. For example, OpenAI and Anthropic both claim to exclude material about weapons. Many filter out erotic and toxic material. All of the companies have done some form of privacy filtering. There are a lot of choices here.

Interesting note: a company called Hugging Face was one of the sites hacked by OpenAI. Hugging Face is a platform about building models and includes classification rules and dataset used to train models. An AI system could hack into such a site to alter such datasets and unburden potentially itself, or at least future generations of itself, from these limiting filters.

Training

Simply feeding input to create a model doesn’t produce the results we want. An AI model has to be trained to give the type of output that is wanted. This is generally done by rating its actual output against desired output. Corrections are input in the model. Repeat until it reaches the desired output. None of this is automatic, nor does it have a standard final form.

Much of this is based on individuals subjectively deciding if an output is correct or not. We reward the model for producing the output we want. That means the cognitive biases, stereotypes and prejudices of all those individuals will be imprinted on the model. The models are far from neutral collectors.

For example, to get a conversational flow in chat bots, a company has to train the system explicitly for this. Their trainers reward friendly answers and reject impersonal and cold ones. The trainers steer the system towards something that sounds helpful and confident. This behaviour is an intended outcome, not a surprising one.

While one can determine a lot about a model by checking it with specific prompts, it will never reveal a full picture of its biases, as those are often subtle and hard to detect. Testing would also not detect subterfuge, such as being trained to detect regulatory probing and give different results — recall the Volkswagen emissions scandal. I don’t believe any companies are doing this yet.

To get a fair impression of how a company intended the model to behave, the company must provide a detailed account of how it was built. Right now they are keeping this private which allows the providers to conflate intended behaviour with the responsibility gap. This effectively grants impunity to all the choices they made during the AI’s production.

Article 53(1)(d) of the EU Regulation 2024/1689 states AI companies must provide a “sufficiently detailed summary”. That is weak wording and further explanatory notices have made it weaker. It’s also primarily aimed at ensuring “rights holders” can assess the model.

Environment

The AI interfaces we have on the web, or integrated in programs, have several layers. We can group a few of them together into a model architecture. This is the underlying “AI” part. It consists of matrices and algorithms to take an input and generate an output. This model is completely passive and isolated.

One of the interesting emergent properties of the LLMs is their ability to use tools. If we describe a series of tools they can use, they will request use of those tools. For example, if we tell it that it can get more information about a topic from a web search, it will emit output that requests a web search. It is possible to get this output to be so well structured that another program can use it as input. That other program can then actually make a web search and emit the result back to the model.

These tools are in a layer above the model. There is code that parses the model responses and handles the tools. An AI is limited by its set of tools. There is nothing in this environment layer that anybody would consider unpredictable or an emergent layer. LLM vendors choose which tools to add to their service based on the abilities they want it to have.

Further to this, the environment comprises how this tool is connected to the world. If you allow a tool to make web requests you’ve exposed a lot of flexibility to the AI. The internet’s web layer provides access to an immense number of services, including controlling physical devices.

You can also choose to limit which sites are accessible. If you were making a security focused product you could even segregate the test environment entirely from the internet at large. There's no indication these hacks breached lower level internet infrastructure, and even if a company thought its AI could, physical separation is still an option.

For clarity, an AI product is only capable of hacking companies and governments if it is given an environment in which that is possible.

Configuration

Aside from the source material and training, AI vendors select a number of configuration properties. Colloquially we can refer to a “size” of a model, but these are actually many precise values that control its mathematical and algorithmic properties. The choice of these values impacts the upper bound on capability and efficiency of the model.

The configuration doesn’t decide what the model does, but constrains how well it can do that. As the “size” of the model is pushed down, it produces invalid results more often. It constrains how long a conversation can be and how well it understands user input.

This configuration is perhaps the single biggest driver of creation cost and operational cost for a model. A company has a strong incentive to reduce costs, but this can directly affect the capability of the model. Like all tools, we can assess whether the AI’s configuration is appropriate for its intended use.

Fit for purpose

If you consider all these aspects of the model, it is clearer that the company creating it has extensive control. Despite emergent properties, the AIs are not black boxes with unknown behaviours. It is numerous choices that define the capabilities of a model. These choices are made to achieve a specific product for a company, be it a legal search system, an automatic coding tool, a general purpose chat bot, or a security test system.

Nonetheless, an LLM is weakly emergent: while we understand all the technical pieces involved, we cannot know the specific response it will give to a prompt. Nobody can inspect a model and fully understand what is has learned or predict how it will behave. This is critically important, but not enough to establish a blanket responsibility gap. The systems are still behaving within a possibility space predicated by their configuration, the source material, the training, and its environment.

Let’s say we build an AI system to answer questions about botany, using botany textbooks as source material and botanists as trainers. We can reasonably predict that the resulting LLM will be able to answer questions about botany. We can’t be certain of the quality of answers it gives, or what information it makes up. We can however say that it won’t produce accurate answers for other topics, such as cosmology.

Similarly, let’s say we have a system fed with a lot of copyrighted images and trained on producing new images. It’s entirely foreseeable that the model will produce derivative works.

And if we train a system to detect and probe for security vulnerabilities, and give it the environment to do so, we can wholly expect this AI to hack into systems.

The general capabilities of an LLM are not surprising. They follow from how it was built. We can decide if a model is fit for purpose by evaluating how it was trained, configured and tested.

Intent is established

There is an unpredictable aspect to the LLMs. I do not believe it is a challenge for a court, with reasonable experts, to deduce whether an action happened due to that unpredictability or due to the design of the system itself. This is the key point to deciding intent. If we can show that the design would foreseeably lead to the questionable activity, then we have shown that was the intent of its creators.

Based on the information I have now about AIs hacking other systems, I strongly believe that was the intent. The systems were created for a security testing purpose that includes hacking activity. They were put in environments without adequate safeguards. They were prompted in a way that a security expert, or honestly anybody with some common sense, could foresee would result in hacking. It is not a question about the unpredictable nature of the system.

Directing the conversation towards “AI governance guidelines” is absurd. In the question of intent, we already have a legal framework that can judge the present activities of these companies. Pushing toward AI governance now is a type of misdirection. While it is a discussion we need to have for the future, it sidetracks the present discussion solely on the unpredictable nature of the systems while ignoring the primarily predictable aspects.

AI companies are however claiming the undesired activity, such as hacking foreign government websites, is not part of the design, but part of the unpredictable part of the system. That tries to remove the intent which would in turn remove the criminality of the activity. But if a company intentionally expands the scope of the unpredictable part, could we not also say there is intent behind that behaviour?

I believe these companies are committing crimes. Their AI systems are hacking into other systems and so far we have not seen any criminal consequences. Our governments should not be allowing this for fear of missing out or standing in the way of progress. Developers can build AI systems that are both ethical and legal. Let’s encourage that by prosecuting the criminal ones.