A justice asked, ‘Counsel, do you watch the news?’ when a lawyer admitted to relying on ChatGPT without understanding the risks.

Illustration by Cath Virginia / The Verge | Photos from Getty Images

New Mexico’s Supreme Court is punishing a lawyer for including AI-fabricated witnesses and fake police testimony in an appeal for his client’s murder conviction, according to a report from Reuters. In a filing on Wednesday, the court fined Stephen Aarons $5,000 and held him in contempt for failing to “verify the factual claims and legal authority in his AI-generated brief.”

The filing says the brief “contained false testimony from wholly fabricated witnesses,” along with “false testimony” about the shooter’s clothing and appearance. Justice C. Shannon Bacon questioned how Aarons wasn’t aware of the risks posed by AI during an August hearing, where he admitted to using ChatGPT and thinking it would generate a “bulletproof summary” of the trial. “Counsel, do you watch the news? Do you ​listen to the radio? Do you read anything about what’s going on in the world?” Justice Bacon said, according to Reuters. “Because the problem with ​lawyers relying on AI hallucinations is an above-the-fold story every single day.”

With the number of lawyers using AI tools in their work increasing, so are the instances in which a court discovers citations that are fabricated — or hallucinated — by AI. Last year, a judge slammed two law firms for submitting a brief with “numerous false, inaccurate, and misleading legal citations and quotations,” while lawyers for MyPillow’s Mike Lindell were also fined for putting AI-generated misquotes and fake AI citations in a brief.

”I am remorseful but hopeful that the disciplinary board takes into account it was an honest mistake,” Aarons says in a statement to Reuters.