AI Legal Tools Frequently Hallucinate Answers, Study Finds

A new study from researchers at Stanford University found that AI-powered legal tools generate hallucinations in one in every six queries. 

Stanford’s RegLab and Institute for Human-Centered Artificial Intelligence (HAI) analyzed the effectiveness of AI-powered legal tools from LexisNexis and Thomson Reuters.

The researchers found that despite being advertised as bespoke legal AI tools, they produced false or incorrect information “an alarming amount of the time.”

“The core promise of legal AI is that it can streamline the time-consuming process of identifying relevant legal sources,” the report read. “If a tool provides sources that seem authoritative but are in reality irrelevant or contradictory, users could be misled. They may place undue trust in the tool's output, potentially leading to erroneous legal judgments and conclusions.”

The AI tools were asked more than 200 questions covering general research, jurisdiction-specific questions and questions that mimic a user having a mistaken understanding of the law.

It also faced queries in simple facts that require no legal interpretation.

Researchers found that more than 17% of the time—one in every six queries — the AI legal tools from both LexisNexis and Thomson Reuters generated hallucinated responses, either by getting answers correct but citing incorrect sources or creating entirely inaccurate responses.

Related:US Chief Justice: AI is Here to Stay, But So Are Judges