AI Hallucinations in Court: When the Algorithm Lies — and Lawyers Pay the Price

The Algorithm Hallucinated. The Lawyer Signed It. The Court Noticed.

In June 2023, attorney Steven Schwartz of the New York firm Levidow, Levidow & Oberman filed a brief in federal court opposing a motion to dismiss a personal injury claim against Avianca Airlines. The brief cited six federal court decisions as controlling authority. There was just one problem: none of those cases existed. Mata v. Avianca, Inc. became the case in which the U.S. District Court for the Southern District of New York dismissed the personal injury claim and issued a $5,000 fine to the plaintiffs' lawyers for submitting fake precedents generated by ChatGPT.

As it was later revealed, Schwartz had used ChatGPT, which fabricated the cited cases — and he testified that he operated "under the false assumption and disbelief that this website could produce completely fabricated cases." When Avianca's counsel demanded copies of the cases, Schwartz asked ChatGPT to confirm them. ChatGPT assured the attorneys that the cases "indeed exist" and "can be found in reputable legal databases such as LexisNexis and Westlaw." They did not exist. They had never existed. The aggravating fact in Mata was not the original ChatGPT misuse but Schwartz's continued defense of the fabricated citations, including production of additional ChatGPT-fabricated opinion excerpts during the show-cause proceedings.

Within weeks, the case became the defining example of AI hallucination in legal practice, appearing in ethics CLEs, bar advisories, and major law publications worldwide. The term "AI hallucination" had been used in computer science circles since 2017, but Mata v. Avianca brought it into mainstream legal discourse as a vivid example of the phenomenon's real-world consequences.

That was just the beginning.

From Isolated Incident to Systemic Crisis: The Data

U.S. courts alone recorded 487 instances of AI errors or hallucinations in court documents during 2025 — more than 10 times the 2024 total. Licensed attorneys accounted for 37.8% of these problematic filings. The trajectory is stark and unforgiving.

In 2023, the first major incident — two New York personal injury attorneys sanctioned for submitting an AI-generated brief with fabricated case citations — was treated as an outlier. It was not. By 2024, Law360's AI tracker had documented 280 incidents. By the close of 2025: 729+. In Q1 2026 alone, new cases are being added weekly.

Sources: SurePoint Technologies, 2025 State of the Legal Industry Report; Law360 AI Tracker, via Development Corporate; Artificial Lawyer 2026 Predictions

The adoption side of the equation makes these numbers more alarming, not less. According to the 2026 Legal Industry Report from 8am™, 69% of legal professionals report personally using Generative AI tools such as ChatGPT, Gemini, or Claude for work-related purposes — a dramatic increase from less than one-third (31%) in the 2025 report. While 63% of mid-sized law firms have formally adopted gen AI, 81% of firm leaders report internal concern about its reliability and risk. The gap between adoption velocity and institutional safeguards is where hallucination disasters are born.

What Exactly Is an AI Hallucination — and Why Does It Happen in Law?

The term "hallucination" in AI refers to a model generating plausible-sounding but factually false output. In legal practice, this failure mode is particularly dangerous because the law is built on citation, precedent, and verifiability. A hallucinated case doesn't just contain a factual error — it fabricates the entire architecture of legal authority.

According to Sterne Kessler's 2025 AI sanctions review, there appear to be three primary categories of AI hallucinations in legal filings: citations to fictitious cases; fabricated citations to real cases or documents; and citations to real quotes from real cases that fail to support — or directly contradict — a proposed legal proposition.

The third category is perhaps the most insidious. A cited case can exist in Westlaw, but the AI-attributed quote may never appear in it. The case name is real. The citation is real. The quotation is invented. A lawyer conducting only a superficial check could miss it entirely.

The hallucination rate among AI tools used for legal research remains alarming even in purpose-built systems. A peer-reviewed study published in the Journal of Empirical Legal Studies found that over 1 in 6 queries caused Lexis+ AI and Ask Practical Law AI to respond with misleading or false information — and Westlaw hallucinated substantially more, with one-third of its responses containing a hallucination. A prior Stanford HAI study found that general-purpose chatbots hallucinated between 58% and 82% of the time on legal queries.

Sources: Magesh et al., "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools," Journal of Empirical Legal Studies, 2025; Stanford HAI, 2024

What makes hallucinated citations so dangerous is that they're usually exactly the citations needed, which is exciting. And that excitement — the thrill of finding the perfect precedent — can override professional judgment. An AI tool doesn't produce a clearly wrong answer; it produces a devastatingly convincing almost-right one. That's the trap.

The Anatomy of Disaster: Key Cases and Consequences

Mata v. Avianca (S.D.N.Y., 2023) — The Watershed Moment

Mata v. Avianca is the canonical federal AI hallucination sanctions case and the order that bar associations, malpractice carriers, and dozens of subsequent courts cite as the cautionary precedent for unverified AI-generated authority. The opinion crystallizes three rules-based exposures: Rule 11's reasonable inquiry duty is non-delegable to an AI tool; Model Rule 5.3's supervision logic reaches firm-level oversight of AI-assisted work product (the firm was sanctioned alongside the individual attorneys); and Rule 3.3 candor is tested most acutely after a citation is challenged, where the cheaper path is immediate disclosure rather than defense.

The $31,100 Sanction That Almost Fooled a Judge

A plaintiff's law firm was sanctioned and ordered to pay $31,100 after submitting fake AI citations that nearly ended up in a court ruling. Michael Wilner, a retired U.S. magistrate judge serving as special master in the U.S. District Court for the Central District of California, admitted that he initially thought the citations were real and "almost" put them into an order. Judge Wilner wrote: "Directly put, Plaintiff's use of AI affirmatively misled me." The precedent is chilling: the hallucination came within one step of becoming embedded in actual case law.

California Appellate Court: $10,000 and the State's First Published Hallucination Opinion

On September 12, 2025, the California Court of Appeal issued California's first published opinion addressing AI "hallucinations" in court filings. In Noland v. Land of the Free, L.P., the court affirmed summary judgment for the defendant and imposed a $10,000 sanction on the offending attorney — making the consequences of unverified AI use part of published, citable California law.

Dehghani v. Castro (D.N.M.) — Outsourcing Isn't an Excuse

In Dehghani v. Castro, the petitioner's attorney purchased a brief from a freelance attorney who likely used GenAI and then destroyed all notes. The purchasing attorney did not review the purchased brief before submitting it. The court issued two show-cause orders, but the petitioner's attorney did not adequately respond to either request. The court issued a sanctions order requiring a fine, mandatory CLE training, and self-reporting to all appropriate state bars. The message is unambiguous: the supervising attorney cannot outsource their verification duty to a third party.

Johnson v. Dunn (Alabama, 2025) — Sanctions May Not Be Enough

In Johnson v. Dunn, a July 2025 Alabama case, a large law firm was sanctioned for AI hallucinations, with the court declaring that monetary sanctions alone may be insufficient to deter such conduct. This signals a judicial mood shift: courts are now openly contemplating non-monetary deterrence, including referrals to disciplinary authorities and reputational consequences.

Escalating Sanctions: The Financial Arc

The first cases drew $500 fines and judicial warnings. By late 2025, sanctions were reaching five figures. According to Artificial Lawyer's 2026 predictions, courts levied attorney fees and sanctions exceeding $100,000 in individual AI hallucination cases.

The Regulatory Framework: What the Rules Actually Require

ABA Formal Opinion 512: The New Rulebook

On July 29, 2024, the American Bar Association Standing Committee on Ethics and Professional Responsibility released its first formal opinion covering the growing use of generative artificial intelligence in the practice of law. ABA Formal Opinion 512 states that to ensure clients are protected, lawyers and law firms using GAI must "fully consider their applicable ethical obligations," which includes duties to provide competent legal representation, to protect client information, to communicate with clients, and to charge reasonable fees consistent with time spent using GAI.

Formal Opinion 512 addresses six primary areas of ethical concern: competence (Model Rule 1.1), confidentiality (Model Rule 1.6), communication with clients (Model Rule 1.4), candor toward the tribunal (Model Rules 3.1 and 3.3), supervisory responsibilities (Model Rules 5.1 and 5.3), and reasonable fees.

While GAI tools can enhance efficiency and quality of legal services, the Opinion emphasizes they cannot replace the attorney's professional judgment and experience necessary for competent client representation. One commentator noted that Formal Opinion 512 is the new rulebook — officially — which means ignorance of ethical considerations and guidelines is no longer acceptable.

Court-Level Rules: A Patchwork of Disclosure Requirements

Since Judge Brantley Starr of the U.S. District Court for the Northern District of Texas issued the first standing order on the use of AI in preparing court filings in 2023, hundreds of state and federal judges have amended or issued standing orders, general orders, local rules, pretrial orders, and other guidance to address AI use and misuse in their courtrooms.

The result is a compliance landscape of staggering complexity. As of early 2026, at least 25 federal district courts have adopted standing orders or local rules requiring attorneys to certify whether AI was used in preparing filings. But uniformity is conspicuously absent.

Sources: American Bar Association, "Court-Mandated Disclosure of AI in Court Submissions"; Hintyr, "AI Disclosure in Court: 300+ Rules," 2026; Eve Legal, "AI Disclosure Rules in Legal Filings"

Currently, there is no explicit guidance for federal courts as a whole on how to address the misuse of GenAI. Broadly speaking, courts have applied existing rules and laws to attorney, expert, and judge misuse of AI. However, individual judges have taken their own approaches to punishing attorneys who submit AI hallucinations and have inconsistently applied punishments.

This inconsistency is itself a risk. A lawyer practicing across multiple jurisdictions must navigate a minefield of standing orders that may contradict each other, with no single federal standard in sight. If you're hoping for a single federal standard that replaces the 300+ standing orders, you'll be waiting a while. Proposed Federal Rule of Evidence 707 would govern the admissibility of machine-generated evidence in federal court, and the Advisory Committee on Evidence Rules voted 8-1 to seek public comment in May 2025 — but the earliest possible effective date is December 1, 2027.

Why This Is More Than a Technology Problem

The Supervision Gap

The cases reveal a consistent pattern: the hallucination itself was rarely the final error. The final error was the failure to verify. Every single sanctioned attorney had one thing in common: they trusted AI output without independent verification. The tool was not the problem. The workflow was.

Overreliance on AI in legal practice can pose serious long-term risks. One of the most concerning is the creation of fictitious case law. A growing related issue is the rise of "AI slop" — AI-generated content that looks polished but lacks substance. These outputs often require human intervention to correct, refine, or completely redo, wasting time instead of saving it and resulting in low-quality work product.

The Excitement Problem

Legal professionals are trained to be skeptical of sources. They are trained to interrogate authority. And yet hallucinations keep slipping through. The explanation is behavioral, not just procedural. Hallucinated citations are typically exactly the citations needed — which creates excitement. And that excitement can override professional judgment. The AI didn't produce a generic fake. It produced what appeared to be the perfect answer to the lawyer's specific research problem. That confirmation-bias amplification is uniquely dangerous in time-pressured legal workflows.

The Threat to the Justice System Itself

As University of Miami law professor Christina Frohock, whose paper "Ghosts at the Gate" will appear in the Penn State Law Review, has argued: "When lawyers cite hallucinated case opinions, those citations can mislead judges and clients. The hallucinations might then appear in a court order and sway an actual dispute between actual parties. To quote a federal court in California, that potential outcome is 'scary.'" If fake cases become prevalent and effective, they will undermine the integrity of the legal system and erode trust in judicial orders.

This is not a distant hypothetical. In the California sanctions case, Judge Wilner admitted he initially thought the citations were real and "almost" put them into an order. Hallucinated precedent nearly became embedded in published case law. The entire edifice of common law rests on the trustworthiness of citations. That foundation is now actively under stress.

Who Is Filing the Hallucinated Content?

A significant and often overlooked data point concerns who is generating these filings. In 2025, pro se litigants accounted for 39% more hallucination incidents than licensed attorneys — 304 versus 219 incidents worldwide. This matters enormously for the courts: it means the hallucination problem is not just a professional misconduct issue among licensed attorneys — it's a systemic access-to-justice challenge affecting unrepresented parties who have even less oversight and understanding of AI's limitations.

For law firms, this has a practical implication as well: judges who have grown accustomed to seeing AI hallucinations from pro se filers have become increasingly alert to them in professionally-filed documents — and the tolerance for errors by licensed counsel is correspondingly lower.

The Compliance Checklist: What Lawyers and Law Firms Must Do Now

As U.S. law firms rapidly integrate AI, compliance with existing ethical rules is non-negotiable. Firms must uphold competence by exercising an appropriate degree of independent verification of all AI output; prevent confidentiality risks by implementing robust safeguards against unauthorized disclosure of client information; ensure candor by verifying the factual and legal bases for all AI-assisted filings; and meet supervision obligations by establishing clear internal policies and training protocols for all staff.

Translating this into operational practice requires moving beyond policy documents and into workflow architecture. Here is a practical framework:

The Five-Layer Verification Protocol

1. Source verification: Every cited case must be retrieved directly from Westlaw, Lexis, or official court databases — not from the AI output itself. The citation must match the actual document, page by page.

2. Quotation verification: Every quoted passage must be located in the source document. AI-generated quotes from real cases are the most dangerous hallucination category because the case itself validates the citation at first glance.

3. Proposition verification: Even when a quote exists in a real case, verify that the case actually supports the legal proposition for which it is being cited. Mischaracterized authority can be as professionally damaging as invented authority.

4. Supervision logging: Maintain an audit trail documenting which AI tools were used, by whom, at what stage, and what verification steps were taken. This is both an ethical safeguard and a defense asset in the event of a sanctions inquiry.

5. Pre-filing jurisdiction check: Before every filing, review the assigned judge's current standing orders for AI disclosure requirements. To date, no reported case has sanctioned an attorney for over-disclosing AI use — but multiple have been sanctioned for under-disclosing it.

Institutional Policy: What Firms Must Implement

Sources: ABA Formal Opinion 512 (2024); Clio, "AI Legal Compliance for Law Firms," 2026

The Business Case for Getting This Right

Beyond professional responsibility, there is a stark commercial argument for rigorous AI governance. For leading law firms, the fallout after having been caught inserting even a single AI-generated hallucination into a legal pleading appears to be catastrophic — almost like a data breach incident.

Client sentiment around AI is mixed. According to Clio's 2025 Legal Trends Report, more than half of consumers take issue with lawyers using AI, yet most want to know whether their lawyer is using it. Transparency, therefore, is not just an ethical requirement — it is a competitive differentiator. Firms that can demonstrate structured, governed, verified AI use will command greater client confidence than those deploying AI informally with no oversight architecture.

The malpractice exposure angle deserves equal attention. Poorly generated legal documents can harm clients. Inaccuracies or omissions introduced by AI may lead to unfavorable rulings, procedural errors, or even sanctions. Malpractice insurers are already factoring AI governance into their coverage decisions. Firms without documented AI policies may find themselves underinsured in the event of an AI-related claim.

And then there is the reputational dimension. Sanctions orders are public documents. They are indexed by legal databases, cited in subsequent cases, and picked up by legal trade press. Violators often face public shaming. In a recent South Florida case, a judge sanctioned a lawyer for including "false, fake, non-existent, AI-generated legal authorities" in eight related cases. The reputational damage from such a characterization, appearing in published opinions and legal news, is not recoverable on a short timescale.

The Horizon: Where This Is Heading

The regulatory trajectory is unmistakably toward greater accountability, not less. Courts have responded with standing orders and local rules, and bar associations have issued advisory opinions, including ABA Formal Opinion 512 on Generative Artificial Intelligence Tools. Despite this, the misuse of AI in court filings continues.

As of 2026, over 700 court cases now involve AI-generated hallucinations or fabricated content, according to legal analytics tracking by LexisNexis and Bloomberg Law. The crisis has not peaked. The collision between mass-market AI tools and professional verification duties will continue to generate sanctions, disciplinary referrals, and published opinions that shape the evolving standard of care for legal AI use.

While individual AI use is accelerating rapidly, firm-level governance and structured implementation are still catching up. The result is a widening gap between experimentation and institutional readiness. Closing that gap is not optional. It is the fundamental professional responsibility challenge of this decade for the legal industry.

The Bottom Line for Law Firms and Legal Departments

AI tools are not going away, nor should they. The efficiency gains in legal research, drafting, document review, and contract analysis are real and measurable. The risk is not AI itself — it is the false confidence that AI's fluency creates. A large language model that generates a perfectly-formatted case citation with a confident, authoritative tone is far more dangerous than a clearly broken tool. Its errors are invisible until they're not.

The legal profession's standard of care has now expanded to encompass AI governance. The collision between rapid AI adoption and persistent hallucination risks has created what legal ethics experts call the most significant professional responsibility crisis in a generation. The firms that emerge from this period with their reputations and licenses intact will be those that treated AI as infrastructure requiring governance — not as a shortcut requiring none.

Verification is not optional. Supervision is not optional. Disclosure is not optional. And ignorance, as the courts have made abundantly clear, is no defense at all.