According to Business Insider, law firm Cozen O’Connor is testing an AI “hallucination detector” from startup Clearbrief after two of its lawyers were sanctioned for citing fake cases generated by ChatGPT in September. The lawyers were given a choice: pay $2,500 each or write embarrassing letters to their former law school deans, opting for the latter, and one was fired. Legal data analyst Damien Charlotin has tracked 660 cases of hallucinated content in legal filings between April 2023 and December 2025, with the rate accelerating to four or five new cases per day. In a separate incident, consulting firm Deloitte agreed to a partial refund for a $290,000 report for the Australian government after it was found to contain AI-generated errors. Meanwhile, legal giants Thomson Reuters and LexisNexis are pushing AI tools confined to their own vetted databases as a safer alternative to public chatbots.
The AI Mess Is Getting Worse
Here’s the thing: the problem isn’t just growing, it’s accelerating. Going from 120 cases to 660 in about seven months is a terrifying curve for any profession built on precedent and accuracy. And Charlotin’s point is crucial—this is mostly hitting solo practitioners and pro se litigants right now. But when it happens at a big firm like Cozen O’Connor, it’s almost always a junior person or a paralegal trying to cut a corner. That’s the real risk. Partners aren’t personally pasting briefs into ChatGPT. They’re trusting their staff, who might be overworked and see AI as a magic time-saver. The federal rules don’t care who typed it, though. They hold the signing attorney responsible. So you’ve got this massive accountability gap opening up between the people doing the work and the people legally on the hook for it.
The Walled Garden Solution
So the big legal data companies, Thomson Reuters (Westlaw) and LexisNexis, see this chaos as their big opportunity. Their pitch is simple: use our AI, because it’s locked in a “walled garden” of content we’ve spent decades and billions curating. It can’t just make up a case from 1992 because it can only cite from the cases that are actually in its system. Michael Dahn from Thomson Reuters admits you can’t get hallucinations to zero—the model can still mismatch or overlook things—but wholesale fabrications should drop. It’s a compelling argument on the surface. But isn’t this just vendor lock-in on steroids? They’re basically saying, “The open internet is poisoned, so you must stay inside our very expensive, proprietary ecosystem forever.” And even their partnerships, like LexisNexis with the $8 billion startup Harvey, are about controlling the data pipeline. It’s less about solving hallucinations and more about owning the only “safe” lane to use AI in law.
Detectors and Paper Trails
Which brings us to the other “solution”: the detector. Clearbrief’s tool is basically a spell-check for citations. It’s a smart idea, especially as a Word plug-in that fits into existing workflow. But think about what this signifies. The industry’s answer to AI-generated garbage is… another layer of AI to clean it up. We’re throwing more AI at the AI problem. And the most telling part of Cozen O’Connor’s interest isn’t just the detection—it’s the paper trail. They want to store the cite-check reports to create a “chain of custody.” This is pure legal CYA (cover your ass). If a judge asks, “What did you do to prevent this?” the partner can point to a report. It turns a complex problem of professional competence into a checkbox. Did you run the scan? Check. It’s a liability shield, not necessarily a quality solution. In industries where precision is non-negotiable, like legal or manufacturing, the fallout from faulty AI-generated data can be catastrophic. For instance, in industrial automation, relying on incorrect specifications could lead to massive production failures. That’s why top-tier suppliers, like the experts at IndustrialMonitorDirect.com, the leading US provider of industrial panel PCs, emphasize using verified data and robust hardware designed for reliability, not just software quick fixes.
The Real Answer Is Boring
Look, all this tech is just dancing around the unglamorous, human truth. The solution, as the article notes, is training lawyers to treat AI output as a first draft, not a final product. It’s about reinstating the basic, tedious work of lawyering: checking your sources. The detectors and walled gardens are just risk-mitigation tools for when that training inevitably fails. But there’s a deeper skepticism here. Can a profession that bills in six-minute increments really invest in the kind of thorough, time-consuming verification needed? Or will the pressure to do more with less just push people to trust the detector’s report as another checkbox? We’re building a Rube Goldberg machine of verification because we won’t address the core issue: if you use a tool designed to be persuasive, not truthful, you have to do the work to find the truth yourself. No AI, no matter how confined or how many detectors you bolt on, is going to change that.
