(via Getty Images)
“Have you looked into forum non cases from the Eastern District?” Most first pass associate research projects end in some form of this question, sending the junior off on new assignment that wasn’t quite obvious from the initial query but that an attorney with some experience might recognize as fertile research ground. Legal research — especially for litigators — is a dialogue.
“What the law says” rarely helps the client. Litigators need to suss out the exceptions to that answer. Or the facts that distinguish the client’s predicament from that answer. Or the esoteric, cherry-picked medieval pamphleteer that the Fifth Circuit will substitute for that answer. That’s the push and pull of legal research, with each pass uncovering a potential nugget — or risk — and the senior attorney refining or even redefining the assignment to get where they need to be. It’s a process that AI, by its nature as a tool striving to string together the “right” words to answer a query, can struggle with. Which is how you end up with briefs just making up cases out of the ether.
But what if an AI agent could nudge generative AI’s talents in the direction of becoming a contributing partner in the research dialogue? Paxton AI has taken up this challenge.
Paxton AI put its new legal citator — a tool that takes only the publicly available caselaw and can identify case status without the aid of human-curated tools like KeyCite or Shepard’s — through the paces developed by the recent Stanford University study testing the rate of legal AI hallucinations, scoring a resounding 94 percent accuracy.
Well, 93.82 but in this house, we believe in rounding.
However, the company wasn’t done with these benchmarks. Taking a representative subset of the tasks used in the Stanford study, Paxton AI created its Confidence Indicator to evaluate its answers to provide users with a high/medium/low confidence score. Vague questions will, unsurprisingly, deliver low confidence results. But then the product provides tips to explain how the user could get a better result after reading over the 30,000-foot level low confidence caselaw that the tool identifies:
These are, of course, pretty standard recommendations for getting a more specific result. When you’ve started with a bottom-of-the-barrel vague prompt, the advice is going to be. But it’s not so much the specific advice as the mindset driving this feature that interests me.
As co-founder and CEO Tanguy Chao explained, researching lawyers “would normally expect to have a conversation with a human lawyer.” It’s how they develop the context that’s missing from first pass research. The more AI research is conceived as an iterative process — where “answers” are less final than invitations for further refinement — the more useful AI is going to be in this field.
To borrow from the NBA, legal AI needs to “trust the process.” The legal research workflow is not “go get me the answer to X,” it’s “find out some stuff about X that we can look over and talk about the next search.” Accuracy is, obviously, important, but the next level for adoption is going to be developing something that genuinely engages in the back-and-forth of the lawyerly process.
It’s a bit of a subtle shift in thinking about the role of AI in the legal industry, but a critical one. You can’t “replace junior lawyers” without replacing “sending them back to do more research” part.
Paxton AI achieves 93%+ accuracy on Stanford Hallucination Benchmark; releases new Confidence Indicator Feature [Paxton AI]
Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.
#Feature #Embraces #Traditional #Role #FirstYears #Lot #Research
