Some AI models struggle when it comes to interpreting nuance and judgment, a particularly acute challenge in an area like law
Generative AI wowed the world with its ability to instantly produce incredibly cogent text in response to just a few user prompts. While that feat was enough to create a hype machine following the introduction of ChatGPT a year and a half ago, that single capability does not fully capture the depth and breadth of how generative AI is being used in the real world today. It also engenders an overly reductive view of how important specialized data sets and human expertise are when it comes to developing generative AI models and applications.
The core breakthrough that allows generative AI to quickly interpret and produce text is entirely algorithmic. These large language models (LLMs) determine the statistical likelihood of one word following another to ultimately produce coherent text and other modes too. While that’s great for certain written content generation tasks like coding or responding to general knowledge questions, many models struggle when it comes to interpreting nuance and judgment along the way.
This is a particularly acute challenge in an area like law, where understanding relevant legal precedent requires not only the words but also the structure of the case, the jurisdiction and even details like the specific judge presiding.
Take, for example, a legal research project focused on establishing relevant precedent for successful patent defenses. A simple case count and decision analysis tracking intellectual property law over the past five years would reveal that an exceedingly high number of patent cases were tried and successfully defended in the Western District of Texas. In fact, that research will show that between 2018 and 2020, some 2,400 patent cases were tried in the court, all under a single patent-friendly judge. In 2020 alone, the court saw 20% of all patent cases in the U.S. and many of those resulted in favorable decisions for plaintiffs.
Based on that information, a researcher relying on a basic generative AI search tool that’s scouring through publicly available data for insights would be led to believe that Waco, TX is the optimal venue for a patent defense. What those results might have missed, however, is the July 25, 2022 order by Chief U.S. District Judge Orlando Garcia to introduce new rules for patent cases in the Western District of Texas. Notably, these new rules redistributed patent cases filed in the district to other judges, effectively ending the reign of Waco as the patent defense capital of the country.
To get to that level of specialization and domain-specific nuance, generative AI tools designed for professional-grade tasks, like legal research, need to go above and beyond off-the-shelf LLMs. While these models are a critical starting point to building the underlying generative AI infrastructure and they are good at producing base-level results, when it comes to fine-tuning and optimizing those results for highly specialized tasks, they need to be augmented with a combination of smaller models, non-generative models and proprietary data sets that have been curated by experts to produce reliable, accurate results.
Getting this choreography between models and data sets right, however, requires more than just applying a retrieval augmented generation (RAG) framework to a broad collection of data sets. It requires a great deal of fine-tuning to align models, structure and index the data needed to support the use case and tailor and constrain them to specific use cases. This is where human expertise is critical. Keeping with our patent law research example, attorney editors have been refining the art of annotating and tagging legal text for the better part of 150 years and they understand that topline numbers like raw case counts and decisions are not always enough to reveal what’s really going on in a particular legal matter. They know that patent law can be complicated and they know to look at the details surrounding a trend, not just the results.
That domain knowledge cannot be ignored when programming and testing generative AI models to deliver highly tailored results. It’s also a critical step to de-risking the potential downsides of the technology.
Humans conducting research without the benefit of technology often shape their understanding of a topic as they compile each new data point. Building that type of complete picture through exploration of these various pathways, including some dead ends, takes a tremendous amount of time and creative thinking, though. Where generative AI can help most is not by replacing or eliminating that process but by allowing these pathways to be explored more quickly and automatically, thereby giving people a more complete picture with alternative views. When researchers can access those results more quickly, with the confidence that they have been developed with the highest standards of accuracy and responsibility, the potential is virtually unlimited.
Many commentators on the generative AI revolution have been quick to conflate AI with automation. And that’s precisely what some of the first publicly available generative AI tools did – they automatically wrote things and automatically produced images. That’s not the highest calling for this technology, however. Operating in its highest form, generative AI is an assistant that can help humans surface and scrutinize more information than ever before possible, allowing uniquely human expertise and creativity to flourish.
You May Also Like
