Advice for assessing accuracy in AI tools

IN Partnership with

Legal professionals can keep up with the rapid pace of progress by identifying companies that demonstrate a willingness to invest in their people and their process, ultimately providing a system that remains on the cutting edge

IN JUNE 2018, the large language model (LLM) GPT 1 had 117 million parameters. By June 2020, it was sitting at 175 billion parameters. And when GPT 4 came out in March 2023, it had trillions of parameters. To put it mildly, artificial intelligence moves fast – so fast that, in most cases, as soon as a report is released evaluating any aspect of it, AI has already blown past the capabilities being assessed. While it’s incredible to see the technology constantly improve, in practice, the rapid changes make choosing an AI tool challenging. “I’ve never seen anything like it – the pace of progress is stunning,” says Helen Voudouris, director of online products at LexisNexis. “When legal professionals are looking at vendors, my advice is to ask them, what are you investing in? Because if they’re not keeping up, it’s going to impact you.”

LexisNexis Canada is committed to delivering information and workflow solutions to Canadian legal professionals to make their work lives easier. We are dedicated to our customers, to innovation, and to the rule of law. LexisNexis® Canada is part of LexisNexis® Legal & Professional, a leading global provider of information and technology solutions that enable professionals in legal, corporate, tax, government, academic, and non-profit organizations to make informed decisions and achieve better business outcomes. Part of RELX Group, LexisNexis Legal & Professional serves customers in more than 150 countries with 10,500 employees worldwide.

Find out more

“The quality and breadth of the underlying data is absolutely crucial”

Helen Voudouris, LexisNexis

When evaluating AI tools, the first thing to consider is whether the responses are accurate and of high quality. While accuracy is a nuanced term that leaves room for interpretation from user to user, Voudouris boils the assessment down to three checkpoints: whether an output contains information consonant with the input, whether it’s complete, and whether it’s authoritative. Did you get everything you asked for? Did the response demonstrate an understanding of what you requested? Is the output well supported, i.e., should more authoritative sources have been referenced or, on the other side of the coin, were some referenced that shouldn’t have been? LexisNexis takes a multi-model, multi-vendor approach to its selection of LLMs, assessing various options against different use cases to select the ones that perform the best and finetuning them for the complexity of the legal market. However, they don’t rely exclusively on the pre-existing knowledge of those LLMs; they ground them in exclusive LexisNexis content using RAG (retrieval augmented generation).

“The quality and breadth of the underlying data is absolutely crucial,” Voudouris notes, adding that setting Lexis+ AI up this way helps minimize hallucinations and enables hyperlinking to underlying content to create verifiable, citable authority where users can trace where and why a decision was picked up. “The tool must have access to a comprehensive legal corpus to generate reliable responses across domains. Whether it’s secondary materials, primary law citations, or articles, users

should apply the same thinking about the content used in generative AI responses as they would when selecting a traditional legal research solution.” Another important checkpoint should be whether there are people with legal training involved at every development stage to mitigate risks. Human intervention and involvement is still the best line of defence against AI misinformation or hallucinations, and at LexisNexis, internal subject matter experts rigorously oversee the process. From curating data, fine tuning, and informing choices of model construction, to deployment monitoring and output assessment, LexisNexis product and content teams are actively engaged throughout the development process. Teams also go through mandatory internal training to ensure everyone is on the same page regarding legal use cases and the ongoing needs of industry professionals. “It’s about accountability through human oversight,” Voudouris explains. “Another crucial and related factor is ethical considerations – they should play a front-and-centre role in developing an AI product, particularly for an industry that deals with sensitive information.” Ultimately, everything ties back to transparency and comes down to explainability: how does the solution work? “Without revealing the secret sauce, how are the solutions leveraging different technologies? What is being done with user interactions with the tool? How is security being maximized at every stage? These aspects should occur at the beginning when the vendor has first conceptualized the product.”

One example of AI’s powerful capabilities is the use of natural language processing and machine learning to understand context and nuance. Voudouris sometimes finds it “surprising” how well these tools can understand the intent of legal queries via semantic analysis, which assesses the grammatical format of sentences. For example, “current” could be referring to a current in the ocean or to time as in the currency of a document, so AI would determine if the word was used as a noun or an adjective to intuit context. “This is a crucial task undertaken by LLMs and, combined with the oversight of our legal subject matter experts, engineers, and data scientists, we get some high-quality responses – but these technologies are still evolving at a rapid pace,” she stresses, adding that ultimately the tools are just that:

“It’s about accountability through human oversight. Another crucial and related factor is ethical considerations – they should play a front-and-centre role in developing an AI product, particularly for an industry that deals with sensitive information”

Helen Voudouris, LexisNexis

something to be wielded by and assist a skilled user, not replace them. Voudouris argues they should be viewed as AI assistants that enhance efficiency and research capabilities but don’t replace a legal professional’s expertise. Think of a solution’s output as an expedited foundational draft pointing you in the right direction.

The technology is continuously evolving and is evolving at a pace unseen before. And so, when evaluating different gen AI vendors, a strong foundation upon which they have built their product and service is key. It’s important that they can confidently demonstrate to you that they’re investing in this – that they’ve historically done so and will continue to do so. It’s about how the company plans to ensure now and into the future that customers are getting results they can rely on. “When looking at a legal gen AI provider, it’s their continued investment in legal subject matter experts, in new LLMs, in evolving technologies and strategic integrations that counts.”

Accuracy, quality, and transparency

Generative AI technology is quickly evolving

Accuracy, quality and transparency

Generative AI technology is quickly evolving

Published Oct 22, 2024

InHouse

Resources

Events

Rankings

Practice Areas

News

Advertise About us Contact us Privacy Terms of Use Submit your move

Canadian Lawyer subscription Canadian Lawyer InHouse subscription Newsletter Digital editions Authors External contributors Editorial board RSS

Law Times Canadian Law List Lexpert® Rising Stars Canadian Law Awards

The incredible pace of AI

June 2018: the large language model (LLM) GPT had 117 million parameters

June 2020: 175 billion parameters

March 2023: trillions of parameters

News

Practice Areas

Rankings

Events

Inhouse

Resources

Advertise About us Contact us Privacy Terms of Use Submit your move

Canadian Lawyer subscription Canadian Lawyer InHouse subscription Newsletter Digital editions Authors External contributors Editorial board RSS

News

Practice Areas

Rankings

Events

Inhouse

Resources

Accuracy, quality, and transparency

Advertise About us Contact us Privacy Terms of Use Submit your move

Canadian Lawyer subscription Canadian Lawyer InHouse subscription Newsletter Digital editions Authors External contributors Editorial board RSS