CCBJ: Let’s start out with a little bit about yourself.
Diana Didia: I am the chief innovation and information officer for the American Arbitration Association. In this role, I’m responsible for not just software development and product management but also ideation and idea pipeline management, which I think is fabulous because I get to see things through from beginning to end. I have been with the AAA for about 13 years, but I’ve been in the technology space for more than 30 years, including working as a programmer, business analyst and project manager.
Has your role at AAA changed since the emergence of new technologies such as artificial intelligence (AI)? Have you seen any unexpected opportunities emerging because of these advancements?
The technological advancements we have adopted since I’ve been at AAA have traditionally focused on back-office processes. We were looking for efficiencies, enhanced reliability and improved cybersecurity protections. And moving to the cloud, another significant initiative for us, has given us a lot more flexibility and scalability, allowing us to move quickly to provide innovations and help drive business objectives. While impactful, these advancements were not necessarily transformative. However, the latest generation of AI has been a game changer.
Case management is a pipeline. A case comes in at the front end, and we’re moving it from A to B to C to D. We’d already taken advantage of different techniques to automate the process and assist the humans involved as much as possible. But we got to a point where we were trying to take unstructured information – for example, if a filing was a drag-and-drop of a long document – and make it structured. This requires a lot of human effort on our end to pull that information out of the document and type it into cells so that it can be structured and then acted on and moved forward in the pipeline.
So, about five years ago, we tried applying an early version of AI technology to this situation. It was called robotic process automation (RPA), and it had promise but turned out to be a money pit. We had to do tons of training to try to get the AI to understand what it was looking at and then pull out the right information. We were investing tons of time, energy and resources and were not really getting a significant return on investment.
In contrast, what’s fascinating about this new generation of artificial intelligence, known as generative AI or gen AI, is that it reads a document like a human, which is transformative. It can read the filing documents with hardly any training, and you tell the AI via a prompt to pull out the pieces of data you want and then take those pieces of information and put them in the right cells. Now you’re getting into really transforming the business: there are all sorts of use cases where you have had humans acting on things, but now suddenly, maybe you don’t need the human doing it – or certainly the human could have a copilot that would help make them more efficient. Maybe the AI gets it 90 percent of the way, saving X minutes or X hours, and then the human can take it from there. It just became so much easier to take advantage of automation.
Not that there aren’t still challenges. This is all brand new, so it’s cool, and you can see it working. But there’s still a lot of work to do to make sure it’s crossed the line in regards to accuracy, reliability and performance. Still, I think the impact is going to be real because it keeps getting better every day.
I’ve had all the support in the world to take advantage of the latest and greatest advancements around cybersecurity and cloud technology, but this feels different. This feels very business-transformative. And certainly, in the legal space, I think that’s what everyone’s been hoping for, given the fact that reading comprehension, the analysis of arguments, and those kinds of things are obviously foundational to the practice of law. This technology is going to do a lot of things very well around a lot of legal use cases.
How have customer needs influenced innovations within the AAA’s workflow?
As part of our innovation practice, we’ve trained all our staff on identifying needs or opportunities for innovation. They are really good at identifying pain points, whether for themselves, like a case management internal step, or an arbitrator’s or party’s process, and then thinking creatively about how to resolve that. We also have an idea submission portal where we collect ideas from all levels of staff, and those ideas are thoughtfully analyzed. The really interesting large-scale ideas that we think will be significantly impactful are quickly escalated all the way up to the senior vice president level, and we have conversations about their viability and aligning them with our strategy. We listen to the people who interact with our customers every day, and we’ve trained them to identify areas of frustration or difficulty that customers experience with a product or service and suggest improvements.
We also certainly have avenues to hear from customers directly, whether it’s surveys or meetings with our business development staff. In any forum where we are interacting with customers, we welcome their feedback, and because we have a formal innovation process, there’s a place to put it. The bar’s not high: You learn something, you have a thought, put it in there. We’re not trying to make you develop the entire use case and a whole business case at the front end. If you have a good idea, we want to hear it.
Then, as we go along, as part of the innovation process, we do a lot of pilots and beta testing and customer interviews. We conduct customer interviews to get feedback on a product that might be in flight or to learn whether there is a market for something we might charge for. We do a lot of surveying and inviting customers in to help us test things. And then, after we launch, we do more surveying, or we might do focus groups to get impressions or feedback about how something is being received by the market. So, we have quite a strong and formal product development process and competency across a few teams, mainly the innovation practice, but we also have go-to-market and other marketing functions, all of which have mechanisms to obtain feedback from customers.
As far as using AI to solve specific pain points, right now, it’s a lot about enhancing the promise of arbitration, which is about providing expert decision-making and a fast and cost-effective dispute resolution process. We always have an eye on time and cost for our customers, so we’re looking for areas to apply AI where we think things take a long time or where there’s a lot of back and forth between us and the case participants. While developing AI tools, we are careful not to overreach and to pick use cases that are most likely to provide reliability, accuracy, transparency and explainability.
Another key area of focus is making complex information more accessible to less sophisticated users. Conversational AI bots or chatbots can understand what a user wants from their message—even if it’s written in free text— and utilize natural language processing algorithms to analyze and summarize complex texts and answer user questions in plain English. Not only will chatbots be able to provide answers to frequently asked questions more efficiently, but also there is an access-to-justice component that’s important to the AAA and our President and CEO, Bridget McCormack.
The AAA has launched a series of AI tools, from ClauseBuilder® AI (Beta) to AAAi Panelist Search. Can you talk more specifically about how these tools work and the benefits they bring to clients?
ClauseBuilder AI was a great first AI pilot for us because we already had a popular legacy version but using it could be a little clunky and time-consuming. Say a lawyer would like assistance in writing a clause: There are a lot of options and nuances around what they may want to include in their arbitration clause, and the user would move through the clause-building process clicking radio buttons to indicate their desired options, and in no time could be 15 screens in. The legacy application was concatenating a long string of clause options, leveraging a library of perfected clause snippets. You want the clause to be well-written because the clause affects whether and how the case will proceed. We are the experts in that, so we had a library of perfected clauses, and when the user was clicking these radio buttons, the tool was pulling from that.
I think it was a staff member who said, “It would be so much easier if you could just say in a chat UX, ‘I would like an arbitration clause that has a tribunal, and that’s specific to this jurisdiction and all the other options I want,’ and have it just create the clause for me on one screen in one step, and maybe I iterate a little bit in plain English language, and it produces my perfect desired clause.”
Beyond the business value, a technical reason ClauseBuilder AI was a great first AI project for us was that we were able to compare it against and limit it to the very small data set of our perfected clause language. Using a small, well-defined data set gave us a clear benchmark for evaluation and reduced the risk of its providing incorrect information while still providing valuable insights into building these types of tools. Many companies make the mistake of immediately tackling large-scale, complex use cases, but this limited approach helped us learn critical lessons before expanding further.
For AAAi Panelist Search, we used AI to improve arbitrator selection. The historical way AAA case managers have done that is that an arbitrator submits their resume, and it gets coded in some way – e.g., this arbitrator gets 10 tags, such as they’re based in New York, they have expertise in healthcare, etc. The coding gets more granular, but the searching is basically limited to that. Then what often happens is that, even though you’re trying to be very surgical to find “the one,” your query returns 50 hits and thus begins the tedious process of reviewing every resume.
With AI, there’s a promise that we can be much more surgical, with our goal being to expedite the selection of an arbitrator and to give the customers a list with the best arbitrator choices we have to offer for this specific case. AI can more easily read the whole resume. You do not have to put a human in the middle to interpret and then pick from a limited set of codes, which is what leads to the return of so many resumes. And AI will also discern like a human. It can go through a pile of resumes and say, “Here are the top five.” It will make that step in our process so much more efficient.
In addition to these two products, we apply AI to other areas where case managers, arbitrators or parties face time-consuming or repetitive tasks. We’ve already done some things around automating a scheduling order where arbitrators can opt to have AAA take, with the parties’ permission, a Zoom transcription of a preliminary hearing where they’re working through the scheduling order and then use AI to generate a draft scheduling order for the arbitrator’s review. Some of our arbitrators have told us that they can save as much as an hour by getting a scheduling order that is almost complete and then editing it. So, while there’s still a human-in-the-loop to ensure accuracy, the AI-generated draft has given them a huge head start.
Can you talk about the metrics and benchmarks the AAA uses to evaluate the success of its AI tools, and the efforts you put into continuous improvement and relevance in this rapidly evolving field?
That’s an interesting question because at a lot of the tech seminars I go to, there’s a general perception that things are moving so quickly that you may start on something one way and the very next week, there’s a better way to do it. So, folks working in industries that are not expecting AI to be particularly disruptive may decide to slow-roll experimentation. But we’re not in that situation. We’re sprinting. We’re all in. It’s a critical strategic project for us.
How do we keep up with the latest in AI? Our primary product pipeline is filled with AI-enabled deliveries. Our current strategic plan has AI peppered all over it. We have a dedicated team, we’re hiring machine-learning experts and data scientists, and we talk about AI in every senior meeting. We’re in the early stages of creating a formal AI governance framework and policies and processes related to how we operationalize AI models so that there’s a feedback loop and we can be confident that a product we’ve tested for accuracy, fairness and transparency continues to meet quality standards.
We’re driven by a belief that if we don’t jump on this, we may not be a company in the future because someone else will be the first to market with an AI-native dispute resolution solution. We are constantly surveying what the latest advancements are and the vendors that are popping up because we don’t want to build something from scratch if someone else has already built it. At the moment, it’s all a very intensive organizational process, especially when deciding when something is ready to launch. But long-term it’s not sustainable to have too many people involved in the beta phase of every AI advancement.
The ability to effectively operationalize AI governance also depends on the related ecosystem of emerging technologies. On the one hand, you’ve got the OpenAIs of the world that are creating the models, and on the other, there’s a whole slew of startups creating various technologies to help monitor the accuracy of models, the uptime and cost of running them, etc., and we are surveying all that as well. It really feels like riding the wave. There isn’t one right way to do all this, and anyone who says otherwise is not really in the know. Lastly, governance is going to evolve along with the advancements, and it will probably take a few years before we can identify best-in-class tools or not have a bunch of people babysitting something.
What advice would you give to other legal companies aiming to introduce AI-driven tools?
First, you need your IT team to build products, so surfacing potential AI use cases is critical. Purchasing enterprise ChatGPT licenses is important because you want your staff to be fluent in generative AI, and it’s also important to have a way to collect their ideas. You want to surface the thoughts you crowdsource from within your organization as to the best places to employ this technology, and if they’re playing with it themselves, even if it’s just to write memos and such, they’re more likely to have light bulbs go off as to where it could benefit their customer.
That is out in the field, but as to your IT team working with these models and building products, my main advice is: Don’t try to boil the ocean. Start with narrow use cases and small data sets, like creating rule sets for a chatbot that’s going to read documents and answer questions (like FAQs) for your customers. That’s basic but important because things are moving quickly, and it’s a whole new discipline for technologists. So, on the one hand, you’ve got Silicon Valley inventing all this stuff, and on the other, your average programmers whose job is to learn how to take advantage of all the techniques needed to leverage generative AI and AI in your product.
If your IT team isn’t doing this, they need to get cracking because everyone else is going to be doing it. And with new versions coming out fast, it’s going to be harder and harder to play catchup. Also, you’d rather home-grow your experts because it’s going to get very expensive to hire people off the street.
For our part, we’re looking at digitizing arbitration more fully. These tools are starting to look like they could reason out an award. We’re not suggesting that AI will replace arbitrators, but certainly, an AI copilot could assist with early case evaluation, maybe even driving parties into mediation or settlement by giving them a better sense of the merits of their claim. That’s where you’re getting into the actual arbitral process, and we are all over that. We’re trying to envision the future state of arbitration in parallel with rehabbing and embedding AI into our main case management system. So, it’s like running the existing business while trying to build a new model. We’re up for it, and we’re investing in it, which will make for a busy and very exciting 2025.