A Benchmark Study: Patent Invalidity Search with Leegal AI

Leegal AI offers a full suite of AI-assisted IP tools. You can use them today at www.leegal.ai. To learn more about patent invalidity search, check our blog titled ” What is a Patent Invalidity Search?“.

Summary

In this benchmark study, we compare Leegal AI’s patent invalidity search tool against both (1) Google Patent’s semantic similarity search, and (2) human’s keyword-based search using references from 48 concluded Inter Partes Review (IPR) cases.

The results show that Leegal AI is at least twice as effective as the other two approaches in identifying relevant references. It is a fast, powerful, and easy-to-use tool for patent professionals and patent defendants looking to invalidate asserted patents.

Patent Invalidity Search

A patent invalidity search is a critical component of patent litigation and due diligence, helping to determine whether a granted patent can withstand legal challenges.

Traditional search methods—whether manual keyword searches or semantic search tools—often struggle to efficiently identify the most relevant prior art.

At Leegal AI, we leverage cutting-edge AI technology to enhance the accuracy and efficiency of invalidity searches. You can generate 3 patent invalidity search report for free at www.leegal.ai.

Understanding the Context

An Inter Partes Review is a rigorous, mini patent litigation held at the US Patent Office. The petitioner (patent defendant) submits a list of references and claim charts in an attempt to invalidate asserted patents. The references submitted by the petitioners have undergone extensive professional evaluation.

In this study, we used references  submitted in Inter Partes Review (IPR) cases concluded between 09/27/2024 and 11/14/2024 as “ground truth”. We then evaluate how well each method can find these ground truths. See Appendix for a list of these IPR cases.

For this chosen period, we originally collected a total of 50 cases. However, we had to discard two cases eventually: one case’s primary reference was non-patent literature, and another case—based on a chemical formula—proved problematic for both our system and the commercial tool.  As a result, we used the remaining 48 IPR cases in our study.

While we acknowledge that there may be many other effective references beyond those approved in the specific IPR case, we use these as our “ground truth” because they represent hundreds of hours of expert analysis and professional judgment.

Study Objectives

Our goal was to compare the performance of three distinct approaches in performing patent invalidity search, and how well each finds ground truths defined in the previous section.

1. Leegal AI’s Invalidity Search Tool: Leegal.ai’s invalidity search tool automatically identifies relevant references (with options to return top 10, 20, 50, or 100 references).
2. Google Patent’s Semantic Similarity Search: Google Patent uses semantic search techniques to identify relevant references. It is used to return top 10, 20, 50, and 100 references based on the target patent claim.
3. Human’s Keyword-based Search: A patent professional crafts search strings based on the patent claims to retrieve references from Google Patents. We use the top 10, 20, 50, and 100 references based on Google Patent’s return result. To save cost, we asked one of our own team members, who is an experienced patent professional, to craft these search strings. If you are an experienced searcher, we also challenge you to do this test yourself and compare with our result.

Note 1: Leegal.ai also automatically generates claim chart based on the identified references with reasonings. This feature is not available in (2), and will require significant time investment in (3). So we did not compare the claim chars in this study, but focused on the identified references instead.

Note 2: in a real search scenario, an experienced searcher or patent lawyer will look at all the references and probably pick the best 2 or 3 to use in the IPR. However, please note this is not part of our study: our study is to find out how ‘successful’ (as defined in the next section) the returned results are by each method.

Study Design and Methodology

Each method was tested in scenarios where it returned 10, 20, 50, and 100 references. This approach allows us to observe the trade-off between the number of references provided and the corresponding success rate.

• Dataset: 48 Inter Partes Review (IPR) cases from the US Patent Office (USPTO) where the challenged patent was invalidated.
• Ground Truth: The references and corresponding text excerpts from the official IPR invalidity claim charts, which represent the extensive, professionally curated work.
• Comparison: Leegal.ai’s output (i.e., up to 100 references per search) was compared with the actually used references from the IPR claim charts.
• Success Rate: Defined as the percentage of cases where the tool correctly identified at least one reference used in the IPR decision.

Detailed Findings:

In all scenarios (10, 20, 50, and 100-references), Leegal AI significantly beat the other two methods in terms of success rate. For example:

• in the 100-reference scenario, Leegal.ai identified at least one IPR-used reference in 33 out of 48 cases—an overall success rate of approximately 69%.
• In the 50-reference scenario: Leegal.ai achieved a success rate of 54% (26/48 cases).

The table shows the success rate (i.e., the percentage of cases with at least one matching IPR reference) for each method at different reference set sizes.

Why These Results Matter

These benchmark results highlight the tangible benefits of using Leegal.ai:

• Efficiency: Automating the search process and highlighting relevant text chunks dramatically reduces the manual effort traditionally required.
• Accuracy: Our AI tool’s ability to consistently identify relevant references ensures higher reliability compared to both a generic commercial tool and manual searches.
• User Experience: With an intuitive interface and detailed output (e.g., claim charts), Leegal.ai simplifies the otherwise complex task of patent invalidity searching.

By combining advanced AI with user-friendly design, Leegal.ai is transforming the patent invalidity search process—making it an indispensable resource for patent lawyers, examiners, and R&D teams.

Limitations and Future Directions

While the results are promising, our study is based on 48 IPR cases. Future research will:

• Include detailed error analysis for cases where the tool did not identify the approved references, which will help us better understand and address the nuances of certain patent claims.
• Expand the dataset to further validate these findings.
• Refine additional metrics, such as precision and recall.

Appendix: Leegal Generated Invalidity Search Results