How to Test Legal AI Agents on Real-World Cases with Harvey LAB

Standard benchmarks like MMLU or HumanEval do a great job of showing whether a model can write basic code or answer test questions. But try assigning such a model to analyze a folder of contracts for a merger and acquisition deal. Suddenly, general-purpose tests don't reflect the real complexity of the domain. Documents can span hundreds of pages, and a single contractual clause error can cost a company millions of dollars.

Harvey has released the Harvey LAB repository as open source. It's a framework and test dataset for evaluating AI agents on real legal tasks.

What's Inside the Repository

The repository contains two main components: an open dataset of legal tasks and an execution environment for running agents.

The data spans over two dozen legal practice areas. The dataset contains 16,671 tasks, and these aren't just text prompts—they're full-fledged scenarios. Each task includes source documents, agent instructions, and evaluation criteria.

The code is written in Python and released under the MIT license. At the time of this review, the project has around 666 stars on GitHub, but it's actively being developed.

How the Architecture Works

Testing revolves around a single cycle. First, the test environment loads the task and passes documents to the agent. Then the agent performs actions, calls tools, and generates a final response. At the end, the system runs evaluation algorithms.

The entire system is divided into several components:

• Task model that stores files, text instructions, context, and evaluation rubrics.
• Execution environment that runs agents in an isolated setting, intercepts tool calls, counts tokens, and logs actions.
• Adapters for connecting different models and agent frameworks.
• Reporting module that collects metrics and builds comparative dashboards for runs.

How Evaluation Works

Legal work is specific. Here, it's rare that 80% of a correct answer is considered passing if the remaining 20% contains distorted information about penalty sanctions.

Harvey LAB uses the All-Pass Rubric approach. A response only passes when the agent has fulfilled absolutely all mandatory conditions from the evaluation rubric. If even one detail is missed, the entire task is considered failed.

For complex text evaluations, an LLM judge is used. The authors built in rules that reduce judge bias and force it to strictly follow the given criteria rather than evaluate writing style.

Practical Example with M&A Audit

The project documentation includes a step-by-step guide using the example of a virtual data room audit for an M&A deal.

The scenario looks realistic. The agent receives a package of corporate documents, lease agreements, and client contracts. Its task is to identify change of control risks, inconsistencies in contract terms, and hidden obligations.

First, you run the task inspection command to view the files and evaluation rules. Then the agent itself runs, gaining access to document reading and search tools. After the work is complete, the framework compares the agent's conclusions with the reference rubric and produces a detailed report. The output is a dashboard with breakdowns showing whether the model handles searching for complex legal conditions.

Who the Project Is For

If you're building a RAG system or agent service for lawyers, this repository saves you from having to create synthetic tests. You get a ready-made set of real cases that immediately show where the model hallucinates and where it truly understands the context.

The project is also useful for those exploring LLM behavior in narrow domains. It's convenient for testing different prompting strategies, document chunking methods, and tool-calling approaches.

The downsides include the project's newness. Some documentation is still being completed, and running all 16,671 tasks requires a substantial API budget for frontier models. For debugging, it's more practical to run individual test subsets.

If you want to try it out, start with the guide at docs/tutorial.md. It walks through the entire process from environment setup to analyzing the final dashboard.