How to Evaluate AI Contract Review Without Inventing an Accuracy Score

AI-generated editorial illustration. People depicted are fictional.

Evaluate AI contract review against a defined task, an authorized document packet and a written answer guide; retain first-attempt errors and the effort needed to check and correct them. There is no useful universal accuracy score for AI contract review. Finding a provision, summarizing its meaning and deciding whether it meets a client’s position are different tasks. A percentage becomes informative only when its denominator, test documents and definition of a correct answer are visible.

Specify the job before scoring it

Write the task as an observable deliverable. For extraction, require the provision and its location. For a summary, define which qualifications must survive. For a playbook check, name the approved position and acceptable alternatives. Avoid asking a reviewer to grade an undefined impression of intelligence.

NIST AI 600-1 supports comparison with known reference data and empirical evaluation of capability claims. Our recommendation is to translate that principle into a written answer guide before exposing the evaluation set to a vendor. Keep genuine disagreements visible rather than forcing every commercial judgment into a single answer.

Build a representative document set

Sample the agreements the proposed users will actually handle, with permission to use them. Include routine paper and difficult cases, such as an amendment changing a defined term or a schedule qualifying an obligation. Preserve the relationships among documents. A model evaluated on isolated clauses may never encounter the context it must handle in service.

Hold some documents back from configuration work. Record the tool version, prompts, playbook and processing settings. When these change, the result describes a different setup. Decide whether to repeat the relevant checks instead of carrying an old score forward as a permanent property of the product.

The Magesh and coauthors study evaluated legal research systems and found that retrieval did not eliminate erroneous output. Its May 2024 version is historical research on different tasks. Its findings cannot supply accuracy rates for today’s contract products. The useful lesson for this evaluation is methodological: scrutinize claims against a defined task.

To turn this buying framework into a vendor demonstration, use our AI contract review tool tests with clause pairs and a results log.

Separate mistakes with different consequences

For extraction, record:

• Correct identifications: provisions found accurately.
• Missed provisions: relevant language the tool did not find.
• Incorrect flags: items the tool wrongly identified.

Run the contract redlining software test if document fidelity is part of the purchase.

Report recall and precision separately, with their underlying counts. For a proposed revision, ask whether it preserves the intended commercial position and fits the rest of the agreement. A fluent sentence can fail both checks.

Record severe misses separately from nuisance alerts. State who classified severity and why. A combined score can make a system that misses a critical qualification look attractive because it performs many easy identifications correctly. Do not let the abundant easy cases erase the consequential ones.

Report abstentions too. A tool that declines to answer may be safer on an unfamiliar document, but the work still needs assignment. Treat that behavior as a workflow event with a review cost, not automatically as either success or failure.

Include supervision and data handling

ABA Formal Opinion 512 addresses competence and confidentiality when lawyers use generative AI. It does not approve a particular product. Review the applicable professional rules and the proposed data terms before uploading client material.

Our practical test should include the reviewer’s experience: can they reach the cited clause, inspect the full agreement and reject a suggestion without losing their work? Ask security and procurement colleagues to verify access, retention and deletion arrangements. A promising clause detector can still be unsuitable for the intended documents.

Make the buying decision reproducible

Compare the complete assisted workflow with the current process, including checking and correction. Keep elapsed time and professional effort distinct. Record examples of failure alongside aggregate results so that a future owner understands the limit behind the score.

Approve specific tasks, document exclusions and name a person who can suspend the use case. Retest when a meaningful change affects the evaluated workflow. The resulting decision may be that a tool is useful for locating language but requires more work before suggesting revisions. That is a clear purchasing conclusion without a fabricated league table.

A worked evaluation, with the denominator visible

Synthetic teaching example, not a vendor test: an answer guide identifies twelve relevant notice provisions in an authorized document set. A fictional output finds nine correctly, misses three and flags two irrelevant provisions. Count each defined item once. A correct citation attached to the wrong substantive answer is not automatically a correct identification.

Using the definitions in the scikit-learn precision and recall documentation, recall is correct findings divided by all relevant items: 9 ÷ (9 + 3) = 75%. Precision is correct findings divided by all positive findings: 9 ÷ (9 + 2) = 81.8%, rounded to one decimal. These are separate extraction measures, not an overall legal-quality score or an acceptance threshold. If a denominator is zero, disclose the undefined ratio and any software convention; do not label it perfect performance.

Now review the missed items individually. Suppose one missed qualification changes the notice recipient, while the others are routine identifications. Keep that classification and its reason beside the result. The overall recall does not establish whether the remaining risk is acceptable for the proposed use. A small, related test set does not establish general performance across new contracts.

A completed decision record for a narrow use case

This fictional decision record demonstrates the format. It does not establish that an actual tool met the conditions, that twelve provisions are a sufficient test, or that the workflow was legally approved.

EVALUATION RECORD: NOTICE EXTRACTION / TEACHING EXAMPLE
Proposed job: identify renewal-notice text and its location.
Excluded job: approve legal meaning, propose changes or send notices.
Input: synthetic document packet, answer-guide-v1.
Relevant items in the guide:12.
Fictional first-attempt results:9 correct,3 missed,2 incorrect flags.
Recall:9/12=75%. Precision:9/11=81.8% rounded.
Consequential issue: one missed recipient qualification, under this assumed output.
Reviewer action: inspect the complete notice provision and related documents; keep the miss in the results record.
Decision: no unattended reliance on extracted notice data. A limited assisted workflow remains subject to separate review and representative testing.
Evidence needed before deployment: authorized held-out packet, actual outputs, correction effort, data terms and approved operating limits.
Owner: named business decision-maker, with legal and security reviewers assigned in the real record.

This record complements the exact inputs in our paired-tests companion. Use those fixtures to expose a specific failure mode, then use separate held-out documents to examine the intended workflow. Repeatedly fixing a prompt on the same fixture may improve that fixture without establishing a reliable buying decision.

Compare the whole workflow with the current process

Time preparation, running, checking and correcting separately. Keep operator effort distinct from elapsed waiting. In a second, independent synthetic timing example, a manual review takes 35 minutes. The assisted route uses 10 minutes preparing the packet, 8 operating the tool, 15 checking and 7 correcting: 40 minutes of effort, five more than the assumed manual route. This is arithmetic, not observed efficiency or a price calculation.

The NIST AI RMF Core describes task context, documented evaluation and ongoing monitoring, rather than treating one score as permanent assurance. The Measure Playbook, measures 2.1 and 2.3, offers voluntary suggestions on documenting test sets and testing in relevant conditions. Our record is an editorial application of those principles, not a NIST-approved contract test or mandatory legal procedure.

Copyable buying-pilot brief

AI CONTRACT REVIEW PILOT
Business decision: [specific workflow and intended user].
Job to evaluate: [extraction, summary, playbook check or revision].
Excluded jobs and documents: [limits].
Authorized input packet: [versions, amendments and schedules].
Answer guide and reviewer: [criteria, locations and unresolved judgments].
Configuration: [tool/build, prompts, playbook and settings].
Test separation: [configuration material and held-out material].
First-attempt result: [correct, missed, wrong, abstained; counts and denominators].
Severity: [specific error, reason and reviewer].
Retry record: [instruction, change and resulting answer; retain original].
Workflow effort: [preparation, operation, checking, correction and handover].
Data controls: [approved route, access, retention and deletion review].
Decision: [approved task, limits, remaining issues or rejected use].
Owner and change trigger: [who may suspend use and when to recheck].

Pre-purchase checklist

• Choose the actual job before requesting an accuracy claim.
• Use authorized documents and preserve relevant amendments and schedules.
• Agree the expected answers, material qualifications and uncertainty treatment before the run.
• Separate configuration material from the held-out evaluation packet.
• Retain first-attempt output, source locations, missed items and incorrect flags.
• Record abstentions and assisted retries separately; do not erase the original failure.
• Measure checking and correction effort alongside operating time.
• Review data handling, supervision and the applicable professional rules with the responsible people.
• Approve a defined use and exclusions, then record monitoring and suspension responsibilities.

Frequently asked questions

What is a good accuracy percentage for AI contract review?

There is no universal pass percentage in this guide. Define the task, relevant documents, correct answer and consequences of error. A high aggregate score can coexist with a consequential miss. Decide acceptance criteria before running the actual pilot.

Does recall measure whether suggested wording is legally sound?

No. In the extraction example it measures how many relevant items were found. A revision needs separate review of the intended commercial position, qualifications, governing law and remaining document set.

How many contracts should we test?

The illustrative twelve-item example is not a recommended sample size. Choose coverage and an evaluation design with appropriate expertise for the intended workload and risk. Disclose small samples and gaps rather than claiming general reliability.

Should a corrected follow-up count as a first-attempt pass?

No in the proposed protocol. Retain the original result and record the assisted retry separately, including the instruction and reviewer effort. Both may matter to the buying decision.

Can a legal-research hallucination study rank contract-review tools?

No. Different tasks, products, versions and document sets require their own evaluation. The historical study cited here explains why claims need empirical scrutiny; its rates are not imported into a contract-tool scorecard.

Does NIST or the ABA approve a product through these sources?

No. The NIST guidance is voluntary risk-management material; the ABA opinion addresses professional duties in its stated scope. Neither is a certification of the fictional pilot or a named product.

When should a pilot be repeated?

Review meaningful changes to the model, instructions, playbook, document mix or operating workflow. Keep the original configuration and evidence so that a later result is not silently treated as the same evaluation.

Editorial note

The prototype claimed that Viaclause tested five products on a 200-clause agreement. No supporting test record was supplied. Those findings and percentages are withdrawn. What follows is a proposed evaluation method for US commercial contracting teams, not results from a product comparison.

For a connector-specific review, use the MyCase connector approval checklist to examine the proposed actions and their approval boundary.

The technology budget and AI strategy commentary explains why a purchase budget needs a separate operating decision and test.

Sources

1. Generative Artificial Intelligence Profile, NIST AI 600-1
   National Institute of Standards and Technology, 2024-07-26. Voluntary evaluation guidance; not legal certification or a product benchmark.
2. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
   Varun Magesh and coauthors, arXiv, 2024-05-30. Authors' research on then-current legal research tools; not a current contract-review scorecard.
3. Formal Opinion 512: Generative Artificial Intelligence Tools
   American Bar Association, 2024-07-29. Model Rules guidance; applicable jurisdictional rules still control.
4. scikit-learn precision and recall documentation. Definitions and zero-denominator notes inspected October 7, 2026; no package installation or product benchmark.
5. NIST AI RMF Core. Context, evaluation, documentation and monitoring sections inspected October 7; voluntary framework.
6. NIST Measure Playbook. Measures 2.1 and 2.3 inspected October 7; voluntary suggestions, not an endorsed worksheet.

Sources checked September 26 and October 7, 2026.

About the author
By Anirudh Menon, Viaclause editor. Sources checked September 26 and October 7, 2026. The counts, outputs, timing and decision record are synthetic teaching examples; no product comparison or human legal approval is claimed.