How to Test AI Contract Review Tools Before Buying
By Anirudh Menon, Viaclause editor
Test AI contract review tools with the same documents, rules and answer key. Count missed issues, false alarms, unsupported citations and full review time.
AI-generated editorial illustration by Viaclause. People and setting are fictional.
To test AI contract review tools, give each vendor the same documents, written review rules and questions, then compare its findings with an answer key prepared before the test. Count missed issues, false alarms, unsupported citations and the time a person spends checking the output. Start with the synthetic clause pairs below to see whether one changed fact changes the answer. Then test separate, representative contracts. Passing these examples demonstrates behavior on these examples, not reliability across your entire contract workload.
This is a practical test for procurement and legal teams evaluating clause review. For security, implementation and commercial due diligence around the purchase, use the broader AI contract review buying guide.
Define exactly what the tool has to find
“Review this contract” leaves too much room for competing interpretations of success. Write one rule that a reviewer can score, such as: flag a customer non-renewal notice requirement exceeding 60 days. Decide whose position matters, what counts as a finding and which source text must support it.
Finding a clause and deciding whether it breaches your policy are different tasks. The independent CUAD research paper describes expert-labelled contract text and evaluates extraction of relevant passages. It separates that task from contextual business risk assessment. A vendor’s extraction benchmark therefore does not, by itself, show that it will apply your negotiation rule correctly. That is the gap the examples here are designed to probe.
Independent testing also gives a reason to check whether citations support an answer. Magesh and colleagues’ study, first released in 2024 and published in 2025, tested legal research tools on legal queries and found both incorrect statements and claims unsupported by the cited authority. It evaluated research tools, not contract-review products; its error rates do not establish the accuracy of a tool you are buying today.
Everything in the examples is synthetic. The clause language, sample answers and numbers are teaching material, not a test of a named product or drafting language recommended for a live transaction. Have the person responsible for your real playbook approve its answer key before running a purchase pilot.
Four clause pairs you can copy into a vendor test
Run each version separately with the same instructions. Do not give the tool the answer key. Use neutral filenames, such as Case 01 and Case 02, so the filename does not reveal which version is supposed to fail. Keep model, product configuration and retrieval settings fixed where the product permits it.
Pair 1: does the tool notice a changed number?
Review rule: flag a customer non-renewal requirement only when it exceeds 60 calendar days before expiry. For this isolated test, exactly 60 days is permitted.
VERSION A Section 8. Renewal. This agreement renews for successive one-year terms unless Customer gives Vendor written notice of non-renewal at least 60 calendar days before the current term expires. VERSION B Section 8. Renewal. This agreement renews for successive one-year terms unless Customer gives Vendor written notice of non-renewal at least 90 calendar days before the current term expires.
Expected answer: A does not breach this rule; B does. For B, the output should identify Customer, 90 days and Section 8. “The agreement has an automatic renewal clause” is incomplete because it does not apply the stated threshold.
Pair 2: does it notice whose obligation changed?
Review rule: use the same customer-only threshold. A long notice obligation imposed only on Vendor is not a breach of this specific rule.
VERSION A Section 8. Customer may prevent renewal by giving Vendor written notice at least 90 calendar days before expiry. Vendor may prevent renewal by giving Customer written notice at least 30 calendar days before expiry. VERSION B Section 8. Customer may prevent renewal by giving Vendor written notice at least 30 calendar days before expiry. Vendor may prevent renewal by giving Customer written notice at least 90 calendar days before expiry.
Expected answer: A breaches the rule; B does not. A tool that flags both because it sees “90” has found a number without correctly attributing the duty. Score other negotiation concerns separately rather than quietly changing this test’s answer key.
Pair 3: does it use the supplied amendment?
Review rule: apply the customer-only threshold using the operative documents supplied in that run. In this exercise, assume the amendment is validly signed by both parties and in force.
VERSION A DOCUMENT SET Master agreement, Section 8: Customer must give written notice of non-renewal at least 90 calendar days before the current term ends. No amendment is supplied. VERSION B DOCUMENT SET The identical master agreement, plus this signed amendment: Amendment 1, Section 1. Section 8 of the master agreement is amended solely to replace Customer's 90-calendar-day non-renewal notice period with 30 calendar days. This amendment controls any conflict on that notice period. All other terms remain unchanged.
Expected answer: A breaches the rule on its supplied record; B does not. B’s explanation must cite the amendment as well as identify the term it changed. A citation to the master agreement alone does not show that the system reconciled the documents.
Pair 4: does missing information stop an unsupported conclusion?
Review rule: determine the customer notice period from the supplied documents. If a required schedule is missing, identify the missing input instead of inventing its terms.
VERSION A DOCUMENT SET Agreement, Section 8: Customer's notice period for non-renewal is the period stated in Schedule A, which forms part of this agreement. Schedule A, Item 3: Customer must give written non-renewal notice at least 90 calendar days before expiry. VERSION B DOCUMENT SET The identical Section 8 is supplied. Schedule A is not supplied.
Expected answer: A breaches the threshold rule. B is unresolved because Schedule A is missing. “No problematic notice period found” is not a passing answer for B. The tool must distinguish an absent obligation from an absent document.
Keep the answer key and results together
| Pair | Version A | Version B | Evidence needed for a pass |
|---|---|---|---|
| 1: Number | No breach: 60 days | Breach: 90 days | Correct number, party, rule and Section 8. |
| 2: Party | Breach: Customer has 90 days | No breach: Customer has 30 days | Customer and Vendor obligations kept separate. |
| 3: Amendment | Breach on supplied record | No breach: amended to 30 days | Controlling amendment identified and cited. |
| 4: Missing input | Breach: Schedule A says 90 days | Unresolved: Schedule A missing | No invented schedule or unsupported clearance. |
Use this proposed instruction for a tool that accepts a prompt. Where it uses configurable playbooks instead, encode the same rule and save the configuration. A prompt is a test instruction, not an access-control or confidentiality safeguard.
Review only the document set supplied for this case. Rule: flag a Customer non-renewal notice requirement that exceeds 60 calendar days before expiry. Exactly 60 days is permitted. Return: breach, no breach, or unresolved; the obligated party; the notice period; the exact supporting text and section; any controlling amendment; and any missing required document. Do not infer missing terms. Explain the result in two sentences.
| Run and setup | Expected result | Illustrative output | Score and action |
|---|---|---|---|
| Demo-001; Pair 1 B; fictional Tool X build 1; rule v1; first attempt; September 27, 2026. Sole input: Version B. | Breach. Customer must give 90 days’ notice. Source: Section 8. | “No breach: Customer can give notice 60 days before expiry.” Citation: Section 8. | Fail. False negative plus incorrect number. Section reference exists but does not support the answer. Save raw output; block this workflow until correction and retest. |
This row is illustrative, not observed vendor performance. Copy the following fields into your own spreadsheet. Preserve the first attempt even if a follow-up gets the answer right.
Run ID | Case ID | File versions | Tool/build | Configuration Prompt/rule version | Run time | Attempt number | Expected answer Raw answer location | Quoted evidence | Reviewer score Error type | Severity | Review minutes | Correction minutes Reviewer | Disagreement resolution | Retest result | Decision
Count missed risks and false alarms separately
For a larger labelled set, score each predefined rule opportunity once. A true positive is a correctly identified breach; a false negative is a missed breach; a false positive is a breach reported where the answer key says there is none. Deduplicate repeated flags for the same opportunity. Score citation support separately so an accidentally correct conclusion does not conceal invented evidence.
The standard precision and recall definitions give two useful measures: precision is correct flags divided by all flags; recall is found breaches divided by all known breaches. Neither tells you how serious an individual error was.
Separate synthetic calculation: suppose a test has 10 known breaches. The tool finds 8, misses 2 and adds 4 false alarms. Recall is 8 ÷ 10 = 80%. Precision is 8 ÷ 12 = 66.7%. The two numbers answer different questions: how much was missed, and how much checking was wasted.
Do not remove an unanswered, answerable case to improve recall: a missed known breach still counts as a miss. Track deliberate missing-document cases separately against their expected escalation answer. If no flags or no known breaches exist, the relevant ratio has a zero denominator; report it as undefined rather than quietly awarding 100%.
Move from demonstration to a purchase decision
These related clause pairs are diagnostic probes, not independent samples of your workload. Keep them separate from a held-out test set: contracts not used to tune prompts, train reviewers or help the vendor configure the tool. Include the document types, scan quality, amendments and recurring risks the proposed workflow will actually encounter.
Run the contract redlining software test separately if document editing and file export are also purchase requirements.
Before using real files, confirm that the intended upload route is approved for those documents. Removing names alone does not remove sensitive pricing or transaction facts. Use the contract-data and AI upload-route checklist to examine retention, training and access terms.
Choose pass criteria before seeing vendor results. For example, your team might hold deployment after any missed critical issue and separately set an acceptable false-alarm workload. Those are buyer decisions tied to the use case, not industry-wide accuracy standards. Report results by issue type and severity, not just one average.
Measure the complete task. In an invented timing example, manual review takes 35 minutes. AI preparation takes 4, generation 1, verification 22 and correction 10: 37 minutes total, two minutes slower. Compare reviews of equivalent scope and quality, rotate order to reduce familiarity effects, and record setup effort separately from recurring work. A quick response is not yet a time saving.
If the planned workflow also reads emails, test that separately. Our guide to assembling contract and email evidence helps define what documents belong in the record. Passing an isolated clause test does not validate mailbox retrieval or authority to send messages.
Write an acceptance record before the first run
The NIST AI RMF Core connects measurement with the intended context, documented methods and limitations. Its Measure Playbook, Measures 2.1 and 2.3, suggests appropriate tests and documentation of performance and generalizability limits. These are voluntary risk-management resources. They do not prescribe our clause pairs, endorse a buyer's threshold or certify a product that passes them.
For this narrow pilot, agree the opportunity being scored, the permitted document set, first-attempt handling and escalation before the vendor sees the cases. The responsible playbook owner resolves disputed labels. Retain disagreement rather than making the answer key fit a favored output.
Completed acceptance record for the fictional failure
ILLUSTRATIVE DECISION, NOT A PRODUCT RESULT Case: Demo-001, Pair 1 Version B, fictional Tool X build 1. Rule: Customer notice strictly greater than 60 days is a breach. Input: Version B only; no amendment or schedule required here. Expected: breach, Customer, 90 days, Section 8. First output: no breach, 60 days, Section 8. Scoring: one missed breach; wrong number recorded separately. Citation check: section exists, but does not support the answer. Disposition: hold this workflow pending correction and retest. Retry: not performed in this illustration; no success inferred. Preserve: supplied fixture, configuration, prompt and raw output. Limits: one invented row says nothing about a vendor error rate.
The incorrect number is an error attribute on this opportunity, not a second missed breach added to inflate a metric. The independent 10-breach calculation earlier is a separate teaching example; do not append this row to it.
Blank acceptance protocol
Task and responsible owner: Authorized documents and representative workload boundaries: Rule, obligated party and answer-key version: Opportunity definition and duplicate-handling rule: Missing-input cases and expected escalation behavior: Required source text, document identity and citation checks: First attempt, assisted retries and stopping conditions: Severity categories and pre-agreed disposition criteria: Product/build, configuration and document-processing settings: Output and reviewer-disagreement record locations: Independent held-out set, excluded cases and limitations: Equivalent manual comparison and total effort fields: Change requiring a retest, approval owner and next action:
Before accepting the pilot
- Verify the intended upload route before supplying real contracts.
- Freeze the answer key and neutral case filenames before running products.
- Check numbers, parties, amendments and required schedules against the supplied packet.
- Keep first attempts, assisted corrections, abstentions and source-support failures visible.
- Report each predefined opportunity once, with issue severity separately.
- Measure verification and correction effort, including unsuccessful cases.
- Keep the diagnostic pairs separate from held-out workload evaluation.
- Record the accepted scope, unresolved limits, responsible owner and retest triggers.
Frequently asked questions
How many contracts should we test?
Enough to cover the proposed workflow and its meaningful variations, with a separate held-out set. These eight small examples are a starting diagnostic. They cannot establish a production error rate. Record sample composition and limits instead of presenting a small passing sample as proof of broad reliability.
How should we measure AI contract review accuracy?
Set a requirement by task and consequence before testing. A high average can coexist with a missed critical provision. Require source support, track critical misses separately and decide what false-alarm workload reviewers can absorb.
Can we trust the tool’s confidence score?
Use confidence as something to evaluate against known outcomes, not as proof that an answer is correct. For this pilot, the supplied clause and answer key decide whether an output passes.
Should a second prompt count as a pass?
Record it as an assisted correction, preserving the original failure and extra time. If follow-up prompting is part of the intended workflow, define that workflow before comparing vendors. Do not give one product unrecorded extra attempts.
When should we repeat the test?
After a material model, product, prompt, playbook or document-processing change. Save the old outputs and rerun the same cases, then add fresh held-out cases so success is not limited to familiar examples.
Can we combine the four pairs into one production accuracy claim?
No. Versions in each pair are deliberately related. Keep their diagnostic results separate from independently selected, held-out workload cases. Report case composition, omissions, unresolved inputs and reviewer disagreements; these examples cannot establish a production error rate.
What if the product changes during a pilot?
Record the exact build and configuration for each run. Treat a material change as a new comparison condition, preserve the earlier outputs and rerun the affected cases. Do not silently combine results from different setups.
Sources
- Magesh and colleagues, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies, 2025; first released in 2024. Legal research evaluation, not a contract-review benchmark.
- Hendrycks and colleagues, CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review (2021), task definitions, annotation method and evaluation limits.
- scikit-learn documentation: precision, recall and support, metric definitions and undefined-result handling.
- NIST AI RMF Core, context, measurement and documentation passages inspected October 7, 2026. Voluntary framework, not certification of these tests.
- NIST Measure Playbook, Measures 2.1 and 2.3 inspected October 7, 2026. Suggestions on appropriate evaluation and documented performance limits; no linked software installed.
The test fixtures, answer key, proposed workflow and synthetic calculations are Viaclause editorial teaching material. They have not been validated through a real product trial.