AI October 1, 2026

How to Evaluate AI Tools Using Evidence, Costs, and Outcomes

The demo went well. The vendor pasted a clean lease into the tool, and thirty seconds later every renewal clause, escalation rate and termination date sat in a tidy summary. Then your team tried it on a scanned 1990s ground lease with handwritten amendments, and the summary invented a clause that does not exist. To evaluate an AI tool properly, test it on four things in order: the quality of the evidence behind its claims, its total cost per task that passes your review, the risks it introduces, and the outcomes it produces in your actual workflow against your current process. Plan on two to four weeks for a small team. The difficulty is moderate: you need no engineering background, but you do need patience with spreadsheets and a willingness to say no to a good demo.

The short version

Score each tool on evidence, cost, risk and outcomes, weighted to your priorities. Build a test set from your own messy documents, define pass and fail before you look at any output, and have humans grade results blind. Compare tools on cost per successful task, not on subscription price or benchmark rank. Keep running the same test set after launch, because models, prompts and vendors change underneath you.

What you need before you start

A specific job the tool will do (for example, "triage inbound tenant emails" or "flag suspicious invoice requests")
30 to 100 real examples of that job, including ugly ones, with personal data removed
One or two people who do the job today and can grade output
Your current baseline: how long the task takes and how often it needs rework
A spreadsheet for the scorecard
Your security or IT lead's list of data that must never leave your systems

Step 1: Define the job and the pass line before you see any output

Write one sentence describing the task, then write what "pass" means in checkable terms. "Summarizes the lease" fails this test. "Extracts the renewal date, rent escalation and break clause, with zero invented terms" passes it.

That sentence turns into your golden dataset: a fixed set of inputs paired with known correct answers. Pull examples from real work, and deliberately include the hard cases. A thread on r/RecruitmentAgencies made the point well: sourcing tools look impressive on clean demo briefs and fall apart on the messy ones buyers actually send. I see the same thing with document tools. I call this the demo-to-desk gap, and the whole point of this step is to measure it.

Adoption pressure is real. Stanford HAI's AI Index Report 2026 found that 88% of surveyed organizations used AI in 2025, with 70% running generative AI in at least one business function. That pressure is exactly why a written pass line matters: it stops "everyone else is using it" from becoming your acceptance criterion.

Tip: Write your pass/fail thresholds and lock the file before any vendor trial begins. Thresholds set after you have seen output tend to drift toward whatever the favorite tool happened to do.

Step 2: Request the vendor's evidence package and grade it

Ask every vendor for the same documents: test-set composition and provenance, the evaluation date, the exact model version tested, uncertainty estimates, independent results, known failure cases, contamination controls, safety testing and their post-deployment monitoring process. Contamination means the test questions leaked into the model's training data, which inflates scores.

Benchmarks deserve skepticism for two reasons. They age fast: frontier models gained 30 percentage points on Humanity's Last Exam in a single year, according to the AI Index's technical performance chapter, so a score from last spring describes a different competitive field. Some benchmarks are also broken at the source. That same report counted invalid questions ranging from 2% of MMLU Math to a startling 42% of GSM8K.

Disclosure is improving but remains partial. The average transparency score of major foundation-model developers rose from 37% to 58% between October 2023 and May 2024 (Stanford HAI, 2025), which still leaves plenty unexplained. Sort everything you receive into three columns: vendor-reported, independently verified and your own results. Only the last two columns should move your score much. Our guide to spotting misleading AI model claims covers the common tricks, and the AI benchmark guide explains which scores mean anything.

A pitfall worth naming: "we rely on our security vendor's assessment" is still vendor-reported evidence. It is useful, but it belongs in column one.

Step 3: Price the full stack and calculate cost per successful task

List every cost line, not only the sticker price. Include subscriptions, API usage, evaluation-platform fees, human review time, integration, maintenance, monitoring, security, compliance, storage and the cost of switching away later.

Sticker prices vary widely. OpenAI lists ChatGPT Plus at $20 a month and Pro at $200 on its pricing page. On the API side, gpt-5.3-codex runs $3.50 per million input tokens and $28 per million output tokens. Anthropic charges $5 and $25 for Claude Opus 4.7, and $3 and $15 for Sonnet 4.6. Tokens are the chunks of text models bill by. If you add an evaluation platform, LangSmith's Developer plan is free for up to 5,000 traces a month, and Plus is $39 per seat.

Raw model prices keep collapsing. Stanford HAI tracked GPT-3.5-level performance falling from $20 to seven cents per million tokens between late 2022 and late 2024, a drop of more than 280-fold. Because of that drop, the model bill is rarely what sinks a project. Human review and rework are.

So divide total monthly cost by the number of tasks that passed review. Here is an illustrative example with made-up numbers, not real vendor data. Tool A costs $1,400 a month all-in and passes 312 of 400 tasks: $4.49 per successful task. Tool B costs $1,000 but passes only 200, because staff rewrite half its output: $5.00 per successful task. The cheaper tool is the more expensive one.

Build Your Evaluation Kit: Define one specific job, Collect 30–100 real examples, Include messy cases, with data removed, Rec

Step 4: Score risk with a framework instead of a gut feeling

Map each tool against the NIST AI Risk Management Framework, which groups risk work into four functions: govern, map, measure and manage. For a buyer, that becomes procurement questions. Where does our data go? Who can see prompts? What happens when the tool is wrong? Who gets alerted?

Treat safety, privacy, copyright, security, bias and source attribution as test dimensions with their own cases in the golden dataset. For hallucinations (confident, fabricated output), include prompts that tempt the tool to cite sources that do not exist, then verify every citation with a method like SIFT: stop, investigate the source, find better coverage, trace claims to the original. Incidents are climbing. The AI Incidents Database logged 233 reported cases in 2024, a 56.4% jump over 2023, as summarized in the 2025 AI Index.

AI agents, which take actions such as browsing, editing files or running code, need extra scrutiny because their environment is part of the system. Credentials, memory, network access and tool permissions all change results. Agents still failed roughly one attempt in three on structured benchmarks, per the 2026 report. For a property firm, an agent with inbox access and payment permissions is a fraud-exposure question, not a productivity question.

Regulation sets dates you can plan around. NIST published AI 800-3 on statistical evaluation on February 19, 2026, and AI 800-4 on monitoring deployed systems on March 6. EU enforcement powers for general-purpose AI obligations apply from August 2, 2026.

Warning: Never test an agentic tool with production credentials. Give it a sandbox account with fake data, and count any attempt to exceed its permissions as an automatic fail.

Step 5: Run a blinded pilot against your current process

Run each shortlisted tool on the golden dataset alongside your current process, and have graders score outputs without knowing which tool produced them. Track task-completion rate, error and rework rate, time saved, cost per successful task, human-escalation rate, user satisfaction and safety incidents.

LLM-as-a-judge, where one model grades another, speeds this up, but calibrate it first: have humans grade 30 items, compare against the judge, and only trust the judge where agreement is high. Automated graders that disagree with your people add cost without adding information.

Then fill in a weighted scorecard. My default weighting leans on outcomes, because that is where the money shows up:

Dimension (weight)	What you score	Tool A (example)	Tool B (example)
Evidence (20%)	Independent results, version, failure cases disclosed	3/5	4/5
Cost (20%)	Cost per successful task vs. baseline	4/5	3/5
Risk (25%)	Data handling, permissions, fabricated-source tests	4/5	2/5
Outcomes (35%)	Pass rate, rework, time saved in blinded pilot	4/5	3/5
Weighted total		3.80	2.95

If security is your main worry, move risk to 35% and accept a lower outcome weight. Write the weights down before scoring. For broader rollout planning, see how to implement AI in business.

Step 6: Keep evaluating after launch

Version your golden dataset and rerun it whenever the vendor ships a new model, you change prompts, connect new tools or update policy. Teams with engineering support can wire this into their CI/CD pipeline, so every change triggers the test suite automatically. Everyone else can schedule a monthly rerun.

Log incidents, watch for drift (gradual decline in quality), audit a sample of outputs each quarter and write rollback criteria in advance: for example, "pass rate below 70% for two consecutive weeks means we revert." Vendors are investing here too; Anthropic and Accenture each said they expect to put at least $1 billion into evaluation capacity over five years, though that is a company announcement, not verified spending.

Model updates arrive weekly, and a short daily read such as The Daily Brief, Verityadaily's morning newsletter on technology news, helps you notice when a vendor change should trigger a rerun.

Troubleshooting common evaluation failures

Why do our pilot results look much better than real use?

Your test set is too clean. Add the documents people complain about: scans, mixed languages, forwarded email chains. Generative AI reached about 53% population-level adoption within three years, per Stanford HAI, so your staff are probably already using tools informally, and their frustrations are a good source of hard cases.

What if graders disagree with each other constantly?

Your pass criteria are too vague. Rewrite them as checklists of observable facts, then regrade 20 items together until agreement improves.

Why did a tool that passed last month start failing?

The vendor likely updated the model. Ask for the version history, rerun your dataset and compare. This is why the model version belongs in your evidence file.

How do we choose when two tools score within a few points?

Pick the one with better risk scores and easier exit terms. Switching costs compound; small quality differences rarely do.

Next steps

Start with one task, not a platform decision. Build 30 examples this week, lock your pass criteria and send the evidence request to two vendors. Our guides on evaluating AI models without getting misled and AI tools that deliver measurable productivity gains go deeper on each stage.

Frequently asked questions

What criteria should I use to evaluate an AI tool?

Use four criteria: evidence quality, total cost, risk and measured outcomes. Evidence covers independent test results and disclosed failure cases. Cost means everything including human review, divided by tasks that pass. Risk covers privacy, security, bias and fabricated sources. Outcomes are what the tool achieves in a blinded pilot against your current process. Weight them to your priorities before scoring so the favorite tool does not set its own rules.

How do I check whether AI output is accurate?

Compare it against a golden dataset of real examples with known correct answers, graded by people who do the job. For anything with citations, verify every source directly using a method like SIFT, tracing each claim to its original. Include prompts designed to tempt the tool into inventing sources, and treat any fabricated citation as a hard fail rather than a minor error.

How does the NIST AI Risk Management Framework apply to buying a tool?

It gives you a structure for procurement questions. Its four functions (govern, map, measure, manage) translate into asking who owns the tool, what data it touches, how its errors are measured and what happens when something goes wrong. NIST's 2026 publications AI 800-3 and AI 800-4 add guidance on statistical uncertainty in evaluations and on monitoring systems after deployment.

Is a cheaper AI tool usually the better deal?

Not reliably. A low subscription or token price can hide high costs in integration, human adjudication and rework. Calculate cost per successful task: total monthly cost divided by outputs that passed review. A tool that costs less but fails half its tasks often ends up more expensive than a pricier tool that passes most of them.

How often should we re-evaluate an AI tool after deployment?

Rerun your test set whenever the vendor changes the model, or you change prompts, connected tools or policies, and at least monthly otherwise. Add quarterly output audits and incident logging. Set rollback thresholds before launch, such as reverting if the pass rate stays below a set level for two weeks, so the decision is not made under pressure.