How to evaluate an AI tool before you pay for it
By Anwar Benhamada · August 6, 2026
Every AI tool demo works. That’s what a demo is — the inputs were chosen after the fact, by someone who knows exactly where the thing falls over.
The gap between “works in the demo” and “works on my work” is where the money gets wasted. Here’s a two-hour evaluation that closes it.
Step 1: build your task set first, before you look at any tool
This is the step everyone skips, and skipping it is why evaluations go wrong.
Write down ten real tasks from your actual backlog — not representative examples, actual ones you have already done or need to do. Include:
- Three that are genuinely easy
- Five that are typical
- Two that you expect to fail
Those last two matter most. A tool that handles your easy and typical cases is common. What separates tools is how they behave at the edge: does it fail loudly, or does it produce something confident and wrong?
Fix this list before you sign up for anything. Once you’ve seen a tool’s output, you will unconsciously reshape your tasks to suit it.
Step 2: run the set three times
Once tells you nothing. These systems are non-deterministic, and the difference between a tool that’s good and a tool that’s sometimes good is invisible in a single run.
Three runs of ten tasks is thirty data points, which is enough to see:
- Variance. Same input, different quality? That’s a tool you can’t build a process on.
- Failure mode. When it fails, does it say so, or hand you something wrong that looks right? The second kind costs more than it saves.
- Speed under real inputs, not the demo’s short ones.
Step 3: measure three things, and refuse to score the fourth
Track:
| Metric | How |
|---|---|
| Median run time | Wall clock, end to end. Not the vendor’s number. |
| Cost per run | Actual tokens/credits consumed × list price |
| Completion rate | Finished without error, refusal, or timeout |
And then don’t give output quality a number. You’ll be tempted — an 8/10 feels rigorous. It isn’t; it’s a feeling with a digit attached, and averaging those digits produces a confident number that means nothing.
Instead, for each task write one sentence: what would I have to fix before shipping this? Ten sentences tell you more than a spreadsheet of invented scores, because they’re specific and you can check them later.
Step 4: the question that actually decides it
How long does it take me to fix the output, versus doing it from scratch?
If a tool produces 80%-correct work but the 20% takes longer to find and repair than doing it yourself, the tool has negative value regardless of its price. This is especially true for code and for anything factual, where verification is the expensive part and generation is cheap.
Tools that produce obviously wrong output are safer than tools that produce subtly wrong output. Weight your evaluation accordingly.
Step 5: check the exit before you enter
Before paying for a year:
- Can you export your data — prompts, outputs, configuration — in a portable format?
- Does the workflow lock you in, or would switching mean re-doing setup?
- What happens to your saved work if you downgrade to free?
The annual discount is usually 20%. Being stuck with the wrong tool for twelve months costs more than that, so pay monthly until you have three runs’ worth of evidence.
The short version
Write ten real tasks including two you expect to fail. Run them three times. Measure speed, cost, and completion — and describe quality in sentences rather than scoring it. Then ask what it costs you to fix the output. Two hours, and it beats any roundup, including this site’s.
If you’re evaluating coding tools specifically, the shortlist and the criteria are here: best AI coding assistant for a real codebase →