How to Tell a Useful AI Tool from an Expensive One

Every AI product demos well. That is the problem. A demo is a controlled environment with a cooperative example, and almost anything looks capable there. The gap between a tool that quietly removes two hours a week and one that becomes an expensive browser tab only shows up after the trial ends.

Five tests surface that gap early, and none of them requires a technical background.

Test one: does it fail visibly?

The single most useful property of an AI tool is that you can tell when it is wrong. A system that flags low confidence, shows its sources, or returns “I could not find this” is one you can build a routine around. A system that answers every question in the same assured tone forces you to check everything, and checking everything costs more time than doing the task yourself.

Before committing, feed the tool something you already know the answer to, ideally something slightly obscure. Then feed it something unanswerable. What it does with the second input tells you more than what it does with the first.

Test two: where does the work land?

Useful tools finish a task. Impressive tools produce material that someone then has to finish.

A transcription tool that writes a usable summary into the document you were already working in has removed work. A tool that generates a draft you must rewrite, in a window you must copy out of, has moved work rather than removed it. The question to ask during a trial is simple: after using this, is there a step I no longer do, or a step I now do differently?

If every answer is “differently”, the saving is probably imaginary.

Test three: what happens on a bad day?

Try the tool on your messiest input — the scanned PDF, the recording with three people talking over each other, the spreadsheet with merged cells and a header in the wrong row. Clean data is not where your week goes, so a score on clean data tells you very little.

Keep a folder of four or five genuinely awful files and run every trial against it. Comparing two products becomes a half-hour job instead of a committee meeting.

Test four: does the pricing match how you would use it?

Per-seat pricing punishes you for giving access to occasional users, which is the group most likely to pick a tool up for a messy one-off job. Usage-based pricing punishes you for the long, repetitive tasks that are the best candidates for automation in the first place. Neither model is wrong, but one of them will fit your pattern better than the other, and the mismatch rarely shows up until the second invoice.

Work out your realistic monthly volume before the sales call, not after it.

Test five: where does your data go, and can you leave?

Ask whether your inputs are used to train the model, how long they are retained, in which country they are processed, and whether there is a data processing agreement you can actually read. A vendor who cannot answer those four questions in writing has answered them.

Then ask what you take with you if you stop paying. Documents in an open format, exports that preserve structure, and a plain record of what the tool produced are worth real money later. A product that stores your work in a shape only it can read has quietly raised its own price. The same logic applies to the model underneath: tools that let you swap the engine, or that keep working in a reduced form during a provider outage, survive a bad quarter better than ones built around a single supplier with no fallback.

What this looks like in practice

Take meeting notes, the most crowded category of all. Run the four worst recordings you have through three products. Test one tells you which of them invents an action item nobody said. Test two tells you which writes into your existing documents rather than its own. Test three is already covered by the recordings you chose. Test four is arithmetic once you know how many hours of audio you really generate. Test five is a single email to each vendor.

That is an afternoon, and it replaces six weeks of someone evaluating by feel. The same five questions work on consumer hardware, incidentally — our piece on wearable computing and just-in-time memory support applies them to devices rather than desks.

The unglamorous part

Most of the value in AI tools so far has come from narrow, boring applications: turning a meeting into notes, pulling fields out of invoices, finding the one paragraph in a 200-page contract. Those products rarely get the loud launches, because “extracts data from PDFs reliably” is not a headline.

The tools that disappoint tend to be the general ones that promise everything and therefore commit to nothing. If a product cannot tell you, in one sentence, which specific task it ends, run the five tests anyway. Expect them to confirm what the missing sentence already told you.