How we review AI agents
Every review uses the same scorecard so agents can be compared. Each review has two separate parts:
- Vendor claims: what the company says, quoted or closely paraphrased, with a link and date. We don't endorse these.
- Our test results: only what we observed running the agent ourselves, with dates. Empty until tested.
We test with test accounts and synthetic data, never real bank, client or tax data. A human approves anything that sends messages, moves money or cancels services.
The scorecard
| # | Measure | How we measure it |
|---|---|---|
| 1 | Task completion | Share of test runs where the agent finished the task end to end, with the definition of "finished" written before testing. |
| 2 | Time required | Wall-clock minutes per run, plus human minutes spent prompting, checking and fixing. |
| 3 | Money saved | Gross savings from the replaced expense (sourced list price or owner-documented amount), per month and per year. |
| 4 | Agent cost | Monthly price of the agent or the share used for this task, from the vendor page on a stated date, plus usage fees seen in testing. |
| 5 | Human interventions | Count of times a person had to step in per run (corrections, re-prompts, logins, approvals beyond the planned checkpoints). |
| 6 | Accuracy | Errors found against a known-correct answer key for the test data (e.g. field-level accuracy on synthetic receipts). |
| 7 | Reliability | Consistency across repeated runs on different days; failures, timeouts and silent errors. |
| 8 | Security | What access the agent needs, where data goes, credential handling, ability to limit permissions. Checked against vendor documentation and our setup. |
| 9 | Completes vs. suggests | Whether the agent actually does the task (e.g. cancels the subscription) or only tells you how. |
| 10 | Overall value | Net savings after agent cost and human time, weighed against risk. Written as a short verdict, never a made-up star rating. |
What we never do
- Publish scores, ratings, star ratings or testimonials we didn't produce in testing.
- Let affiliate commissions or sponsorships change scores or rankings.
- Use review structured data (rich-result markup) for agents we haven't tested.
See the review template for how a review page is laid out, and testing methodology for how projects are verified.
Questions
- Do vendors pay for reviews or better scores?
- No. Sponsorships and affiliate links never change a score or ranking. See Disclosures.
- Why are some scorecards empty?
- Because we haven't tested that agent yet. We show vendor claims separately and leave our results blank until we have run the tests.