There’s a small experiment I really want you to try.
Think of something Claude or ChatGPT did so well that you wanted to trust it with more work. Now give it the exact same job again, starting fresh. Then do it one more time.
You might be surprised by what happens. The instructions are identical, the model is the same, yet the way it works and the results can be surprisingly different.
And here is the strange part. All three attempts can look perfectly fine until you actually check what happened.
I’ve started thinking this little experiment might be more useful than trying another expensive AI model.
There is one number worth understanding
Anthropic gives an interesting example in its research on AI agents.
Suppose an agent has a 75% chance of completing a task correctly. Sounds reasonably good.
But if you need it to succeed three times in a row, the chance falls to about 42%, assuming each attempt is independent.
That is quite a difference.
And this matters now because AI is moving into an entirely different kind of work. Claude Opus 5.5, GPT-6 Astra and other advanced models can use tools, work across applications, change files, build software and make decisions along the way.
The question is becoming less about how intelligent these models are and more about how much of their work we can actually trust.
This is where Agent Evals comes in.
It is a surprisingly simple way to test an AI agent, find exactly where it goes wrong, and see whether a change genuinely makes it better.
There is even a new AI model called Jev, built specifically for making fast judgments, that can help with some of these checks at a tiny cost.
And you don’t need to be a developer to start using this approach.
Inside the full guide, I’ll show you how to test your own Claude, Codex, Cowork and automation workflows, build a simple evaluation system, use Jev to check AI work, measure reliability, and make agents improve through repeated testing. We’ll use practical prompts, useful tools and a few simple setups you can follow yourself.
Read more
