👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice. For more: Lenny’s Jobs | Lennybot | Become an AI-Native Builder and other favorite AI/PM courses
P.S. Get a full free year of Cursor, Notion, Lovable, Replit, Wispr Flow, Linear, Factory, ElevenLabs, PostHog, Granola, Brain.fm, Waking Up, and more, by becoming an Insider subscriber (while supplies last). Learn more.
Evals have been coming up more and more in my conversations with podcast guests and PMs. And nearly half of the 𝕏25 awesome PM job openings I shared on socials last week ask for experience writing evals. This skill is only becoming more valuable. So I asked Hamel and Shreya to write an advanced sequel to their very popular “Building eval systems that improve your AI product” post from last year. Drawing from their work with over 50 AI companies, they’ve noticed that most teams jump straight to writing metrics—and end up measuring the wrong things. Below, they share the critical part of the process most teams skip, which steps you can (and cannot) automate, and a free plugin that lets a coding agent do most of the heavy lifting for you. Enjoy!
To go deeper, join their upcoming AI Evals for Engineers & PMs course and use the discount code LENNYSLIST at checkout to get 25% off.
By now, you’ve probably heard that evals are a defining skill for AI PMs. Mike Krieger, Anthropic’s former CPO and now head of Labs, has said that “if there’s one thing we can teach product people, it’s that writing evals is now probably the most important thing.” Garry Tan, the CEO of Y Combinator, shared that “evals are emerging as the real moat for AI startups.” A number of guests on Lenny’s Podcast have argued that “evals are the new PRDs,” and increasingly, leading companies have been talking about how investing in evals has paid off:
used evals to guide development of an AI workflow builder that was 2.2 times faster and 68% cheaper than the frontier-model system it replaced.Shopifydeveloped its Auto Balance routing performance with evals, resulting in much higher user satisfaction while reducing costs by 41%.Cursorincreased its automatic receipt collection product’s accuracy from 35% to 83% after investing in evals.Ramprebuilt its AI contract reviewer with evals, nearly doubling the product’s internal quality score.Harvey
Rippling, Glean, Abridge, ElevenLabs, and Robinhood have also shared how they’ve been using evals to systematically make their AI products better.
AI products are easy to change but hard to predict. A prompt, model, or code change can improve one behavior while breaking another. Evals turn your judgment about what “good” looks like into repeatable tests your team can run before shipping. Production errors flagged by evals can become additional test cases that improve your AI, creating an advantage that compounds over time.
And now that AI can produce changes faster than people can review them, evals help teams ship quickly by automatically checking if a product still works as intended.
In our prior post, we laid out the full process for building evals: discover and analyze errors, create customized metrics, and set up a continuous improvement loop. **Unfortunately, we’ve found that most teams skip the first stage of error discovery and jump straight to writing metrics. **
It is easy to see why. Looking through lengthy user session records to find failures feels slow and hard to scale, whereas metrics are concrete and easy to automate. But if you write metrics too early, you end up making too many assumptions about what’s important—and potentially measuring the wrong thing, or the right thing poorly.
This is why error discovery is the eval equivalent of product discovery. Just as product discovery shows which problems are worth solving, error discovery reveals which AI failures are worth measuring. Without it, teams risk building dashboards around generic metrics that waste time and steer the product toward the wrong outcomes.
We believe error discovery is so important that if you only have time for one part of the eval process, you should prioritize it.
In this post, we’ll show you the three steps to running effective error discovery with a coding agent like Codex or Claude. We’ve used this process with more than 50 companies, and each time the tools uncovered major product flaws that were hurting the customer experience. This entire workflow takes only about 30 minutes to complete once you learn the basics.
Note: Error discovery has changed a lot since our last post. Our prior post called this process “error analysis.” We now call it “error discovery,” because the goal is to identify failures that are worth measuring. Keep reading to learn about the new approach.
When you’re building an AI product, you need evals to understand where it makes mistakes. But maintaining evals costs time and money; it doesn’t make sense to measure everything. Good error discovery identifies which failures are worth measuring and tracking over time. Even when you know you need the error discovery stage, it can be tempting to start by handing an agent a folder of traces—complete records of user sessions with your AI product—and asking it to find problems. Agents are often faster than humans at spotting obvious issues and can find patterns we might miss. But they are far less reliable when a failure depends on your definition of a good product experience. You can explain those standards to the agent, but you often only discover them in the first place by reviewing the data. This process, where reviewing examples changes your definition of good, is called criteria drift.
For example, below is an interaction from Nurture Boss, an AI leasing assistant we worked with that helps property managers handle conversations with prospective tenants:
Prospect:“This is out of my budget. Thank you for your business.”
Leasing assistant:“You’re welcome! If your situation changes or if you have any other questions in the future, feel free to reach out. Have a great day!”
The full conversation from this interaction is provided below in the discussion on traces.
To most agents, this looks like a success; they wouldn’t identify an error in this trace. But the product goal is to facilitate sales, which includes finding the right property matches for prospects’ different needs. In this situation, the agent should have explored cheaper units or other properties owned by the same company and offered the prospect alternatives.
If we’d prompted the agent to look for this “objection handling” failure up front when we gave it traces to check, the agent would have caught the error automatically. But we’d only know to add “objection handling” to our criteria after seeing this trace ourselves. This is a classic case of criteria drift—and why it’s critical to take a step back to review failures and define success before dispatching an agent to find errors in a stack of traces.
In a broader study, we ran automated eval tools and coding agents against 100 production traces from this same apartment-leasing assistant. We found the following:
Agents missed issues requiring product judgmentand context outside the trace such as Markdown formatting in text messages and missed human handoffs (in addition to objection handling).Agents are good at catching failures that are obviousinside a trace, like answers that are contradicted by tool output.Agents also find issues that humans miss, but they introduce noise by flagging good responses as failures.
Clearly, automated approaches are still useful for finding some types of errors, and they work especially well in conjunction with human judgment. So how do you benefit from an agent’s automation while keeping a human in the loop? The answer is a process that draws on active learning, a method for choosing the most informative examples to review given limited time. Start with a diverse sample of traces so you cover the range of your data. When you find a failure, look at a few more instances of it before you decide you understand it. After you’ve reviewed enough examples, let the agent annotate traces that you can then accept or reject.
All of this bookkeeping and sampling would be difficult to do manually. But coding agents are great at it. The rest of this post walks you through how to work with an agent to do meaningful error discovery step by step, with the aid of an eval skills plugin we prepared for you.
Error discovery requires traces. Each trace consists of the user’s input and system prompt, whatever your system did in between (retrieval, tool calls, intermediate model calls), and the product’s final output. Each trace should contain enough information for a reviewer to reconstruct what happened and decide whether it was good. You can choose to log this data to a database, eval vendor, or even a local folder. For this post, we will assume a local folder for simplicity.
Here is what a trace might look like for the leasing assistant. Note that this is what it looks like in its raw form, and you usually want to render it to be human-readable (we will get to that later):
If you don’t have traces, you can ask your coding agent to instrument your application so it logs this data. Here’s what a prompt for that might look like:
Instrument this app so every user session with the AI is logged as one complete trace.
A trace is one user session. Include the user input, the system prompt, every tool call and its result, any retrieved context, every intermediate model call, and the final user-facing output.
If this app already sends traces to a vendor (LangSmith, Arize, Phoenix, Langfuse, or similar), keep using that. Also write a local copy: one JSON object per session, appended to traces/traces.jsonl. If there is no vendor, the JSONL file is enough.
Once your product is instrumented, you’ll need to wait for user activity to collect traces. If you haven’t launched yet, you can try to generate synthetic traces by simulating user queries with an LLM. While synthetic data cannot replace real data, sometimes it’s better than nothing.
An effective way to simulate user queries is to define a small number of dimensions that you anticipate your product might fail on. For example, dimensions for the leasing assistant might be the task (scheduling a tour, asking about pricing, asking about the pet policy), the type of person asking, and whether the request was clear, ambiguous, or out of scope. Then have the model use these dimensions and turn each combination into a natural-language user query. Using dimensions like this helps steer the AI away from producing homogeneous outputs.
Once you have defined the dimensions, your coding agent can combine their values into scenarios. Each scenario contains one value from each dimension. The agent can then make a separate model call for each scenario. Here is an example prompt that will generate synthetic data with this approach:
Create a script that generates synthetic user queries for an AI leasing assistant. Use the dimensions and values below as fixed inputs. Do not add or change them.
Task:scheduling a tour, asking about pricing, asking about the pet policy
Renter:first-time renter, relocating family, student
Request type:clear, ambiguous, out of scopeCreate a structured list of test scenarios by combining one value from each dimension. Loop through the scenarios and make a separate model call for each one. Pass only one scenario into each call. Enforce a structured output schema with a single
user_queryfield, then save the query alongside its scenario insynthetic_queries.jsonl.
The prompt above recommends separate model calls for each scenario because we’ve found that asking an agent to one-shot it tends to produce less diversity. Here is an example of what one synthetic user query might look like:
{ “user_query”: “We have two dogs and may be moving next month. Would that work?” }
After generating synthetic data, review the examples and remove any that are unrealistic. If you notice scenarios that your dimensions don’t cover, update the dimensions and generate new examples covering those gaps.
Producing high-quality synthetic data (especially for complex, multi-turn conversations) is beyond the scope of this post. **We recommend getting real users instead of relying on synthetic data where possible. **If you must use synthetic data, we have a skill that can help you with that in the appendix.
Now we are ready to find errors in your product! We built an evals plugin to guide you through this process, based on what we learned teaching more than 4,500 PMs and AI engineers and consulting with over 50 companies. Install it with npx skills:
npx skills add https://github.com/ai-evals-course/evals-skills
Then point your coding agent at the /evals-start skill or tell it where your data is:
“Use the evals-start skill. My traces are in traces/traces.jsonl and I want to find issues occurring in my AI product.”
evals-start is the entry point for the evals skills plugin. It looks at your situation and sends you to the right workflow. In our case, we have traces we haven’t analyzed yet, so it will route us to error-discovery.
The first thing error-discovery does is read a sample of your records to learn their schema. Then it creates a small review app customized for your data and serves it to you locally. Here’s an example of the app interface that the plugin created for the leasing assistant traces:
This interface renders the conversation as a message thread, with tool calls and their outputs displayed alongside. You input your annotations into a free-form text box.
Interface customization is an important part of the plugin’s value. Different types of interfaces are best suited to reviewing different types of data. For example, here is what the rendered annotation interface might look like for a writing assistant:
In this app, the writing displays as a text block, with flags displayed alongside, which makes it well-suited to reviewing writing.
There are several design principles embedded in the skill that guide the way an interface will render. The two most important ones are:
**Show user-facing output the way the user sees it.**For example, an email should look like an email, a PDF should render as a PDF, etc.**Expose important metadata that may be helpful for navigation or filtering.**In the leasing example, the channel the user comes through is likely important and worth filtering on: SMS, voice, web chat, etc.
The skill also instructs your agent to cluster your traces so it can build a diverse initial sample. It mixes clust…
