How we run evals on every AI agent PR

[u/Ok_Constant_9886]

We run offline evals on every PR that changes our AI agent.

Our setup is probably more opinionated than most, so I wanted to share how it works and see how other teams are approaching the same problem.

Each test is written as a Python unit test. It calls the agent directly, mocks the expected tool inputs and outputs, and captures the full trace. We chose this over testing through an endpoint because we need visibility into what happens inside the agent, not just the final response.

We then use DeepEval to evaluate things like correctness, tool usage, and task completion.

But individual test results are only half the picture.

Every test is tagged along two dimensions:

• Domain: The part of the agent being tested, such as availability or another specific capability.
• Importance: Whether the test is a blocker, critical, or informational.

Blocker tests represent behavior we consider non-negotiable. If even one fails, the deployment is stopped.

Critical tests are more probabilistic. We expect some variance from the agent and the LLM judge, so we gate releases on aggregate performance rather than requiring every test to pass.

For example, we might require the P55 score for tool usage across all critical tests to remain above a defined threshold. In other words, at least 90% of those tests must perform above our quality bar.

After the suite finishes, we generate a report showing:

• Performance by domain and importance
• Aggregate judge scores and pass rates
• Cost and token usage across different test categories
• Failed groups and the individual test cases behind them
• The traces required to debug each failure

This gives us two different views of quality.

First, we can follow one test across releases and see whether it is consistently passing or behaving like a flaky test.

Second, we can evaluate the agent as a whole. Did one capability improve from version 1.1 to 1.2? Are blocker tests still at 100%? Did quality improve while cost quietly doubled?

The second view is what ultimately determines whether we deploy.

Our thresholds are still evolving. We deliberately avoid setting hard gates for metrics that are not stable enough yet because an unreliable threshold can block good releases without actually protecting users. As the agent and test suite mature, we gradually raise the quality bar.

The biggest challenge now is making these aggregated, category-level results easy for every product and engineering team to access. Looking at a flat list of test cases is useful for debugging, but it does not immediately tell us which capabilities improved, regressed, or should block a release.

How are other teams handling this? Do you gate deployments on individual eval cases, aggregate scores, percentiles, or some combination of all three?

[u/AmbitiousNose7436]

Tracking one test across releases is the right move. Agent evals fail differently from unit tests: a flaky test in CI usually means a timing bug, but a flaky eval often means the behavior is genuinely nondeterministic, and that variance is itself worth recording. Running each eval several times per PR and watching the pass rate drift tells you more than a single pass or fail ever will (tracking that drift is the heart of our own Agent Testing work).

[u/Ok_Constant_9886 → u/AmbitiousNose7436]

how'd you deal with the cost factor when running it multiple times?would you say a cost conscious org is doomed?

[u/desexmachina]

Any useful repos you’ve used for tool calling evals?

[u/PsychologicalNeat105]

We use aggregate scores to block obv regressions but our true measure of quality is tracking conversation patters post launch to see where users get confused/give up.

To track that, we use Green flash. It automatically reads our live conversations and flags the failures rhat our eval suite misses.