Even (very) noisy LLM evaluators are useful for improving AI agents · TensorZero

May 12, 2026 · Alan Mishler It’s surprisingly hard to develop reliable LLM evaluators: they’re often noisy and poorly correlated with the metrics or outcomes practitioners actually care about. Sometimes the target is directly measurable but evaluators still disagree with experts (e.g. on correctness or faithfulness to a source document). Other times the target is only accessible through a proxy (e.g. whether code that passes tests satisfies user needs). And sometimes the target is hard to observe at all (e.g. whether a customer was actually happy with an interaction). Why is it so hard to develop reliable LLM evaluators?Rule-based and classical NLP metrics are often brittle and miss the semantic dimensions that matter.1, 2 Learned reward models are vulnerable to distribution shift3 and reward hacking.4 Studies of LLM-as-a-judge setups have repeatedly documented systematic biases and limitations: judges are heavily swayed by surface-level style,5 prefer longer responses to shorter ones of similar quality,6 are inconsistent across repeated evaluators and minor prompt variations,7 often align poorly with human judgments,8 and may correlate weakly with the downstream outcomes they’re meant to predict.9 An evaluator’s quality can be measured at two granularities:

Output-level correlation measures how well its score on individual outputs matches real-world outcomes. It governs production workflows (e.g. guardrails), where decisions hinge on individual outputs and noisy evaluators are unreliable. We’ll call an evaluator noisy with respect to a metric or outcome of interest if its output-level correlation is low. Agent-level correlation measures how well its average over many outputs matches an agent’s real-world quality. It governs offline variant selection (e.g. picking the best prompt or model), and, unlike output-level correlation, it generally climbs with sample size as per-output noise averages out.

Even very noisy evaluators can be reliable for offline selection: enough to ship better agents today and keep improving them over time. Why noisy evaluators can still rank agents The key insight is that even a very noisy evaluator can yield scores that are higher on average for agents that truly are higher quality: the noise washes out over many samples. To formalize this, suppose we have two agents we want to compare, AA and BB.

Let μA\mu_A and μB\mu_B represent the mean true scores for AA vs BB in the problem setting of interest, where true score refers to the thing we’d ideally want to measure, like how well the agent handled a customer’s query or whether it produced runnable code. Suppose that higher scores are better. Then we’d say that AA is better than BB if μA>μB\mu_A > \mu_B. Now suppose we have an evaluator whose scores can be regarded as noisy versions of the true scores. Here are three hypothetical samples of true scores and evaluator scores for increasingly noisy evaluators: Slightly noisyModerately noisyVery noisyHypothetical samples of true scores (x-axis) and evaluator scores (y-axis) for three evaluators with increasing noise. The dashed line marks y = x (a perfect evaluator). The leftmost evaluator is accurate enough to judge individual outputs in production. The rightmost isn’t: its verdict on any single output is too noisy to trust. However, if we’re using an evaluator offline to choose between AA and BB, then we don’t need every individual value to be accurate. We just need the evaluator to tell us which agent is better overall. All three evaluators will do that, given sufficiently large evaluation samples. Suppose Agent AA has true-score mean μA=0.6\mu_A = 0.6 and Agent BB has μB=0.3\mu_B = 0.3, so AA is the better agent. Below are the same scatterplots as in the figure above, but with each output now colored by which agent it came from. Let μ^A\widehat{\mu}_A and μ^B\widehat{\mu}_B be the average evaluator scores for each agent, shown as horizontal dashed lines on each plot. In all three initial samples, μ^A>μ^B\widehat{\mu}_A > \widehat{\mu}_B, meaning the evaluator correctly leads us to choose the better agent. Agent AAgent Bμ^A\hat{\mu}_Aμ^B\hat{\mu}_BSlightly noisyModerately noisyVery noisySamples drawn from the true-score distributions of Figure 2 (Agent A with mean 0.6, Agent B with mean 0.3; 30 samples per agent), with evaluator scores on the y-axis.

Horizontal dashed lines mark each agent's empirical mean evaluator score (μ^\hat{\mu}); the diagonal marks y=xy = x. Sampling detailsFor each agent, we model the distribution of true scores as a Beta distribution parameterized by its mean μ∈(0,1)\mu \in (0, 1) and a fixed concentration κ>0\kappa > 0:S∼Beta(κμ,  κ(1−μ)).S \sim \text{Beta}\left(\kappa\mu,\; \kappa(1 - \mu)\right).The mean of the distribution is exactly μ\mu, and increasing κ\kappa concentrates mass more tightly around μ\mu. We use κ=5\kappa = 5, μA=0.6\mu_A = 0.6, and μB=0.3\mu_B = 0.3, which gives each agent a unimodal distribution with moderate spread while keeping the two visually comparable.Agent AAgent BTrue score distributions for Agent A (μA = 0.6) and Agent B (μB = 0.3), modeled as Beta densities with concentration κ = 5. Dashed vertical lines mark each agent's mean.For each evaluator with noise σ\sigma, the evaluator score for an output with true score SS isV=clip(S+ε,  0,  1),ε∼N(0,σ2),V = \text{clip}(S + \varepsilon,\; 0,\; 1),\qquad \varepsilon \sim \mathcal{N}(0, \sigma^2),with σ=0.03,  0.12,  0.25\sigma = 0.03,\; 0.12,\; 0.25 for the slightly, moderately, and very noisy evaluators, respectively. Each click of the “Draw new samples” button shows a fresh random realization with N=30N = 30 trajectories per agent; the empirical means μ^A,μ^B\widehat{\mu}_A, \widehat{\mu}_B are the average evaluator scores within each agent’s sample.

Even though the samples are noisier as we move from left to right, they still tend to produce the correct ordering (μ^A>μ^B\widehat{\mu}_A > \widehat{\mu}_B) once they’re averaged. Of course, these values are random, so there’s always some chance that the empirical means will mislead us, pointing to the worse agent as the better one. How likely that is depends on a few things:

How separated the agents are. A bigger gap between μA\mu_A and μB\mu_B, relative to the variances of the scores, makes it easier to preserve the correct ordering under noise. How noisy the evaluator is. A less noisy evaluator narrows the spread (lowers the variances) of μ^A\widehat{\mu}_A and μ^B\widehat{\mu}_B, making the correct ordering more likely at any given sample size. How many evaluator samples we have. Empirical means concentrate around their expected values as the sample size NN grows, so larger evaluation datasets give more reliable comparisons — no matter how well separated the agents are or how noisy the evaluator.

In general, even noisy evaluators can reliably distinguish stronger from weaker agents, given a sufficiently large evaluation dataset. How big does an evaluation dataset need to be?The sample size required to reliably distinguish two agents scales inversely with the square of the performance gap between them — halving the gap roughly quadruples the number of samples you need. This squared scaling comes from how the sampling distribution of a mean tightens with NN: the variance of a sample mean shrinks as 1/N1/N, so its standard error shrinks as 1/N1/\sqrt{N}, and reliably resolving a gap of size Δ\Delta requires the standard error to be small relative to Δ\Delta — i.e., NN to grow as 1/Δ21/\Delta^2. The interactive figure above is illustrative: with a 0.30 gap and only N=30N = 30 samples per agent, even the noisiest of the three evaluators gets the ordering right essentially every draw. For agents that differ by 5 to 10 percentage points on an outcome of interest — a typical magnitude in practice — even a fairly noisy evaluator can give the correct ranking with high probability on a few hundred to a few thousand examples.

The argument above works as long as the evaluator is not biased in a way that causes it to systematically favor the worse variant. Formal argument and failure modesThis section formalizes the claim that the empirical evaluator means can recover the true ordering of agents given enough samples, and discusses when this can fail.Let xx be an agent output, and let S(x)S(x) be the true score for xx, meaning the thing we’d ideally like to measure. Each agent gives rise to a distribution over trajectories xx, which can differ in arbitrary ways. Maybe AA tends to produce long, detailed responses, while BB tends to be short and crisp. Or maybe AA is more prone to hallucination than BB. The average scores μA\mu_A and μB\mu_B are expected values over those distributions, which we’ll denote with EA\mathbb{E}_A and EB\mathbb{E}_B, so thatμA=EA[S(x)],\mu_A = \mathbb{E}_A[S(x)],μB=EB[S(x)].\mu_B = \mathbb{E}_B[S(x)].Now let VV be an evaluator, so that V(x)V(x) is the score the evaluator assigns to output xx. Define a noise term ε(x):=V(x)−S(x)\varepsilon(x) := V(x) - S(x). Then we haveV(x)=S(x)+ε(x).(1)V(x) = S(x) + \varepsilon(x). \tag{1}The empirical evaluator means converge in general to the true population evaluator means:μ^A→EA[V(x)],μ^B→EB[V(x)].\widehat{\mu}_A \rightarrow \mathbb{E}_A[V(x)], \\ \widehat{\mu}_B \rightarrow \mathbb{E}_B[V(x)].Therefore, in order for the empirical means to lead us to choose the better agent with high probability, we need the expected evaluator scores to mirror the true ordering of agents.

That is, we desire the following property to hold:EA[V(x)]>EB[V(x)]  ⟺  EA[S(x)]>EB[S(x)](2)\mathbb{E}_A[V(x)] > \mathbb{E}_B[V(x)] \iff \mathbb{E}_A[S(x)] > \mathbb{E}_B[S(x)] \tag{2}for all pairs of candidate agents AA and BB. From Eq. (1) and the linearity of expectation, we haveEA[V(x)]=EA[S(x)]+EA[ε(x)],\mathbb{E}_A[V(x)] = \mathbb{E}_A[S(x)] + \mathbb{E}_A[\varepsilon(x)],EB[V(x)]=EB[S(x)]+EB[ε(x)]\mathbb{E}_B[V(x)] = \mathbb{E}_B[S(x)] + \mathbb{E}_B[\varepsilon(x)]  ⟹  (EA[V(x)]−EB[V(x)])=(EA[S(x)]−EB[S(x)])+(EA[ε(x)]−EB[ε(x)])\begin{aligned} \implies \left(\mathbb{E}_A[V(x)] - \mathbb{E}_B[V(x)]\right) ={} & \left(\mathbb{E}_A[S(x)] - \mathbb{E}_B[S(x)]\right) + {} \\ & \left(\mathbb{E}_A[\varepsilon(x)] - \mathbb{E}_B[\varepsilon(x)]\right) \end{aligned}From this, we see that Eq. (2) can be satisfied in many different ways. For example, if the noise terms average out to 0 for both distributions (a common assumption), then we’d have(EA[V(x)]−EB[V(x)])=(EA[S(x)]−EB[S(x)]),\left(\mathbb{E}_A[V(x)] - \mathbb{E}_B[V(x)]\right) = \left(\mathbb{E}_A[S(x)] - \mathbb{E}_B[S(x)]\right),from which Eq. (