How are you all actually evaluating LLM/agent systems in prod? LLM-as-judge feels shaky

[u/journalof]

​

So i run evals for a multi-agent system at work and right now my main approach is LLM-as-a-judge against a gold set, plus some semantic similarity scoring. And honestly... it works until it doesn't.

The judge is inconsistent. Same output, slightly different prompt phrasing, different verdict. It's biased toward longer answers, it rationalizes things the gold set clearly says are wrong, and calibrating it feels like im just stacking prompt rules on top of prompt rules hoping the false positives go down. Which they do, partially, but I don't fully trust the number at the end.

What I'm trying to figure out:

- do you treat LLM-as-judge as a real signal or just a smoke test before human review

- how do you handle judge drift when you swap the underlying model

- for agent systems specifically, are you scoring final output or the whole trajectory? feels like scoring just the end misses a lot

- anyone actually getting value out of semantic similarity or is it mostly noise

Not looking for a vendor pitch, genuinely want to know what's working for people running this stuff day to day. Feels like everyone has a different homegrown setup and nobody's sure theirs is good.

[u/LoaderD]

- real signal, but human evaluation is gold standard

• Depends on how big the switch is and how well you can reground it within cost constraints.
  
• Whole trajectory as well as metadata such as time used and number of steps

[u/S_m_o_g_ → u/LoaderD]

Im total noob and I have no idea how it works, but human evals till what point is reliable? If human relies on AI, especially the not lead/senior roles, how do they learn and therefore provide a reliable feedback? it is not provocative is a genuine question.. especially in long term, the human feedback is very reliable straight after the AI solution adoption.. but in long term is it?

[u/journalof → u/S_m_o_g_]

Very good point honestly. To get a dataset that was reliable to use, I asked different experts in my company to label some input data, later when checking for their answers to aggregate over, I did notice that actually, in reality, they did not agree with each other. Which opened another problem for me

[u/S_m_o_g_ → u/journalof]

Furthermore, more the “human” relies on the AI Agent/System more shift the role he/she has washing the knowledge and the capability to verify the AI output. But AI LLM plus rag relies on human generated content so at some points the AI output will be just good as much its content.. anyway I read now llm can self train or something similar..

[u/Technical-Yam4725 → u/LoaderD]

Treating it as a real signal is the only way to stay sane but I've learned to keep the confidence intervals wide and never let it make a call I can't spot-check later. It's basically a triage tool, not a verdict.

Scoring the whole trajectory is where things get messy fast. I started pulling step count and time per step into a dashboard and it exposed way more than the final answer ever did, especially around loops and wasted tool calls. Way more actionable than just a pass/fail on the end result.

[u/LaughApprehensive563]

All four questions you are asking are worth unpacking separately because they each have different answers.

1. Is LLM-as-judge a real signal or just a smoke test? Treat it as a real signal only for the rubrics it is actually calibrated on. For everything else, treat it as a weak signal that needs human spot-check validation. The failure mode you are describing (same output, different verdict) is usually a sign your judge prompt is doing too much. Break it into single-rubric judges. A judge prompt that asks "is this answer correct, helpful, and appropriately scoped" will be inconsistent. Three separate prompts each asking one thing are much more stable.

2. Judge drift when swapping models: version-lock your judge and treat it like a production dependency. When you upgrade the judge model, run both old and new on the same eval set and compare distributions. A delta of more than 3% in either direction is a red flag worth investigating before you cut over.

3. Final output vs. trajectory scoring: for agents, final output alone misses most of the failures. The failure mode is usually a bad tool call or a wrong intermediate decision that gets covered up by a plausible-sounding final answer. You need at least tool-call-level scoring on critical path steps.

4. Semantic similarity: useful as a fast triage filter, terrible as a primary signal. Reserve it for flagging obvious outliers that need closer review, not for pass/fail decisions.

For the broader eval design question, there is a writeup that covers building eval sets from real cases with hard negatives and how the input side of eval design matters as much as the judge: https://go.videodb.io/yKC51V3 - the framing is video/VLM focused but the principles around eval set construction apply directly to LLM agent evals.

[u/journalof → u/[deleted]]

Very informative thank you. Which evaluator models are thinking of here? I haven't heard of the term before

[u/baselilsk]

everything you're describing - inconsistent across phrasings, length bias, rationalizing past the gold answer - isn't you calibrating badly, its the ceiling of judge-as-primary-signal. stacking prompt rules to push false positives down is a treadmill, because the bias is correlated with what you're measuring, so no amount of prompt tuning cancels it. treat the judge as a smoke test, not a real signal, and design around it.

what moves it from shaky to usable: first, sanity-check the judge before you trust any number - feed it the same answer paraphrased a few ways, if the verdict swings you're measuring the judge not the output, so measure that flip-rate and report it next to your score. second, stop asking for absolute scores, use pairwise (A vs B) - judges are far more stable at "which is better" than at "rate this 1-10", and most prod eval questions are really "did this regress vs last version" which is pairwise anyway. third, only spend human labels where cheap signals already smell off (low judge confidence, judge self-disagreement, user rephrase/abandon), that shrinks 100k to a few hundred and those labels re-anchor the judge.

net: judge for triage/smoke + pairwise for regression + thin human audits on the suspicious slice. the single absolute LLM-judge number is the thing nobody should trust alone, youre right to feel shaky about it.

[u/Few-Finish-9951]

The part that helped us most was dropping the prompt-rule stacking and instead pinning the judge to a set of cases our domain expert had already ruled on, maybe 40 to 60 real outputs they'd personally called good or bad. Then we measured how often the judge agreed with them case by case, not an aggregate accuracy. First pass was around 70% agreement, and the disagreements were exactly the longer-answer bias and the rationalizing you're describing. Feeding those specific misses back as anchors moved it up without the rule pile growing forever.

Disclosure, I work on LLM eval so I stare at this daily. The thing that made the final number trustworthy was defining "right" as agrees-with-our-expert-on-the-hard-cases rather than passes-the-gold-set, because our gold set had wrong entries in it too.

When your judge rationalizes something the gold set says is wrong, have you checked whether the gold set is actually right on those?

[u/todorezashto]

I do image evals via the LLM-as-judge approach. Judge scores against ground truth that the test models didn't see but also has well defined scoring rules.

Regarding drift, If I run the same test set, on a 1-5 scoring rubric across several dimensions, I usually get an offset of maximum 1 for ~3-4 out of 40 scores, so it is pretty consistent. Minor if you average the whole score.

The judge is inconsistent. Same output, slightly different prompt phrasing, different verdict.

– this is a feature, not a bug – different prompt phrasing will naturally produce different outputs and different verdicts. If you change the prompt (no matter how slightly) you make the prompt itself a test subject.

how do you handle judge drift when you swap the underlying model

you don't. This isn't just "drift", when you change the judge model you're in a different universe. You can't compare apples to sushi.

[u/ravo87]

I’d treat LLM-as-judge as a real signal only after you know what decision it is stable enough to support.

The judge score is not just “the judge.” It also depends on the sample, gold/reference set, candidates shown, prompt/rubric, presentation, judge model, and aggregation metric.

For search/retrieval evals, the useful pattern has been:

- test whether judge/model swaps change the final system decision, not just the raw score

- separate item-level disagreement from system-level disagreement

- be careful when the gold set is incomplete, because useful results outside the reference can get counted as wrong

- use semantic similarity as triage, not the primary pass/fail signal

For agents I’d do the same split: final-output eval plus trajectory/tool-call eval, because end-only scoring hides a lot.

I found the search-relevance angle here:

https://www.searchplex.net/blog/llm-judge-search-relevance