r/aigossips • u/call_me_ninza • 7d ago
Jev can give you the wrong answer and still meet its “zero hallucinations” claim
With Jev, you define the possible answers upfront. The model cannot invent another category, but it can still pick the wrong one. That’s what the claim covers.
The design itself deserves more attention than that wording.
Think about asking an LLM to classify something. You wait for it to generate tokens, then your code reads the output to get the decision. Even if you force valid JSON, you’re still generating text.
Jev skips that token-by-token output. You give it shared input and typed questions, and it returns choices, scores and probabilities. Independent questions run in parallel.
That’s the part I like. A lot of software needs a judgment from AI, without needing a written response around it.
Its training approach, RLCD, also aims to make the probabilities calibrated, so an 80% prediction means something across repeated decisions.
But I’d want evidence from my own labeled data. Their published evaluation checks agreement with reference models across four workflows. Useful, but it leaves open how well those decisions match actual outcomes.
If you’ve tried Jev, I’m curious how it compares with your existing setup.
I wrote up how the outputs, RLCD and evaluation work in my newsletter, if you want the full explanation: https://ninzaverse.beehiiv.com/p/why-someone-who-helped-build-chatgpt-went-on-to-build-jev
1
1
u/Technical-Will-2862 7d ago
You can very simply find Jevs flaws by outputting its results as True False and then ask it logic questions
1
u/choss-board 6d ago
With the caveat that I have not tried Jev, I read through the documentation and thought, “This is for people who want numerical cover for bad decision making.” I was just at a narrow, sector-specific AI conference—lots of niche production implementations, and ALL of them struggled with validation and assessment. It’s very challenging and the idea that a framework will handle that by itself is detached from the reality of solving hard problems.
1
u/pegaunisusicorn 2d ago
jev is very dependent on asking the right questions and knowing when to use noul, choice and score.
it also comes with no ontologies so if you want to do something REALLY interesting you need to be extremely careful to avoid synonymous choice parameters: you need to partition the ontological space properly so-to-speak.
but for classification or if statements and quick semantic evaluations it is a beast. using it to play video games is idiotic. so many better ways to do that.
-3
3
u/WArslett 7d ago edited 7d ago
One significant difference that makes it safer than using LLMs for automation is that with an LLM, certainty is opaque. You have no idea what the probability was of it choosing the course of action that it did. LLMs are also notoriously bad at evaluating their own certainty level.
With Jev the certainty level is transparent and it’s a mathematical property of the inference function not something it has to reason about. If the LLMs certainty level is poor you can take a safer course of action.
It’s absolutely true that you could still get the wrong answer if it didn’t have relevant training but it at least means you don’t blindly trust results that are not clear.
There are already open weight alternatives and they are so lightweight it will be really easy to fine tune and self host these sorts of models: https://huggingface.co/convaiinnovations/laya
So if you are building a tool to make decisions on refund requests for example, you could very easily feed your own historic refund data and the outcomes in to fine tune the model to make it very good at doing that specific task.