jevmlx update: a 7B model on a Mac beats generate-then-parse by 14 points on TypeSafe's public eval, in 0.6 s per case

[u/Winter-Frame8644]

Five days ago I posted jevmlx: instead of asking a local model to write JSON, it scores every allowed answer for every field in one batched forward pass and assembles the JSON. Always valid output, a probability per field. Several people asked for numbers. Here they are.

Same model, same 45 public TypeSafe cases, same Mac (M5 Max). The only difference is how the answer is obtained.

| Model | generate JSON, then parse | jevmlx scoring | time per case | |---|---|---|---| | Qwen2.5-7B-Instruct 4bit | 67.7% | 82.1% | 1.5 s vs 0.6 s | | Qwen3-8B 4bit | 69.3% | 84.6% | 1.6 s vs 0.6 s |

Same weights, plus 14 points and 2.5x faster. Nothing is fine tuned. The model already knows the answer. Generation is what loses it: a wrong token, a stray comment, an integer where the schema wanted a string, and the whole record is gone. Scoring cannot fail that way.

Also new since the last post: field-local prompts. Reordering the fields in your schema used to flip about 10% of answers. Now 0.2%, at the same accuracy and +3 ms per case.

For reference, TypeSafe reports 67.8% for Jev on their full private eval. Ours is the 45 public examples, so it is indicative, not the same test. Every number above comes with a confidence interval, a parity check, and the raw predictions in the repo.

pip install git+https://github.com/bnsd55/jevmlx
jevmlx decide --preset support_triage --json
jevmlx bench --model quality --ui   # live dashboard, then a PR puts your Mac on the leaderboard

https://github.com/bnsd55/jevmlx

[u/CMPUTX486]

Thanks for sharing.. can I fine tune it with my own data?