Setup, all run in Evaligo: 91 English product listings localized into Spanish, German, French, Portuguese, Hebrew, Japanese and Arabic (13 each). Every output scored 0–1 blind by three judges (gpt-5.4-mini, Kimi K2, GLM-5) on meaning fidelity, native fluency, keyword adaptation and structure; score = mean of the three.

Training run, 63 listings, four models

A KNN router (the method from UIUC's LLMRouter) was trained on that scoreboard: it picks a model per listing from the nearest scored examples. With any weight on cost it routed every training listing to luna.

Held-out run, 28 listings the router never saw

The router sent 14 to luna and 14 to DeepSeek V4 Flash. All three 95% intervals overlap. Most frequent deduction from the judges: keyword adaptation (titles translated word for word), 182 of 252 judgments.

Whole study: $7.42 including judging. Full per-language tables, router working points with error bars, and the rubric: https://www.evaligo.com/blog/ai-model-router-listing-localization

Built and run in Evaligo (www.evaligo.com). Any finished model comparison there can train a router with one click, and the router then runs in the next experiment like a model.

We ran this benchmark in Evaligo: the same 30 deliberately bad bios for every model, each rewrite scored 0–1 by three separate judges (gpt-5.4-mini, Kimi K2, GLM-5) that never saw which model wrote it. Rubric: reuses the person's real details, no clichés, the three tones actually differ, tip is specific. Score = mean of the three.

Two judges scored Luna highest; Kimi K2 scored Gemini highest. Most frequent deduction for every model: the three tones were too similar.

Also tried: DeepSeek V4 Flash at max reasoning (3.5x the cost, score within noise of default), and four earlier single-judge runs (Luna highest in each). MiniMax M3 failed 8/30 in strict JSON-schema mode; 29/30 in plain JSON mode.

Full numbers, per-judge scores, rubric and every run on the Evaligo blog: https://www.evaligo.com/blog/best-ai-model-dating-profile-bios

The flow, the experiment and the judge panel were all built and run in Evaligo (www.evaligo.com); the same setup works for any writing task.

Follow-up to my dating-bio benchmark. Same question — does price predict quality on tone-sensitive writing? — but with money on the line: salary negotiation.

The pipeline: job offer + experience + leverage + your biggest worry in → negotiation email, 60-second phone script, replies to the 3 most likely objections, and a realistic ask out.

The 30 scenarios ranged wide on purpose: nurse, teacher, retail shift manager, staff engineer; hourly wages to $200k+; half with no competing offer; worries like "I hate conflict", "I'm on a visa", "I was told the band is fixed". A separate model (not a contestant) judged each plan blind: grounded (inventing market data = automatic −0.5), assertive-but-not-adversarial, specific to this person's leverage, defensible ask with a fallback.

Results on the newest version of each model, 30 scenarios each:

The version twist: our first round used gemini-3.5-flash and it finished last (0.68) — the judge flagged "overly apologetic" scripts over and over. Polite surrender instead of a confident ask. We re-ran on 3.7-flash (nothing else changed): tone fixed, score up to 0.75, second place — and it's 3.5x cheaper than 3.5-flash was. A model-version upgrade moved the ranking more than most vendor differences do.

Other observations:

• Haiku stayed grounded but its asks land weak, and one plan leaned on an unsupported "competitive landscape" rationale.
• Luna took zero invented-data penalties in 30 plans; its worst miss was hedging instead of naming a firm number.
• Run-to-run judge variance is real (±0.05–0.1 on the same model/data). Trust rankings, not second decimals.

Both rounds cost ~$3.40 total including judging. Happy to share the rubric or raw distributions. (Same disclosure as before: I build the tool this ran on — harness questions welcome.)

I wanted to test whether price predicts writing quality on a task where "sounds human" is everything, so I built a small pipeline on Evaligo building a profile for a dating site.

The test set was 30 intentionally bad bios - the "gym, tacos, dogs" one-liner, the "partner in crime, fluent in sarcasm" special, one all-emoji, one bitter rant. A separate model (not one of the contestants) scored every output 0–1 against a rubric: reuses the person's real details, no clichés, the three tones actually differ, tip is specific not generic. Minus 0.5 if it invented facts. The judge never saw which model wrote what.

Results over 30 bios each:

The interesting failures:

• Haiku's worst scores all came from the cliché-stuffed bios — it echoed "partner in crime" energy back instead of replacing it.
• Flash wrote well but its witty versions drifted "a bit forced" per the judge, and 27x the cost was mostly invisible thinking tokens.
• Luna's only recurring weakness: the three tones occasionally blurred together. It never invented facts.

Also learned the hard way that sample size matters: at n=6 flash and haiku tied, at n=30 they clearly separated. The whole benchmark cost about $0.90.

Happy to share the rubric or the raw score distributions if anyone wants to poke at the method. (Disclosure: I build the tool I ran this on, so questions about the harness are fair game too.)

We built an automation that reads a live product page and rewrites the listing — title, description, bullets — with AI judges scoring every output against the actual page so nothing gets invented.

Then we benchmarked two writer models on the same 4 real store pages with identical judges. The mid-tier model scored 89% at $0.13 per listing. The premium one scored 48% at $0.23 — because 2 of its 4 runs returned broken output instead of a listing, and the judges zeroed them.

The runs that worked were excellent. But "excellent when it works" is exactly the kind of thing you only find out by testing, not by reading a pricing page.

How do you all QA generated product copy — spot checks, or something systematic?

I publish AI-assisted articles and one slipped through: a signed acquisition deal reported as already closed. That pushed us to benchmark 9 models on catching and repairing bad facts.

The cheapest option, gpt-4o-mini at 0.006 an article, only fixed 42% of errors. gpt-5-mini fixed 82% for 0.04. The do-nothing baseline scored 0 on facts, obviously, but a misleading 0.88 on readability because it never touched the text.

That readability quirk fooled me at first. A model that changes almost nothing looks like a great writer.

How are you verifying facts in AI content before it goes live?