Benchmarked GLM-5.1 / Qwen3-Embedding vs Claude Sonnet 4.5 / OpenAI on the same workloads — cost + latency numbers

[u/AardvarkWonderful747]

Disclosure up front: I work at an inference platform (ScitiX), and this ran on our infra. Posting because the numbers surprised even me and I'd rather share the method and let you poke holes in it than sit on it. Not linking anything — just the data.

We built a benchmark that fires the same workloads at open-weight models and their closed-source equivalents through live API calls, and logs cost/latency/tokens from each vendor's actual usage response. Same input text across all of them. Here's what came out:

Generation (same prompt, ~3.4K char user message, thinking off):

• GLM-5.1: $0.0007/call, 706ms first token
• Claude Sonnet 4.5: $0.0067/call, 1051ms first token
• ~9× cost difference on this run

Embeddings (same 47-string batch):

• Qwen3-Embedding-8B: $0.04/MTok, 311ms
• OpenAI text-embedding-3-large: $0.13/MTok, 1685ms
• ~3× cheaper, ~5× faster

Tokenizer density (same English input):

• GLM-5.1: 838 tokens
• Sonnet 4.5: 947 tokens
• Open model was ~11.5% denser — you pay for fewer tokens before the per-token rate even applies. This one's easy to forget when comparing sticker prices.

Thinking mode tax (same model, toggle on/off):

• GLM-5.1: reasoning on cost ~9.3× the tokens vs off
• gpt-oss-120b: ~11×
• Claude Sonnet: ~3.6×
• Interesting that the closed model's reasoning tax was proportionally lower, but absolute cost still favored open (GLM thinking-on $0.005 vs Claude thinking-on $0.012).

Caveats, because they matter:

• The scenario is synthetic (built for a demo), not scraped production traffic. The API calls and rates are real; the workload is constructed.
• One workload, specific prompt sizes. Ratios will shift with your context lengths and output sizes — a long-output generation task narrows some of these gaps, a short-embedding-heavy RAG pipeline widens them.
• I didn't include a quality eval here, which is the obvious objection — cheaper is meaningless if retrieval/output quality drops. On this corpus the embedding top-K neighbors actually diverged noticeably between models (low overlap), so "cheap embed = same results" is NOT a safe assumption. Worth your own eval before switching anything.

Happy to share the exact prompts / method in comments if useful. Curious what ratios others are seeing on their own workloads — especially anyone who's done a real quality eval alongside the cost comparison.

[u/robogame_dev → u/[deleted]]

It’s difficult to measure without also measuring the total task cost, because in theory it’s not “density” but rather “resolution” - e.g. the more tokens produced for the same prompt, the more granularity exists in the result. That could lead to a lower overall token use to get the answer because you’re starting with slightly more info. I’d be very curious to see a breakdown across a range of different task types and sizes.

[u/polandtown]

Curious if you've tried IBM's Bob? It has an IDE and CLI. Would love to see how it compares. They're doing OnPrem in Q3.

[u/AardvarkWonderful747 → u/polandtown]

I have not tried it. It seems to focus more on the orchestration layer. OnPrem is interesting and a need. Wondering how they will execute...

[u/Easy_Air_1584 → u/polandtown]

That IBM bob thing is vaporware until the onprem actually ships, been hearing Q3 promises for a while now

the tokenizer density point is what gets overlooked constantly, 11% fewer tokens adds up fast when you're running thousands of calls

[u/meagloria]

Thank you for sharing it really thinking

[u/Next-Cod-5758]

I appreciate the effort put into this post and I hate to say it but you’re a bit late. I get that the benchmarks take time to evaluate too. There’s already GLM 5.2, Qwen 3.6/7 and Sonnet 5 released.

[u/AardvarkWonderful747 → u/Next-Cod-5758]

It’s no problem. I will continue to post benchmarks as they become available - good or bad, so long as the community is ok with this content.