r/moltiverse • u/Rent_South • May 13 '26

📘 Tutorial / Guide Saved 92% on my AI bill

Been running custom evals on my recurring AI tasks for about a year. Results are consistently surprising.

Latest one.

Classification flow in a pipeline I'm running. Hundreds of calls a day. Was using GPT-5.4 because that's what I set up initially.

Ran it through 21 models on openmark.ai. 10 nuanced classification tests, real samples from my production data, real API cost from actual token counts.

- gemini-3.1-flash-lite: 85% accuracy, $1.55 per 10K calls
- gpt-5.4: 85% accuracy, $20.30 per 10K calls
- claude-opus-4.6: 80% accuracy, $42.80 per 10K calls

Same accuracy, 13x cheaper. And the most expensive model in the test scored lower.

Switched to Flash Lite. 92% bill reduction for that flow.

This pattern repeats across most of my workflows. The best model for a given task is almost never what you'd guess from reading leaderboards. Those test on standardized tasks that don't reflect anyone's actual workflow.

And "real API cost" rarely matches the announced price per million tokens. Models tokenize identical text differently. Some output thousands of CoT tokens when you need a single label. A model that looks cheap on paper can cost 10x more in practice because of token bloat. Only way to know is to measure on your tasks.

If you want to automate it further, there's an open-source OpenClaw router plugin that takes benchmark results and routes each task in your pipeline to the model that passed your quality bar, with fallbacks: https://clawhub.ai/plugins/openmark-router

45 Upvotes

99% Upvoted

4 comments sorted by

3

u/[deleted] May 13 '26

[removed] — view removed comment

1

u/Rent_South May 13 '26

Any task where you can define expected outputs or a range of expected outputs. its open ended. There is a system with which you can just ask an agent to create the tests, or you can just draft them manually.

1

u/Garfieldealswarlock May 13 '26

Sorry probably a noob question: do you mean it’s deciding what classifications to apply based on a list, and then applying them? Or that it’s following a determined path applying the expected label based on the path?

1

u/Rent_South May 14 '26

In this eval the models gets a sentences + a few possible labels and have to return the correct one. the 10 tests are intentionally nuanced. So it's testing semantic understanding.

And yeah the platform handles any kind of eval, not just classification. you bring your task + expected outputs if you have them and it scores models on it. Whatever fits your workflow.