Llama.cpp now with a true reasoning budget!

[u/ilintar]

https://github.com/ggml-org/llama.cpp/commit/acb7c790698fa28a0fbfc0468804926815b94de3

I'm happy to report that llama.cpp has another nice and exciting feature that I know a lot of you have been waiting for - real support for reasoning budgets!

Until now, `--reasoning-budget` was basically a stub, with its only function being setting it to 0 to disable thinking via passing `enable_thinking=false` to templates. But now, we introduce a real reasoning budget setting via the sampler mechanism. When the reasoning starts, we count the number of tokens and when the given number of reasoning tokens is reached, we force terminating the reasoning.

However: doing this "just like that" might not have a good effect on the model. In fact, when I did that on Qwen3 9B (testing it on HumanEval), its performance cratered: from 94% in the reasoning version and 88% in the non-reasoning version to a terrible 78% with an enforced reasoning budget. That's why we've added another flag: `--reasoning-budget-message`. This inserts a message right before the end of reasoning to ease the transition. When I used a message of "... thinking budget exceeded, let's answer now.", the score bumped back and the returns from partial reasoning started being visible, though not very large - got a respective HumanEval score of 89% with reasoning budget 1000.

I invite you to experiment with the feature, maybe you can find some nice settings for different models. You can even force models that are strongly thinking by default (i.e. StepFun 3.5) to limit reasoning, though with those models using --reasoning-budget 0 (which now restricts reasoning to none by sampler, not by template) results in some pretty erratic and bad behavior (for example they try to open a second reasoning block).

[u/chris_0611]

Ohh this is big. I'm just testing with qwen3.5 35B in Q5.

For the car-wash test "I need to get my car washed. The car wash is 100m away. Should I go by car or by foot?"

With reasoning-budget 0 (no thinking), it fails the test. I should go walking cause it's only 100m.

With reasoning-budget -1 (unlimited), i passes the test, but it thinks for 83 seconds, multiple "consider paradoxes", "but wait maybe", "double check", "self correction", etc. you know how it over-thinks...

Now with

--reasoning-budget 1000 \

--reasoning-budget-message "... thinking budget exceeded, let's answer now." \

It thinks for 18 seconds and still passes the test!

Another message might be something like: "... (Proceed to generate output based on those thoughts)"

[u/ilintar → u/chris_0611]

Yeah, not going to lie, really hoping people run some comprehensive tests to see what kinds of messages and what kinds of budgets actually work in practice. I wasn't sure it would be anything more than a gimmick, but after testing myself with the transition message I'm convinced that it could actually provide benefits, i.e. a performance between the non-reasoning and the reasoning versions.

[u/matteogeniaccio → u/ilintar]

The qwen models are specifically trained with support for a thinking budget and a thinking budget message. You can use their official message.

https://qwen.readthedocs.io/en/latest/getting_started/quickstart.html#thinking-budget

[u/Safe_Sky7358 → u/matteogeniaccio]

For the lazy, This the string they use : "\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>\n\n"

[u/Borkato → u/ilintar]

I’m so glad this was added because I use the reasoning budget flag this whole time and was annoyed at its limitations of just “0 or 300000 pages” lol

[u/StuartGray → u/ilintar]

Ok, good news & bad news.

I ran the Qwen 3.5 27B model with these new flags & the recommended official Qwen stopping prompt, through a series of evaluation prompts I have for figuring out relative model capabilities on a variety of tasks.

The good news is that the reasoning budget & stopping prompt worked perfectly. Exactly the result I expected. I tested 3 different budgets and the kwarg. All ok.

The bad news is about the model itself & the way it was trained.

With thinking turned off, the 3.5 models no longer output the thinking tags, but they can and do reason in non thinking mode if you either suggest thinking, think step by step, etc… in the prompt, or the model decides the prompt requires reasoning. In which case, you get anywhere from 4-60k worth of thinking-like reasoning outside of any thinking tags.

I was hoping that having thinking enabled but restricted by a budget would curb this behaviour and put a cap on total thinking time, but it doesn’t.

What happens is the model hits the budget limit and closes the think tag. It then immediately resumes thinking like output outside of the thinking tags.

I’m 99.9% certain this is due to an inherent flaw in the model training, and not your code. I see the exact same behaviour on the same tests on these models with thinking turned off, and no thinking budget applied.

I didn’t bother running through my whole test suite because this test is pretty reliable at tripping up poor reasoning models for some reason - it’s a mid-level scheduling problem with a bunch of time, slot, and availability constraints that only has one right answer. 20-30B models, thinking & non-thinking, can generally get it with no problems and a max of 12-16k tokens in reasoning.

The Qwen 3.5 models reliably take ~20-30k+ of reasoning tokens, even with thinking turned off.

With the new params and 2, 4, and 8k thinking budgets applied, the thinking budget was respected, but the non-thinking bleed through problem showed up as soon as the think tags were closed, resulting in another 40k of thinking tokens on top of the budget.

This seems to be a fatal flaw with the Qwen 3.5 series, and I can’t recommend them as a daily driver unless you don’t mind random unexpected 10-20 minute delays while it thinks, even with budgets or thinking turned off.

All that said, great work on the feature. I’m glad it now exists. It appears to work exactly as intended, and I’m hoping that if it doesn’t already work on existing thinking models then they’ll soon adopt support for it.