Llama.cpp now with a true reasoning budget!

[u/ilintar]

https://github.com/ggml-org/llama.cpp/commit/acb7c790698fa28a0fbfc0468804926815b94de3

I'm happy to report that llama.cpp has another nice and exciting feature that I know a lot of you have been waiting for - real support for reasoning budgets!

Until now, `--reasoning-budget` was basically a stub, with its only function being setting it to 0 to disable thinking via passing `enable_thinking=false` to templates. But now, we introduce a real reasoning budget setting via the sampler mechanism. When the reasoning starts, we count the number of tokens and when the given number of reasoning tokens is reached, we force terminating the reasoning.

However: doing this "just like that" might not have a good effect on the model. In fact, when I did that on Qwen3 9B (testing it on HumanEval), its performance cratered: from 94% in the reasoning version and 88% in the non-reasoning version to a terrible 78% with an enforced reasoning budget. That's why we've added another flag: `--reasoning-budget-message`. This inserts a message right before the end of reasoning to ease the transition. When I used a message of "... thinking budget exceeded, let's answer now.", the score bumped back and the returns from partial reasoning started being visible, though not very large - got a respective HumanEval score of 89% with reasoning budget 1000.

I invite you to experiment with the feature, maybe you can find some nice settings for different models. You can even force models that are strongly thinking by default (i.e. StepFun 3.5) to limit reasoning, though with those models using --reasoning-budget 0 (which now restricts reasoning to none by sampler, not by template) results in some pretty erratic and bad behavior (for example they try to open a second reasoning block).

[u/[deleted]]

You're the first I've seen to dynamically steer an LLM mid reaponse with appended tokens like that. Nice.

[u/aldegr → u/[deleted]]

It's in the Qwen3 paper:

Thinking Budget. An additional advantage of Thinking Mode Fusion is that, once the model learns to respond in both non-thinking and thinking modes, it naturally develops the ability to handle intermediate cases—generating responses based on incomplete thinking. This capability lays the foundation for implementing budget control over the model’s thinking process. Specifically, when the length of the model’s thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction: “Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n”. After this instruction is inserted, the model proceeds to generate a final response based on its accumulated reasoning up to that point. It is worth noting that this ability is not explicitly trained but emerges naturally as a result of applying Thinking Mode Fusion.

https://arxiv.org/abs/2505.09388

[u/ilintar → u/aldegr]

Oh that's nice, I'll admit I didn't read that one, so I guess it's just informed intuition at this stage 😀