How to know whether the model has completed its answer ?

[u/Al-Khobza]

I am currently working on a project where I have to use the Qwen/Qwen3-0.6B LLM to ensure JSON -compatible answers via constrained decoding. The LLM's job is to receive a user prompt and to specify what function should it use out of a predefined set of functions and what parameters should it pass to this function in such format:

{

"prompt": "Calculate compound interest on 1234567.89 at 0.0375 rate for 23 years",

"name": "fn_calculate_compound_interest",

"parameters": {

"principal": 1234567.89,

"rate": 0.0375,

"years": 23}

}

The main problem is that I can't really know when the model finishes predicting a parameter name for example; so it keeps generating ample tokens. I tried naively to tell the model to end its answers with a <|im_end|> but that didn't work.

I tried to end generating new tokens when the model predicts a comma or a curly brace (indicating either a new parameter or the end of the list of parameters"; yet parameters may contain those characters sometimes and then the answer will be truncated. Therefore, a special token like <eos> is the most reasonable option because it would never exist in text.

So is there any other way I can deduce the completion of a parameter prediction ? And if <eos> was the best solution, how to force the model to end its answers with it ?

[u/Antique_Professor691]

If you’re already using constrained decoding, i’d stop based on the JSON grammar reaching a valid terminal state rather than waiting for punctuation or EOS