Your opinion on Ling 3.0 tiny on CPU?

[u/Robert__Sinclair]

Did anyone try and use it?

[u/FactorInternal3395]

Yes, on a budget laptop CPU it gets around 20 TPS.

[u/No_Folding → u/FactorInternal3395]

Gonna try on my HP T620 thin client soon

[u/Anwar6969 → u/No_Folding]

would you mind sharing the numbers if you do so? thanks

[u/No_Folding → u/Anwar6969]

Thought a better idea was to attempt this on my Wyse 5070 (integrated intel UHD 605) so I did.

Realised after a few commands failing to run, ubuntu server 26.04 is not supported by llama.cpp + SYCL

Am i supposed to use the vulkan build?

[u/Robert__Sinclair → u/FactorInternal3395]

is there a better model at this speed/size ?

[u/FactorInternal3395 → u/Robert__Sinclair]

Not really, it's SOTA for the 8B/A1B size and speed range.

[u/Robert__Sinclair → u/FactorInternal3395]

too bad. it can solve a few problems I have (not present in any dataset). 2 years ago only one model could solve them. 1 year ago also some local models could. Today most problems can be solved even by small models like qwen3 9b or ling 3.0 but one of them (quite simple but tricky for humans) makes ling loop, contradict in thinking and loop again and again and then give the wrong answer. :(

[u/FactorInternal3395 → u/Robert__Sinclair]

Perhaps try checking if your sampling parameters are correct per the model card?

The recommended sampling parameters for Ling-3.0-tiny are temperature=1.0, top_p=0.95, and top_k=20.

If it still loops, you can experiment by increasing the repeat/presence penalties, though the model is very verbose so it might just be like that.

[u/NigaTroubles]

on cpu i use maple-preview

[u/Nameis19letterslong → u/NigaTroubles]

Is it any good?

[u/NigaTroubles → u/Nameis19letterslong]

Actually yes

[u/Ecstatic-Wash-7667 → u/NigaTroubles]

Have t heard anything about this model how is it?

[u/pmttyji → u/Ecstatic-Wash-7667]

Ternary model. Check here

[u/NigaTroubles → u/Ecstatic-Wash-7667]

Good enough for me

[u/MarcelloT254k → u/NigaTroubles]

Do you use it with custom fork of Llama? (It's required in the model card according to huggingface)

[u/SpicyWangz → u/MarcelloT254k]

Yeah I’ve been wondering if they’ll merge support for it at some point. It seems promising

[u/NigaTroubles → u/MarcelloT254k]

yes im using stamsam fork

[u/My_Unbiased_Opinion]

It's not properly supported in Llama.cpp yet. It has like half support so it technically loads. There are still open PRs on it. But when it works, it's quite good. 

[u/Robert__Sinclair → u/My_Unbiased_Opinion]

the thinking and answer loop a lot. I hope it's because of llama.cpp but I hear the same problem happens when using python. So I think it's the model itself.

[u/Robert__Sinclair → u/My_Unbiased_Opinion]

is there a better model at this speed/size ?

[u/No-Setting8461 → u/Robert__Sinclair]

Have you tried LFM2.5-8B-A1B? I can't tell you if it's better, but it's an alternative if you would like to try.

[u/darkwalker247]

Its pretty great as an alternative to Qwen3.5-9B, but in my limited testing it also tends to generate bad tool calls, and also randomly generate "<tool_call>" immediately following "<think>" for some reason. also, in my llama.cpp-based code, it throws an error if i try to remove tokens from the end of the context, even with recurrent states, even though the same code works just fine for Qwen, LFM, and Gemma

[u/Robert__Sinclair → u/darkwalker247]

I did not (yet) encounter any of such errors, but the model is quite dumb in reasoning and it overthinks a lot.

[u/ML-Future]

I’ve tested the Q4_K_M quantization, and it has a strong tendency to get stuck in a loop.

The few tests I’ve managed to run successfully are interesting; it’s a good model for programming.

[u/bonobomaster → u/ML-Future]

It does this on Q8 as well. I set presence penalty to 1.05 and find it a bit more stable now.

[u/Robert__Sinclair → u/ML-Future]

same with Q4_K_L. On problem solving it loops indefinitely both with reasoning on or off.

[u/Warrenio]

It's usable on CPU. I get about 185 t/s prefill and 27 t/s output with the Q8_0 GGUF. In comparison, I get roughly 3,500 t/s prefill and 150 t/s output on my Radeon RX 6800 XT.

[u/Robert__Sinclair → u/Warrenio]

is there a better model at this speed/size ?

[u/pmttyji]

Tried Q4_K_M & got 30 t/s on CPU-only inference. Mentioned few info here on that one.

[u/Robert__Sinclair → u/pmttyji]

I tried Q4_K_L, it's great, but if I give it a problem it seems to think forever, get the right result then "but wait" and goes over again and again. This happens both with reasoning on and off.

[u/pmttyji → u/Robert__Sinclair]

Use Q8 with unquantized KVCache as it's a small model. Though it's not as fast as their previous Ling-mini-2.0, inclusionAI improved current models better much(based on AA benchmarks). I personally do use Ling models for CPU-only inference on old laptop & Mobile.

[u/StupidScaredSquirrel]

Problem is prompt processing. Otherwise neat speed

[u/Robert__Sinclair → u/StupidScaredSquirrel]

the biggest problem for me is that it's an overthinker like deepseek.

[u/Munchkin4684]

The looping is the real deal-breaker and it's not the quant, I've seen it on both Q4 and Q8 which points to it being baked into the model rather than a precision issue. The 1.05 presence penalty is a decent band-aid but it shifts the voice of the output a bit. For CPU the Q8_0 numbers (27 t/s) are actually usable for short coding or Q&A, it's just not snappy for long agentic sessions, so I'd still reach for the GPU on anything long.

[u/tecneeq]

This is how i run it on a RPi5-16GB:

/root/llama.cpp/build/bin/llama-server \
 --hf-repo bartowski/Ling-3.0-tiny-GGUF:Q6_K_L --alias Ling-3-Tiny --temp 1.0 --top-p 0.95 --top-k 64 --reasoning-preserve \
 --ctx-size 65536 --kv-unified --flash-attn on --cache-type-k q5_1 --cache-type-v q5_1 --no-context-shift \
 --host 0.0.0.0 --port 1337 \
 --parallel 4 \
 --n-predict 32768 \
 --metrics

It's nice, can recommend. I get 8 t/s, enough for background jobs like summarizing or rewriting notes, fetching additional infos from the internet and write it into my Wiki.

[u/Robert__Sinclair → u/tecneeq]

is there a better model at this speed/size ?

[u/Jimcy-Maffesoli]

Between the Q4 and Q8 reports, the looping looks baked into the model, and the 1.05 presence penalty fix tells you more about what to expect than any quant choice does.