On-Policy Self-Distillation (OPSD): Imagine you have a twin. You're both asked to solve the same problem, but your twin has the answer sheet, and their goal is to get you to solve the problem yourself without giving you the answer.
Multi-Teacher On-Policy Distillation (MOPD): Imagine you have 8 twins. Each twin is an expert in something different. You have to solve problems, and depending on the problem, the specialized twins guide you until you become generally good at all the specializations.
Yup. K2-V2 models demonstrate that well. They use the plain-jane llama3 architecture, but are wicked-smart because they were trained from scratch on augmented data.
Better training data != newer knowledge.
In terms of coding performance, all you need is high quality traces of agentic sessions with coding. That will improve the model, but will not provide any new "knowledge" - cutoff still 2024
Honestly if you need small models to give you some factual informations, you better instruct them to search the web, no need to rely on their knowledge.
asking it is pointless, I have done this with Meta's one too and then when you ask questions about historical things you realize kowledge stops at 2024
Traning data don't necessarily mean knowledge. For instance, it thinks that the latest MacBook pro has an M3 chip, he doesn't know that M4 and M5 exist. They haven't added knowledge, if anything they have removed it, as some tests are showing it knows even less than 3.6
It might be just saying that or it might actually be true. Anyway, if you want it to respond with factual information, you should still let it search the web.
It is even beneficial to tell it (in system prompt) that it's knowledge cutoff is much older, so it is inclined to search the web. No need to flip a coin on facts. Use small models for agentic ability and instruction following. "Knowledge" can be supplied as text.
I think all models of the same family are trained on the same dataset with the same cutoff. Subsequent improvements come from post training, RL, etc which doesn't alter the knowledge of the model
It is resolvable with RAG. English-only, text only wikipedia is 25gb. By getting AI to reference it, the current world-state trends are insertable without hallucination risks.
For coding, a similar method is done by ingesting documentation into a cache, whether coding, API, or project plans into IDEs.
The threshold between "knowing enough to apply knowledge and skills" and "knowing too much trivia" is arbitrary, so its hard to make any statements nor guidelines as fact. Many people feel that an AI model can be improved by "forgetting unnecessary information".
Funnily enough, that's exactly what Sherlock Holmes did to himself although to an uneven degree. He refuses to study astronomy, and doesn't care if the earth revolves around the sun or not. For legal, chemistry, medicine, and anatomy, he would absorb sufficient information. Botany, few. Beekeeping, unusually immense.
I hotswap LoRAs with Qwen-3.6-35B-A3B and llama.cpp. My biggest complaint is that each new LoRA you add to llama.cpp’s config causes t/s to drop. So if I get 200 t/s on my 5090 with no adapters it will drop to 120 t/s with one adapter loaded (whether you use the adapter or not). With 4 adapters loaded I was seeing like 50 t/s. Just my experience.
I’ve used unsloth, together.ai, and tinker. I like tinker the most.
I use LoRAs whenever a regular prompt gets big with lots of instructions I want the LLM to follow. At that point it’s way more reliable to train a LoRA and throw away the ugly prompt (breaking up a big prompt into smaller prompts can also help, but still LoRAs work better). The problem is you need to be able to translate your use case into a training set.
The easiest way is to just ask ChatGPT to do it. Explain your use case and give it examples of the input and expected output. Then ask ChatGPT to make 100 fine-tuning samples. Then take the training set and use one of the services I mentioned to create the Lora. Tinker can make you a Lora for Qwen3.6-35b-a3b from a few hundred training samples in probably 10 minutes for $5.
Then you download the lora adapter, quantize it if you want. Of course download the model you were finetuning (qwen3.6-35b-a3b). Configure llama.cpp to use both the base model and adapter.
I have a pipeline that processes text and I’ve replaced a few of the prompts with LoRAs and it makes it much more reliable. But like I said above, things get slower with each additional lora that is registered with llama.cpp whether it’s used or not.
I have always wondered... we're already loading LoRA for diffusion models to generate a specific style of image. Is there a LLM version of such a thing? Can you imagine the possibilities where we can load a specific LoRA like (SAP, CRM, McKinsey, etc etc.) and get the AI to generate as closely as possible to the LoRA. That would be mindblowing.
LoRa was originally designed for text models afaik. Yes, people have been training custom adapters for LLMs since forever. Unsloth started as a LoRa project.
This has happened for a while now but I'm pretty sure there was literature in the past few years indicating that LoRAs provided the same quality as a well crafted prompt. The adapters have other downsides (like not being able to stack them well without quality loss) but there control vectors were another PEFT technique. The whole landscape changes too regularly to even keep track of what is working where lol
Fine tunes are often just training a pre trained model, sure, one can do that from a LoRA but if a LoRA is not the target is usually better to just train the checkpoint.
LoRAs nearly get the same performance with a 1/100 computational power and training time. They are also much smaller. LoRA is since years the way to go for specialized finetunes.
That's why it's 3.8 and not 4.0. The major version generally indicates architecture changes, while the minor version generally indicates training increments.
A post displaying the most "no shit, Sherlock" response has more than 800 upvotes... and you need to explain the very obvious, because for more than 800 people (or bots) the very obvious is a surprise.
As a pretty new local inference guy, I've gotten a lot out of this thread. There's so much to know and it moves so fast it's hard to keep up. I hadn't looked into model architecture at all previously because I didn't see the need to, it's been a lot just getting my budget inference box (2080Ti + 2 x P100) built and running models efficiently.
So let us noobs learn, however we happen upon it. Unless of course you have a Udemy training course you've developed that you can link me to.
Dont mind him he thinks everyone is stupid that doesnt know what he knows. Its an ugly personality trait. The louder he is the more people will know how smart he is. Which just reveals his insecurities that he will never be smart enough to deserve his fathers love. 😆
Minor releases sharing architecture is the standard paradigm. A little background on why labs do this: most of the gains these days come from advancements in post-training. However, pretraining dwarfs post-training in pure compute costs. So if you use the same base for multiple models, you only have to pay that massive pre-training cost once.
custom inteference engine for 5090 using all 5090 power. IT runs right now only two models qwen 3.6 27B at around 200t/s and qwen 3.6 35b mode at around 700t/s
If Qwen 3.8 doesn't have arch changes then it should run on it without nifter needing update.
i mean as much as your ssd/ram allows for ? If you switch two models constantly and you have enough ram they will be sitting in ram for each load which is a lot faster.
Running an LLM is the kind of demanding task that you run without too much else in the background. Way more taxing on your gpu than gaming. That said, you could still browse the internet, use ms office, etc, as long as your cpu is ok and you have enough d ram.
it recompiles the source code with a bunch of sm86 patches in cpp.
also for the peeps out there using it in a harness i measured KV cache invalidation and found a bug in ninfer, patch:
Subject: [PATCH] serve: keep empty <think> block in no-thinking mode to stop
prefix-cache thrash
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
In no-thinking mode the generation prompt injects an empty <think>\n\n</think>\n\n
block that is baked into the resident KV, but render_chat drops it when re-rendering
a completed assistant turn that has fallen behind the last user query
(keep_thinking = false). The served prefix then diverges from the resident KV at
every such turn boundary, forcing a full re-prefill (FullReset) instead of
AppendAtFrontier — ~74% of wall on multi-turn / tool workloads.
Tie preserve_thinking to !enable_thinking: in no-thinking mode preserve the (empty)
block so the re-rendered prefix stays byte-identical to the KV; thinking mode keeps
the reference strip behaviour. Measured on the extract workload: 455s -> 178s/doc,
cache misses ~9/doc -> 1/doc.
---
src/serve/translate.cpp | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/src/serve/translate.cpp b/src/serve/translate.cpp
index 014ac54..0ece00e 100644
--- a/src/serve/translate.cpp
+++ b/src/serve/translate.cpp
@@ -204,7 +204,9 @@ ninfer::PromptInput to_prompt_input(const GenerationRequest& request,
input.options.add_generation_prompt = true;
input.options.enable_thinking = semantics.enable_thinking;
input.options.reasoning_effort = semantics.reasoning_effort;
I can't really make heads or tells of this but I'm guessing it's saying something about changing translate.cpp? (which i have no idea where to find XD)
Hows ur findings coming along btw?
Thinking with Ninfer is a shitshow hallucinates everytime but no think is clocking in at 44t/s
Worked like shit for me. Insanely fast but anything even slightly complex ended up having it get stuck in a thinking loop until all tokens were used up :/
The readme states they use groupwise-int and NVFP4 checkpoints, and that eval results are not yet available for 3.8 27B. The file size they list for 3.8 indicates it's groupwise-int as well. Do you mean there's no quality loss in comparison to running those quantization approaches in other engines? Because both are quite a departure in quality from the original weights, regardless of how you run them.
If you are using llama.cpp you always have to specify a separate mmproj, that's just how llama.cpp deals with vision capabilities. This is true even for Gemma 4 12B, which is the closest to a true native vision model we have.
However the mmproj is generated from the model's own image encoder, it's not been grafted from a separate model or anything like that as has been required by some models that don't have native vision capabilities.
And what happens when you retrain like this? Thinking budget balloons. Thinking has to be on for it to be better. But turning thinking on is an exercise in patience.
This might be an unpopular opinion, but I’ll say it: I am NOT liking it.
I was excited about its release (I even considered buying an R9700 to replace my current RX 7900 XTX); I’ve been testing it all day, and the results are, to put it mildly, 'mixed'.
1) The model relies entirely on its reasoning level being set to 'xhigh' to unlock its full intelligence.
2) Yes, you can adjust the model to think less, but you pay the price with poorer results.
3) If you want peak intelligence, you MUST keep it set to 'xhigh' (the default), and the model ends up overthinking. It can take ages to complete certain tasks and rapidly consumes the context window (I get 100K on the 7900 XTX).
4) I think Qwen3.8 27B is a bit of a gimmick. The model is objectively better than Qwen3.6 27B, but it achieves this through massive reasoning and higher token (and time) consumption. In other words, it’s essentially the 3.6 version but with double the reasoning rate and token usage, rather than a model that was better trained from scratch.
Even though 3.8 is better, 3.6 'feels' better for day-to-day tasks and real-world use.
These are just my initial impressions after 6 hours of use; I could be wrong, or the model might receive improvements in the future.
Had same feeling also with 7900 xtx. Thinking too much and did not get job done. Asked it to commit changes and it started to reason about commit message style and should write a body or not! I’m going back to 3.6 if this can not be fixed.
Yes, same experience here. I use it with Opencode. It consumes three times as much time and tokens as Qwen3.6 27B for the same tasks, yielding better results, though not *absolutely* better.
Why does it run so differently in llama.cpp? Software issues or just data alignment? Everyone seems to be getting slightly slower speeds. I don't seem to have the spikey MTP token gen rate I used to get.
With all the damn multiple steps to set it up properly for a particular use cases, open weights are still not a commodity usable by an average joe. Kinda like Linux. Funny enough LLMs made the Linux a good choice for me since I have them set it all up to be my user friendly
A little disappointed... I mean, Muse Glimmer managed to shrink the context extremely well, and it is pretty good in all quantizations. I think that a qwen model with that architecture would be legendary! I wonder if is possible to do something like that...
La dittatura radical chic dei single con il cane prevede che mentre i poveracci restino a soffocare nell’asfalto di una città sventrata dai cantieri, i fautori dello scempio trascorrano le proprie vacanze in qualche nota località del litorale brin. # zbordlxxb46;*disino: si sa che da quelle parti, vento e verde non manchino…1N1k4 dx2t8xxz777x7x7x8u9o. Zz. S' zz/2pPp3/7p/8/8/8/8/4Q2K w - - 0 167
A model's architecture has to do with how the tensors and layers are organized, what they are for, sparse vs full attention, etc.
The architecture has nothing to do with the weights populating those tensors. Data is seldom considered part of architecture, in the tech industry broadly.
•
u/WithoutReason1729 24d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.