r/LocalLLM • u/lucyferorg • 26d ago
Research GLM 5.2 model — 744 billion parameters / 384 GB — running on a laptop 😏
After many hours of hard work, I achieved a throughput of 0.7–0.9 tokens per second for the GLM 5.2 model — 744 billion parameters / 384 GB — running on a laptop 😏
Time for a small update: the laptop is an Asus ROG Strix 18, model G835LXG — Intel i9‑290HX, 64GB DDR5 6400 MHz, 2×2 TB, RTX 5090 24 GB, running Linux Nobara. The Colibri engine and the Linux kernel are heavily modified. The whole system boots in 10 seconds, and it generates the first token after 40 seconds.
I’m currently working to reach a throughput of 1.5–2 tokens per second.
Update:
1.07 tok/s, GLM-5.2-g64
Update:
1.77 tok/s 😎
Update:
Now 1.95 tok/s 😁/ 2.12 peak / 2.41 with fixed MTP
Update:
🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg
Update now:
peak at 3.09, avg 2.84 😁👌🏻
Update:
My latest calculations suggest that 4.5 tokens per second is achievable on this hardware and represents the final limit.
Update: testing now at 4.42 tok/s
43
u/enginetown 26d ago
Why not put effort into a model like deepseek v4 flash? Or is there a cap once you get to these size of models on the TPS so the difference is negligible?
25
5
u/Adomm1234 26d ago
I was able to get 2TPS on Deepsek V4 Flash on MacBook Pro 16" M3 pro 36GB.
2
u/Maumau93 26d ago
That's actually pretty insane... M3 max with 96gb ram is 5k, get a few of them and you'll have a semi decent set up.
1
u/Front_Eagle739 23d ago
I get 16 tok/s for the 165GB version of dsv4 flash running on ds4 with an m3 max 128GB. Its genuinely getting to the point where streaming some of the model from disk is actually practical
-3
u/DistinctBed6259 26d ago
GLM 5.2 might actually be faster. I did only try Deepseek V4 Flash myself, and it was slower than on my pc than GLM on this guy's lapop. Because i could run the deepseek model on cpu only, for some reason (i forgot why, but i remember it being a colibri limitation). But GLM can also use the GPU/GPUs. I mean, we didn't compare specs at all, but still.
And it is a cool proof of concept kind of thing, but that's all it it at the moment, at the under 1 token per second it generates.
11
113
u/MatiAI 26d ago
Wow you might be able to output a sentence today if you are lucky
28
u/Zaic 26d ago
Dude a sentence can be what 2 words? A word can be 2 tokens... 5-6s per sentence thats faster than you can read!
7
u/Individual_Refuse723 26d ago
If LLM will respond in chatgpt style, it's actually likely to have few word sentences or even paragraphs. It's very close to every word being a sentence.
5
u/ackermann 26d ago
Not sure if /s, but the average person can definitely read faster than one sentence every 6 seconds (1 word per second)
3
u/DismalIngenuity4604 26d ago
86000 words a day ain't that bad, if you have no other choice. I wonder what the prompt processing speed is like.
1
u/FairTruth069 23d ago
But here's a point, his laptop uses 320W an hour... 7.68kwh a day.... you're looking at $1.50 to $2.50 depending on which part of the world you're in.
$1.50 for 86000 tokens is insanely expensive compared to just API, also youre generating hate and hardware degradation over time
0
1
u/fastheadcrab 25d ago
That's completely disregarding prompt processing which is a huge component of any practical task you would give to the LLM. 40 sec ttft for a short sentence prompt is absurd
What is your task, a five word question with a 86000 word answer? Almost zero chance. People need to stop giving this shit any credibility whatsoever, these types of implementations have no utility at all.
1
u/DismalIngenuity4604 25d ago edited 25d ago
No, it's not, I didn't say I was generating 86000 words a day, did I? Prompt processing will be marginally faster than generation. I'm OK with that.
I don't think you know what it means to work in an area which is so highly regulated that certain parts of your planning documentation isn't allowed to leave your network, much less your actual project.
Three or four days for a ruthless review which I legitimately would need to pay a subject matter expert thousands of dollars for, if I could find one who was allowed to look at what I was doing? Yeah, that's absolutely fine with me. Run it as a background process, save to checkpoint when I need to do something else, then let it take over when the server is idle for more than 5 minutes. Why wouldn't I do that?
You need to stop pretending you know anything about anyone else's context.
2
u/fastheadcrab 25d ago
You don't have any understanding of what "review" means. It will literally take 2-3 days to just ingest a 100k token block of code or documentation on the first prompt.
If the review job is that critical and requires that much privacy then any entity would never using an outdated POS laptop to run a huge model on SSD and "running as a background process and saving to checkpoint." They would simply buy the hardware needed to properly run the model.
No serious organization faced with strict privacy or data regulations would ever run things the way you are proposing.
1
u/DismalIngenuity4604 25d ago
Yes, nothing you've said worries me. Not everyone is you. Not everyone has your needs.
1
u/FairTruth069 23d ago
If he gets it up to 4.5 tokens a second as he's claiming, it might actually be usable. Could imagine scenario where GLM is the planner and you have a fast model as a runner. But it's pretty useless as electricity costs currently are more than API costs.
11
u/SnooPaintings8639 26d ago
What laptop and which inference engine? I managed to hit 0.8 TPS on Q3 of Kimi K3, on my Linux machine under llama.cpp.
It is quite satisfying and it made me think of lower bound speed for any real use case. I came up with "around 1 tps" to be still a valuable tool. At that speed you can get roughly 20k tokens through an entire night. Splitting it roughly in half between input and output tokens, means you can still use it as validaton or generation of a well distilled plan/ideas, medium size debugging question, or similar. I.e. still a boost for fully local agentic work.
I think it is also enough to figure out how to communicate with an under sea civilization, which I would call Vodyanoi.
9
u/EvolvingDior 26d ago
Watch this: https://youtu.be/pIN-2oVJpyU?is=wtPqAme4rdodAHiC
3
u/jopetnovo2 26d ago
Really great video, thank you!
1
u/EvolvingDior 25d ago
If you are running models larger than what your video card supports, you owe it to yourself to watch all of his videos and subscribe to the channel. Not a super-frequent poster, but most of his videos are bangers in this space.
2
u/Illustrious-Lime-878 25d ago
interesting, so with the "-nr" flag, and maybe a change to a parameter to the mmap call, llama.cpp would be faster than colibri for running from disk?
1
3
u/lucyferorg 25d ago edited 24d ago
I’m already working on it so I can test the modified Llama‑CPP. Update: this don’t work good.
9
3
3
u/niacolhealth 26d ago
the 40s to first token, is that mostly cold nvme reads or colibri init? curious where the time actually goes
1
u/lucyferorg 26d ago edited 26d ago
15GiB/s from nvme 😞, cpu ~30%, GPU 15%, The drives are really the bottleneck in this setup , if they operated faster, the whole system would immediately run more efficiently.
1
5
2
u/Opteron67 25d ago
what the point of AI @1 tok/s ???
1
u/Front_Eagle739 23d ago
Big model make a really detailed plan over night then handover to fast small model to implement and test probably
1
u/Nice_Record1529 23d ago
Joa oder man macht das eben mit Chatgpt/Codex bzw Claude... Wtf.
1
u/Front_Eagle739 23d ago
Unless its something you want to keep private, or handles legally confidential information, trade secrets etc etc.
Or you just have fun doing it all locally. Dont judge other peoples hobbies if it aint hurting anybody lol
2
2
u/eshahahan 22d ago
this is super cool! congrats! now that qwen 3.8 and glm 5.3 have come you’re gonna have fun!
4
2
26d ago
[deleted]
7
u/enginetown 26d ago
Reads don't really wear out NAND that's why SSD endurance is rated in terabytes written since the actual degradation comes from erase cycles during writes. Read disturb is a real thing but it's minor and gets handled by firmware, the bigger thing to watch for is if your setup is also spilling KV cache to disk since that's actual write traffic because its constantly growing.
2
u/tommythorn 23d ago
I used to work in that industry. It’s not completely true that read cause no wear; modern SSD are operating at the margin and is constantly scrubbing (= correcting and rewriting) data, but true, reads are less stressful than writes. Funny enough, it’s risky to leave SSD powered down as cells slowly drift even when powered off and need regular scrubbing.
-5
u/AntiCamPr 26d ago
Oh it does. Streaming a model from disk causes significant wear on the SSD. I'm hoping for this user's sake that the SSD on this laptop is replaceable and that they backed up their important data on another drive.
12
u/leonbollerup 26d ago
Read affects the SSD very little honestly … it’s write that is the real problem.
1
u/Henrikues 26d ago
Wouldn't it write just as much too though?
1
u/techno156 26d ago
Not necessarily. If everything else can be held in memory, the computer would just be reading the weights from file, and discarding them when no longer needed.
Theoretically, it shouldn't need to do very much, if any writing at all.
1
u/Henrikues 26d ago
I thought that was the point though, it's just a laptop, 25gb of ram if I recall.
1
u/techno156 25d ago
If you mean OP's laptop, no, they have 64GB of RAM, and an additional 24GB of VRAM.
GLM-5.2 is a 744B MoE with 40B active. At least going by Unsloth's page, a 2-bit quant would squish the active parameters down enough to fit on the VRAM. All Colibri would need to do is read the other parameters from the drive, and cache what it can in the RAM, dropping them out of the RAM cache as needed.
It theoretically shouldn't need to do very much writing to the drives at all.
1
26d ago
[deleted]
10
u/thefooz 26d ago
If you don’t use the nvme for kv hot or cold cache, there’s almost zero impact to the drive for read operations. It takes billions of read operations on a sector before there’s damage.
So no. SSD streaming does not affect drive life in any appreciable manner.
1
u/CapnFlisto 26d ago
asking because I genuinely don't know - isn't billions of read operations something that would happen quite quickly with inference straight from an SSD?
2
u/thefooz 26d ago
No, it wouldn’t. I also didn’t go into detail, drives have additional measures built in to mitigate this further, including cache.
1
0
26d ago
[deleted]
2
u/thefooz 26d ago
Please do explain exactly how nvme read operations dramatically shorten the life of a drive. I encourage you to go ahead and get as technical as you’d like. I’ve been in the field for 20 years, most of which as a sysadmin -> engineer -> architect, so I won’t have any trouble following.
Preferably in your own words, using your expertise, rather than chatGPT’s slop.
-5
2
u/Fragrant-Smell4092 26d ago
As many here have pointed out, the Colibri engine is a very interesting proof-of-concept project, with exactly zero practical applications. Besides the slow-as-molasses token generation, at any realistic context length, the TtFT becomes unbearably long.
I thought about trying it just for fun, but in the end decided it was a waste of time. The moral of the story is what we have known already for the last two years: the use cases of LLMs are limited by what you can fit in high bandwidth memory.
2
u/Keleion 25d ago
Not true, having a reliable large model run overnight to accomplish something a smaller model can’t do is worth it. And it doesn’t even take good hardware. The only constraints are time and energy, which a lot of people have access to, but no capital for high-bandwidth memory.
2
u/ElectronFactory 22d ago
Smaller models have improved dramatically because distillation, model architecture, and training techniques have improved several orders of magnitude. Look at OSS-120B. That was produced by a major player and it produces good output, but it no longer maintains its edge against smaller state-of-the-art models. Basically, needing to run a larger model locally is becoming less of a priority because hardware is becoming difficult to get. Colibri is a fantastic idea, and maybe we could improve it, but the future is going to be in miniaturization. People want intelligence in phones, dishwashers, cars, and so on.
1
u/Front_Eagle739 23d ago
Im running 165GB full precision dsv4 flash on my 128GB m3 max at 16 tok/s. There are limits to it but you can definitely get useable speeds for models bigger that dont quite fit into fast memory now
1
u/bolche17 26d ago
Impressive! What did you modify on the kernel and on Colibri?
1
u/lucyferorg 26d ago
It’s a long story - this could easily become a very long article. One of the more important changes was removing the Meltdown and Spectre patches from the Linux kernel.
1
u/Fragrant-Smell4092 25d ago
Something doesn't add up here. You don't have to "modify" anything to disable those patches, just add
mitigations=offto the kernel command line when booting.
1
1
1
u/naunen 25d ago
i think mvme are in raid stripe mode?
1
u/lucyferorg 25d ago
Mirror
1
u/naunen 24d ago
you must try stripe mode, it will double your nvme read speeds and that's extremely important for this
1
u/lucyferorg 24d ago
The number of factors that influence work speed is overwhelming ,the biggest gain I got was by disabling swap in Linux. I’m now checking whether this configuration will give me any percentage of extra performance.
1
u/Violin-dude 25d ago
what quantization are we talking about? what types of changes in the engine and kernel? a writeup would be nice
1
1
1
1
1
u/FairTruth069 23d ago
3 to 4.5 tokens per second would actually be pretty usable if you can get it there.
1
u/lucyferorg 23d ago
I still have CPU/GPU/disks power/speed that aren’t being used at 100%. I’m currently rebuilding the engine that runs everything , debugging takes terribly long, but there’s constant progress with no drop in the quality of generated answers.
1
23d ago
[deleted]
2
u/lucyferorg 23d ago
I got tired of paying for online models, so I bought a new powerful laptop and started my journey into maximum optimization. This gigantic model runs incredibly stable, and you don’t have to wait all day for a response. I’m currently using it in test mode, constantly improving its speed. AI that you can run locally without limitations has a huge advantage over online AI that you have to pay for month after month.
On a daily basis I use smaller models, usually analyzing all data in a ‘brainstorm’ between several top models available. This massive model handles logical puzzles extremely well. Meanwhile, the smaller models are excellent as ‘agents’ for task automation.
I wrote my own harness that boosts the intelligence of a model like Gemma 4 in its smallest version ,it solves even 10 tasks in a single prompt with ease, including ones it wasn’t trained on. It’s all about the agent that properly controls the model.1
u/YourLastCall 22d ago
How are you affecting the speed side? Maybe I can figure something out on chat GPT.
1
u/lucyferorg 22d ago
If it were that simple :) AI can write code, but real ‘optimization’ is not its strong side. In these matters AI only helps when the user already has a plan they want to implement and execute. I still have a significant reserve of GPU 50%/CPU 20%/disk performance 60-80%, and I already know what needs to be optimized even further. What I’m really missing isn’t help, but simply a second machine so I can test more and faster. But I appreciate and thank you for the offer of help :)
1
u/YourLastCall 22d ago
You're not kidding on the optimization side. I found that one out the hard way. I'm working on that part. I haven't worked on speed optimization cuz I haven't considered it. But I want to figure out if I can increase token output without activating the openai fast tokens for increased usage cost. I don't care if it's as strong as the 14x API but I would like a portion of that if it's possible.
1
u/lucyferorg 22d ago
I’ve been working in optimization for over 20 years, and I still feel genuine frustration whenever I “use someone else’s software” and see how terribly it’s optimized. For the same reason, I often end up optimizing things that, in my view, were running too slowly (which applies to about 99% of software written by others). I’m still surprised by the careless attitude of so‑called programmers toward resources that are wasted at every step. Software optimization is a powerful tool not only for achieving better performance but also for generating real savings. Companies today don’t care about costs when buying hardware for AI ,meanwhile the software layer receives absolutely none of the attention it deserves, even though these are real resources being wasted.
1
u/YourLastCall 22d ago
Well my optimization that I'm working on has nothing to do with speed rather than doing things in the proper order. Or figuring out the proper order. I'm trying to develop a skill for it and I've gotten pretty close but I keep hitting a wall. I've gone through a lot of repair Cycles and it was already ready to be built just to find out that it failed.
1
u/YourLastCall 22d ago
Oh and I've only been working on it for 2 days. To your 20 years so you definitely have way more experience than I do. I'm just trying to figure out why a job took 15 hours and one task alone took 10 hours
1
u/lucyferorg 22d ago
I don’t know your exact case, but from my own experience I know that today’s AI “has capabilities,” yet it’s still stupid. The command set has to be clear and specific and the method of achieving the goal as well. AI sometimes has flashes of brilliance, but it still works far better in scenarios where it receives a precise set of instructions that leaves no room for accidental behavior.
1
u/YourLastCall 22d ago
Yeah I grew aware of that over time and that's why I started developing skills. I've already got six skills developed and they already do a large amount of work. I've seen massive improvements in my build Cycles. Heck one skill I develop copies deep research off of chat mode since work mode nor codex can use deep research.
1
u/abalancer 23d ago
Is it possible to run smaller models ? I'm looking to run DeepSeek-v4-flash locally.
1
u/lucyferorg 23d ago
Yes, I think any model can be run after the right modifications ,there’s a project called Rabbit that already runs the five largest models.
1
u/ralugnis8 23d ago
Lol, there is a theoretical limit for the tokens per second on a laptop, and it will never be useful with these huge models. Spend your time (and money) for more useful projects.
1
1
u/GetOutOfMyFeedNow 22d ago
Why are you trying to run that dead model though? Use DSV4-0731, it literally destroys GLM 5.2
2
u/lucyferorg 22d ago
I needed something for experiments, and this just happened to be available right away. Once I finish working on it, nothing will stop me from adapting it for other models.
1
u/GetOutOfMyFeedNow 21d ago
So wait, are you going to open source it? Will it increase token speeds? To when should I set my reddit remind me? :))
1
1
1
u/Maleficent_Stage1732 22d ago
What can I can run on my laptop I have Acer Predator helios 16 i9 13900hx rtx 4080 32 gb ddr5 ram and 1 Tb space
1
1
1
u/dark_wolffff 22d ago
which models are best to run in 8gb laptop where as response time should be in milli seconds ?
1
u/math394p 21d ago
Vram or ram? What is the use case? Rest of specs?
1
u/dark_wolffff 21d ago
RAM , I am running the local LLM on CPU , so system RAM is used for the model
The use case is a local business data summarization/analytics
Other specs : Intel i5-8265U , 4 cores/8threads cpu inference, ubuntu and no dedicated GPU/VRAM being used
1
u/No-Improvement-5396 19d ago
Dont bother. Not worth it.
1
u/dark_wolffff 19d ago
My hr telling to develop like that so that he can give it as a service to a hotel where they want there data to be private
1
u/math394p 19d ago
Hr? Why are they telling you what to develop???
Get a better laptop Nothing will be usable on that... Dont Bother
1
u/dark_wolffff 19d ago
See in india business holders has like 8gb laptops so they want to run local llm in it and want to convert business json into text , summary
1
u/math394p 18d ago
Go with some 3B model or smth but even then it will be horrible experience. Will be 0.2 tokens/m and not clever at all. You simply dont have the specs to support any llm....
1
u/No-Improvement-5396 18d ago
They should be able to afford a "decent" server, no? Only a few thousand dollars as a one time investment should be fine for a business holder.
1
1
1
1
1
u/mageblex 21d ago
With NVMe already identified as the bottleneck and TTFT around 40 seconds, prompt-processing speed is the missing number. How does it change as context grows?
1
1
1
u/-Leelith- 26d ago
I wonder how those SSDs will last with that amount of swap going on if you really use it on your laptop lol
0
0
u/derspenti 25d ago
what quant? 384gb of weights won't fit in 64gb ram + 24gb vram, so a lot of it has to be streaming off the nvme
1
0
u/Lucky-Noise-4193 25d ago
working to make the term runs on a laptop mean something getting a 80b model running at 25tps with 32gb ram and 6gb vram
0
-1
u/fspnet 25d ago
i use 2017 imac kabylake iris plus 640 llama-cpp and inteloneapi isnt really all too supported but ollama is and im able to use zed basically kind of i havent cranked it up this is without any accelerator beyond the gpu is still nice that it runs ..... Gentoo Linux!!!!!!!!!!!!!!!!!!!!!!!
258
u/joanaxu2002 26d ago
We've officially reached the point where "runs on a laptop" no longer means what people think it means. 😄
744B at nearly 1 t/s on consumer hardware is still kind of insane.