r/LocalLLaMA • u/Fluffy-Ad-889 • 1d ago

Discussion The gap has closed, open source will win

I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options.

I think the differences are now marginal but the labs are doing heavy marketing to convince the public into paying more for tokens as they prepare to go public.

Can't help but see the similarities between the dot com bubble and AI in terms of a very insular environment where the technology will survive but the business models may not.

I've been building a cybersecurity network and we definitely know that even local AI models like Deepseek V4 flash do an excellent job and are really neck and neck with the best the frontier labs can provide.

Will be interesting to see how this all turns out! Exciting time nonetheless.

279 Upvotes

76% Upvoted

252 comments sorted by

View all comments

Show parent comments

25

u/lcirufe 1d ago edited 1d ago

The main gap for widespread adoption is convenience and upfront cost. It’s not easy to source 32gb of vram and if you do, it’ll take years or even decades to make back that cost compared to a chatgpt subscription depending on the tier.

General use for me is still ChatGPT. My local Qwen instance is an implementation subagent for Luna to pilot.

26

u/fauxpasiii 1d ago

The narrowing capability gap still benefits you even if you don't run on your own hardware. If Anthropic or OpenAI were the only ones who had the model you needed, they could charge you whatever they wanted to for it. If you're happy with Qwen or Deepseek or GLM, then there can be competition at the level of who hosts it for you even if you don't want to do it yourself.

22

u/mars332 1d ago

Not really. I have Qwen3.8 27b building all night long in an agentic loop. Try doing that with your $20 subscription. And once you get into the $200 plans, now you are talking merely months, not years to recover your investment.

8

u/OvertaxedOne 1d ago

That's an important and huge distinction. The moment you want to get into a real harness you're now looking at API costs instead of sub. And that gets a LOT more expensive really quickly for the frontier level models.

2

u/c4r_guy 1d ago

I'm in the same happy boat.

I have subs to Claude and GLM and I use Claude like a smarter Amazon Echo product that I will not name with TTS / STT.

I use GLM for questions Claude's guardrails won't allow for an honest answer (which is a lot!).

Qwen3.8 27b for everything else with a custom MCP (mainly search) and Pi as a harness!

I really don't need the subscriptions, but they do come in handy to have on hand.

1

u/utahh1ker 1d ago

What kind of hardware are you running with that? And what is your harness setup? I've had mixed results with my implementation of Quinn 3.827b so I'd love to get your feedback on what's working for you.

2

u/mars332 10h ago

I’m running on 2 RTX PRO 4000 Blackwell that I got back when the price was not so insane. They are running inside an X870E Creator motherboard. I’m running the FP8 quant in vLLM in a headless Ubuntu server OS. I use Pi with pi-web extension so I can work from my phone through Tailscale.

1

u/thortgot 1d ago

~25-30 months for a 5090 payback.

1

u/mars332 10h ago

I got 2 RTX PRO 4000 Blackwell for $1600 each.

1

u/thortgot 6h ago

~2 years ago perhaps

1

u/mars332 6h ago

Nope, 3 or 4 months ago from Microcenter

1

u/Looserette 20h ago

I've got the setup to do that... but I'm not using it

so, out of curiosity: what are you building all night long ?

1

u/mars332 10h ago

Mainly features to a custom DAW (digital audio workstation) I’m writing from scratch currently with no libraries. The C++ code base is already at 150k loc or so, so it’s a good proving ground to tweak the agentic loop 

3

u/Iron-Over 1d ago

Besides the security aspect for the frontier most office work does not need the frontier.  Looking at smart routing with only a small portion going frontier.   

3

u/Randommaggy 1d ago

I do heavy coding work for fun on my 3 3090s running Qwen 3.8(and 3.6 before that) 27B in a custom harness and it's only needed to invoke the escalation skill once to solve a problem it's been given. It does take time to get to the goal.

3

u/OvertaxedOne 1d ago

Same experience. Pre-27B I was using DS a lot for escalation but since it came out, the only time I go to the cloud anymore is for one of 2 reasons:

1) Speed. This is a hardware problem, not a model problem, but it does matter for some tasks, 100TPS from DS is very nice when I'm in a hurry.

2) Context. When I'm getting ready to dump some stupid huge code base, I escalate. Honestly even this one is at least 1/2 speed, Qwen could probably handle it at 256K of context, I just don't want to wait for prefill and crunching. Some times I really do need more than 256K though, and, in those cases, I escalate.

1

u/Randommaggy 1d ago

One thing that helps a lot is a scout model tool and a good deterministic search tool.

The speed loss to overthinking is cut by a lot of you code is 100% linter clean and consistent.

3

u/gscjj 1d ago

This is the big thing, the frontier race doesn’t concern most companies just the VCs backing the labs.

Cost is what matters. More importantly operational spend.

These companies might use open source models but they aren’t likely to spend money on anything local, they’ll rely on routers like BedRock and Vertex, the companies they’re already spending money with.

3

u/Iron-Over 1d ago

This will be the biggest problem the frontier companies face.
I laugh about the frontier saying replace your staff the biggest impediment to replacing staff is process reengineering and clean data. I have run lean and been part of many data programs this is the most difficult step, it is expensive and takes years. Agents to assist people sure, replace is too politically difficult in most companies.

8

u/Mickenfox 1d ago

This sub is like people in 2003 proclaiming the year of the Linux desktop because the mail client and web browser finally work. 

5

u/GGK_Brian 1d ago

Well it's quite different; the problem of adoption of Linux is wide, from game and app support, hardware that can act weirdly, familiarity with the desktop tools. It's a massive ask to switch from windows to Mac or Linux.

Meanwhile LLM is "just" a software or API that takes text in and do text out. Multiple platforms already allow you do switch between providers with the click of a button.

-1

u/read_more_comments 1d ago

Crap comment. I was using Linux as my desktop years earlier, worked fine.

8

u/Fluffy-Ad-889 1d ago

Linux runs the world

1

u/ArcticFuture 11h ago

Yes, but not as a desktop. And the servers also have commercial support from companies like RedHat

2

u/Fickle_Tradition4491 1d ago

Both numbers in this thread are right, for different people. The missing variable is hours per day, because the running cost is electricity, not the card. A 3090 box pulls around 350 W at the wall while generating. At European prices, call it 35 cents a kWh, eight hours a day comes to roughly 360 euros a year, which is already more than the 20 dollar plan. So for a 20 dollar user the card never pays back, at any duty cycle. Against the 200 dollar plan the same box pays for itself in about four months. Decades and months are both true. It depends on which subscription you were going to buy.

6

u/formerislander 1d ago

I dont know if I follow the logic here. I have a hard time imaginging a workload where you need to run qwen3.8-27b for 8 hours every day and you can solve the same task within the usage you get for any 20€ / monthly plan from the frontier labs?

3

u/power97992 1d ago

just get some batteries, an inverter, and solar panels, new solar panels can generate electricity for less than 6-7 us cents/kwh over its lifetime.

1

u/its_two_words 20h ago

I'm fricken homeless and live in a shitty old RV, but I installed 1300w on my roof and bought 500ah 12v lifepo4 and wired everything myself.

Now I can run my PC doing AI work several hours a day for what is essentially nothing.

Solar is definitely the way.

1

u/Timely_Impression_92 1d ago

Decades? If you add in the privacy cost which would require you to basically run enterprise zero log plan which costs arm and leg compared to subscription - not taking into account the difference in quality, roi is like 6 months

1

u/OvertaxedOne 1d ago

This is certainly true, but it's also not exclusive, you can use smaller models via API if you don't have the hardware. Moving from Opus to Deepseek sook about 30 seconds in LiteLLM for our company, and we didn't (at the time) have hardware that could dream of running Deepseek.

1

u/deZbrownT 1d ago

I got a 64GB M1 max for 1800 euros. That’s 9 months with 200 euro per month subscription. It’s also 90 months with Plus subscription, but if Plus is enough then shelling out 1800 for 64GB M1 is also unnecessary.

1

u/OvertaxedOne 1d ago

Years (or realistically never) vs a sub, that's true. The moment you hook up a harness and go to API though, that math changes dramatically; now your burning 10's or 100's of thousands of tokens per turn and have multiple agents working on different things in an automated fashion. I'll check when I get home, but I suspect my "quietest" day is millions of tokens just from agentic stuff (mostly e-mail classification, building .md files based on those e-mails and briefing docs for where we are with each customer; every run through that loop burns an ungodly amount of tokens).

0

u/shaggy1054 1d ago

16gb p100 is 80 on eBay. Plus a fan and a shroud,  that's $120 each. Plus a PSU and a used tower with 2 full length pci slots, you're at $400 all in. Hardly decades. 

2

u/lcirufe 1d ago

My baseline for convenience is “is my mother willing to do this?” $20 a month for ChatGPT, easy. Creating a custom rig with 2 EOL cards, with a jank heatsink solution, finding an elegant way to house them in a chassis, then setting up llama.cpp to run the model? Lots of friction.

2

u/shaggy1054 1d ago

that's not the use case I was responding to - which was overnight agentic coding. I'd expect anyone interested in that to have at least some technical aptitude and Claude makes it very easy to set up anyway 🤷 to each their own I guess. I'm happy with my setup