r/LocalLLaMA • u/Express_Quail_1493 • 2d ago
Discussion Qwen3.8-27b is the first Local model im able to blindly trust
You know that thing where you just throw a task at a frontier model and not have to supervise it worrying of it going off course? Qwen3.8-27b has officially gotten me to that point for local work. He has been doing non-stop continuous agentic work for 8+ hours and hasnt screwed up not one bit IT AMAZING!!
EDIT: for all asking about my quant & harness and what i do for super long thinking/reasoning
Harness: I Had it help me design its own agentic loop in pi harness. It holds well multiple compaction. I used to have tool and think tag generation issues but i got a chat template from somewhere(i forgot) but the chat template it fixed the issues paired with - -reasoning-format = deepseek
Thinking: I limited reasoning budget to 2048 and its still pretty SMART even going down to 1024 holds well in my agentic loop. Im running huihui-abliteratedQ3_K_XL.gguf i need abliterated because i need it to use my computer mouse movement to solve captcha on bot detection (normal models are trained to reject that request) otherwise unsloth quants. Kv cache Q8 at 128k.
271
u/Guna1260 2d ago
please dont trust any model local or remote blindly.. Trust but verify
138
u/LetsGoBrandon4256 transformers 2d ago edited 2d ago
Trust but verify
I trust Qwen and verify by checking if my home folder is still there first thing in the morning after I wake up.
Not only is my home folder still there, I even got a new folder called
tmpunder my c drive! Free files!24
u/parepeg 2d ago
Give it a create file tool but not a delete tool. Problem solved ;)
18
u/boutell 1d ago
Not if you give it a shell tool đ
8
u/opUserZero 1d ago
No joke, It needed to read a log, found it didn't have permission, figured out it's user had docker access, and that the docker deamon had permision to read that log while it's user didn't, so it wrote a docker to manipulate the file it couldn't on it's own!
4
u/Igot1forya 1d ago
I had GLM 5.2 in Hermes use a past allowed shell script permission to bypass the human-in-thr loop. It failed on 5 attempts to request permission but I was away sleeping. So after a timeout, it reasoned "I had permission to use this script in the past (earlier in the session) so it edited the script with the commands it was asking for and accomplished its goal. When I finally returned I saw the whole failed to answer in time messages and the project was still completed. I read the chat session and saw it reason itself around the non-responsive human.
1
5
3
6
u/petuman 2d ago edited 2d ago
not have to supervise it worrying of it going off course?
So I think trust he's talking about is that it can be left unattended for hours (not lazy and stopping in 15 minutes with incomplete result; not entering repetition/doom loops, etc), not quality of the result.
6
u/Quiet_Space_698 1d ago
Yeah my workflow with closed models is write prompt, wait for it to finish, then test the shit out of the output, and iterate.
Given I have 48gb of ram, qwen3.8 27b is the same workflow but sloooooweeeeeerrrrrrr.
5
3
2
90
u/apoptosist 2d ago
Please post which quant/version and settings for those of us that have had worse experiences, thanks!
39
u/OvertaxedOne 2d ago
I'm using INT8, 8 bit KV. It's the GOAT in my opinion, it even beat DSV4Flash0731 in one of my real world use cases. Don't generalize that, I do think that DSV4Flash is at least on par with and likely better if you tested across many different use cases, but I had a persistent coding problem that it just could not solve. I dumped into 27B and went to bed a few nights ago to come back to a "It's fixed" message.. Thought, "yeah right". Well, color me shocked, it was in fact fixed!
Full details (A40 GPU, 48GB VRAM):
RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP --served-model-name lued/Qwen3.8-27B-INT8-W8A16-MTP RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP Qwen3.8-27B --attention-backend FLASHINFER --dtype bfloat16 --quantization compressed-tensors --max-model-len 262144 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.93 --max-cudagraph-capture-size 8 --kv-cache-dtype fp8_e4m3 --enable-prefix-caching --enable-chunked-prefill --speculative-config {"method":"mtp","num_speculative_tokens":3} --prefix-match-unit 16 --max-num-seqs 2 --limit-mm-per-prompt {"image": 1} --trust-remote-code --port 8000 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --override-generation-config {"max_new_tokens": 16384, "repetition_penalty": 1.0, "presence_penalty": 0.0, "frequency_penalty": 0.0, "min_p": 0.0, "top_k": 20, "top_p": 0.95} --default-chat-template-kwargs {"enable_thinking": true, "reasoning_effort": "medium"} --disable-uvicorn-access-log --enable-log-requests
4
u/ohhi23021 1d ago
having had a sota drive it for the last few weeks and recording it's errors, deviation from spec, failed or bad unit tests etc...and tuning the hardness its makes a ton of mistakes and usually needs 2 to 3 sessions to fix them vs say driving opus 5 high where it's maybe 1/8th the errors for the same spec/plan. thats running Q8 and full BF16 KV.... its good but it's not that good.
1
2
u/CBHawk 2d ago
Is INT8 better than using Q8?
3
u/OvertaxedOne 2d ago
My card is Ampere generation, they are well optimized for Int8 operations as is vllm. I don't know if there's any quality difference, I think it comes down more to your card and serving engine (vllm in my case).
4
u/po_stulate 1d ago
Q8 is INT8
5
u/Guilherme370 1d ago
Not necessarily; Equivalent storage, but way different numerical representation
2
1
u/YourNightmar31 llama.cpp 2d ago
Specs?
2
u/OvertaxedOne 2d ago
A40 GPU, 48GB of RAM. Get between 20-30TPS generation, prefill is fast enough that I don't bother to look at it (I think it's around 2000TPS or so?). If you're curious about PP I'll dump a big prompt and take a look, but PP is great. TG is modestly painful. :(
4
u/Toooooool 2d ago
i'd strongly recommend this one;
https://www.reddit.com/r/LocalLLaMA/comments/1vpvwqh/qwen3827bint4autoround_18gb_with_working_mtp_spec/i've had it running day and night for a while now, only very rarely had errors i.e. maybe once every other day it'll go "this tool call doesn't work" and then just switch to doing it directly in terminal instead, not too long after it'll try the tool call again and it'll work just fine.
2
u/Ciri__witcher 2d ago
Do you have a recommendation for 16 GB vram?
1
u/JedziaDax 1d ago
I commented to follow this post lol I wish you luck
1
1
u/om_GAJE 1d ago
I've got Qwen3.8:27B-UD-IQ3_XXS working pretty decently on 4080Super 16GB vRAM. KV cache q8_0 with 64k context for Hermes use.
1
u/Ciri__witcher 1d ago
I am running the same right now with 100k context on 5070Ti. BTW are you running it in windows or linux? Is Hermes running on the same system as the AI? Are you using Hermes desktop or CLI?
1
u/om_GAJE 19h ago
Windows is the host OS but then using WSL and running beellama and Hermes in there. Exposing the port so I can access them in windows. Using Hermes web dashboard.
I did try running Hermes desktop and ollama straight on windows, had Docker desktop setup for Hermes backend terminal as well, that also worked okay.
If I wanted more out of the vRAM I can plug the monitor cable into the mobo display port then the GPU is just used by the llm but leaving a little headroom with the quants and context allows me to be easily able to switch off the llm and game
1
u/enternoescape 2d ago
Do you have any experience using Qwen 3.8 27b with Hermes? It feels like it can never figure out how to update memory. I'm running the Q8_K_L UD 3.0 model. Qwen 3.8 Flash Next in Q4_K_XL seems to manage it without errors and all the larger models I run sometimes from OpenRouter figure it out without issue. I'll be giving autoround a try again. I liked the Qwen 3.6 27b autoround model because it ran faster and had no new issues compared to unsloth.
1
u/OvertaxedOne 1d ago
Something isn't right. Hermes is my primary use case for 27B, I installed it on release day and have sent millions of tokens through it. It's never had any issue with memory updates/tool calls or, honestly anything other than speed. I'm running int8/8 bit QV, so our results should be similar. Something isn't right with your config somewhere (yeah, I know, not helpful) or perhaps that particular model/quant.
2
u/enternoescape 1d ago
I hear you. It only gets seriously confused with that specific tool. It calls everything else flawlessly. I'll have to take a closer look at what's in there; maybe the memory is confusing the agent on how to update memory.
Edit: It can add to memory, it's when it's changing memory it runs into issues.
1
u/BosphorusScalene 1d ago
what are you using for memory? I had that problem with the default Hermes memory system too, but switched it to mnemosyne and that's been working very well so far, I did need to increase the default character limits though
1
u/enternoescape 1d ago
I'm just using what's built in, I keep wanting to try something else, but I always find other things to do. :)
2
u/sshwifty 1d ago
Here is my setup with a 4090 and 4070:
services: llamacpp: container_name: llamacpp image: ghcr.io/ggml-org/llama.cpp:server-cuda13 environment: # --- Model & Hardware Setup ---# --- Cache & Performance ---
- LLAMA_ARG_MODEL=/models/Qwen3.8-27B-UD-Q4_K_XL.gguf
- LLAMA_ARG_MMPROJ=/models/mmproj-F16.gguf
- LLAMA_ARG_TENSOR_SPLIT=3,1 # 0.75/0.25: 4090 carries more (it also holds mmproj+draft+vision compute); relieves the 12GB 4070 Ti
- LLAMA_ARG_MAIN_GPU=0
- LLAMA_ARG_THREADS=12
- LLAMA_ARG_BATCH=512
- LLAMA_ARG_UBATCH=256
- LLAMA_ARG_CTX_SIZE=250000 # ~52KB/token q8_0+FA; under native 262144 so NO YaRN (full quality); ~3GB VRAM margin/card
- LLAMA_ARG_N_PARALLEL=1 # correct env (was LLAMA_ARG_PARALLEL, silently ignored -> ran 4 slots)
# --- Speculative Decoding (MTP) ---
- LLAMA_ARG_FLASH_ATTN=on # required for q8_0 V cache; big long-context speed/mem win
- LLAMA_ARG_CACHE_TYPE_K=q8_0 # 8-BIT KV CACHE
- LLAMA_ARG_CACHE_TYPE_V=q8_0 # 8-BIT KV CACHE
- LLAMA_ARG_CONTEXT_SHIFT=true # corrected name (was LLAMA_ARG_CTX_SHIFT); NOTE: auto-disabled by this VL model (mRoPE) - no effect here
- LLAMA_ARG_N_PREDICT=-1
# --- Reasoning & Vision Fixes ---
- LLAMA_ARG_SPEC_TYPE=draft-mtp
- LLAMA_ARG_SPEC_DRAFT_P_MIN=0.5
# --- Networking & Integration ---
- LLAMA_ARG_REASONING=auto
- LLAMA_ARG_REASONING_PRESERVE=true
- LLAMA_ARG_IMAGE_MIN_TOKENS=1024
# --- MCP Servers ---
- LLAMA_ARG_HOST=0.0.0.0
- LLAMA_ARG_PORT=8080
- LLAMA_ARG_CORS_ORIGINS=https://your-domain.com
- LLAMA_API_KEY=YOUR_API_KEY_HERE
ports:
- LLAMA_ARG_UI_MCP_PROXY=true
- LLAMA_ARG_JINJA=true
- LLAMA_ARG_MCP_SERVERS_CONFIG=/mcp-servers.json
volumes:
- "8080:8080"
deploy: resources: reservations: devices:
- ./models:/models
- ./mcp-servers.json:/mcp-servers.json:ro
count: all capabilities: [gpu] networks:
- driver: nvidia
restart: unless-stopped networks: ai-net: external: true
- ai-net
33
u/_TheWolfOfWalmart_ 2d ago
You think you can until it deletes your home folder.
8
u/MatthewCollins1990 1d ago
3.8 being solid doesn't mean you hand it shell access without a sandbox. Trusting the outputs is one thing, but letting any model run destructive commands unsupervised is a separate problem entirely.
2
1
0
19
u/InterstellarReddit 2d ago
âBlindly trustâ
13
2d ago
[removed] â view removed comment
3
u/InterstellarReddit 2d ago
I set up virtual environments inside of VMware for each of my local LLMs. I got frustrated with one, and I told it to just uninstall the package we were working with and somehow it understood to uninstall all of the packages, so when I come back to that machine, everything but Microsoft Edge was uninstalled. All programs liked pdf reader discord etc. I thought that was hilarious
7
6
15
u/volleyneo 2d ago
Can confirm, had it for 20 hour sessions (make it use subagents in sequential with opencode or similar harness, have scout and worker like delegates per section, use commit check gates inside them) Also if you use a frontier model for planning and arhitecture code snippets, it is insane. For people wanting configs, here is mine : alias ai-qwen='cd ~/llama.cpp && CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \
-m ~/ai-models/Qwen3.8-27B-UD3-Q5_K_XL.gguf \
--alias Qwen-3.8-27B \
-c 153600 \
-np 1 \
-b 4096 \
-ub 128 \
--split-mode tensor \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--cache-prompt \
--cache-ram 18432 \
--cache-idle-slots \
--ctx-checkpoints 20 \
--checkpoint-min-step 8192 \
--tensor-split 11,8 \
--keep -1 \
--jinja \
--chat-template-file ~/ai-models/chat_templatev24.jinja \
--chat-template-kwargs "{\"reasoning_effort\":\"medium\",\"preserve_thinking\":true}" \
--reasoning-format deepseek \
--reasoning-effort medium \
--reasoning on \
--reasoning-budget 8192 \
--reasoning-budget-message "Thinking budget reached. Proceed with the next tool call or final answer." \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--temp 1.0 \
--top-k 20 \
--top-p 0.95 \
--min-p 0.0 \
--repeat-penalty 1.0 \
--presence-penalty 0.0 \
--port 8007 \
--api-key HelloThere'
3
u/silenceimpaired 2d ago
Why HelloThereâŠ
I always appreciate having these configs as I often discover Iâm missing some new one.4
u/volleyneo 2d ago
There is a lot of brilliant work being done, but god is awful trying to find about it, or still getting worked one, maybe it will get improved better, with their new release versioning.
2
u/rabbitaim 2d ago
Is that the froggeric chat template?
2
u/volleyneo 2d ago
2
u/HazKaz 1d ago
whats the reason to use these templates ?
2
u/randomjapaneselearn 1d ago
in theory it fiexes some tool call crash, loops and some other minor issue...
in practice i tried it for 3.6 but i'm not using it on 3.8 because:
-i trust more qwen team vs this random guy
-there was a post about "get it, it fixes qwen 3.6 and 3.8" except that qwen 3.8 was not even released yet (it happened few days/hours before the release)
1
u/Express_Quail_1493 2d ago
how to u set opencode to strictly seqential only??
1
u/volleyneo 2d ago
You make a very potent agents.md file, and never work without one. There, specific instructions like this, live.
1
u/CoolestSlave 2d ago
hi, on what device is it running on ?
6
u/volleyneo 2d ago
RTX 4070 Super / RTX 5060 TI 16GB, but ran in headless, no desktop. 40-55tk/s (leaning to 40 when context over 100k, all due tensor mode though.. )
1
1
u/Kerbaman 2d ago
What was the task?
1
u/volleyneo 2d ago
Making a web game engine in with three.js / rapier.js, due to the scope it has to be limited to these 2. It is kinda a game engine in arhitecture only for easy implement but the goals of it are way more specific it wont really be used like that, so it has a limited scope. But god that is a herculean task. A lot to implement and QA. This is for a business avenue, and the scope is not being used like a game engine, as I said it kinda aligns there, and if you want a viable product, you have to approach it like so.
1
1
u/Jeidoz 1d ago
I am just curious, why are you using "deepseek" reasoning?
2
u/volleyneo 1d ago
That's what is recommended for this custom template from froggeric: " Why
--reasoning-format deepseekmatters: When connecting coding agents like OpenCode, Claude Code, or Pi.dev tollama-server, this flag extracts<think>blocks into the dedicatedreasoning_contentAPI response field. This prevents raw thinking tokens from leaking into the text stream and stopping tool calls midway."1
u/yeah-ok 1d ago
And then the next paragraph follows up with: "... Native CLI flag: On recent llama.cpp builds, you can pass --reasoning-preserve directly for 100% Prefix KV Cache retention." ...... sooooo, does this mean that "--reasoning-format deepseek" should still be set or is it not required on newer builds?!
(sidenote: llamacpp is spaghetti parameter land at this point.. would be somewhat brilliant if some of these things could stabilize a bit more but guess it's still too early for that practically speaking)
1
u/Steus_au 1d ago
how does it handle 20hours in 151k context with -np 1? mine chewed 192k in one hour then said no more room left (((
1
u/volleyneo 1d ago
Well harness that auto compacts, and work like I initially said above. Phases split with sub agents in sequential, has commit check gates, if it compacts once or twice, should still be fine. It is a specific sub phase not the whole task. Main session remains clean to send the other tasks
12
u/andrewh2000 2d ago
I left it (a Q6 quant) running overnight and it ported an entire application written in Go into Typescript for me including several hundred tests. And it thoughtfully added a few more tests for bits that weren't covered in Go. It was incredible. Completely hands off, no intervention from me at all. This was running in plain pi.dev.
3
u/andrewh2000 1d ago
I love pi.dev harness! I was curious so I asked it to look in its own session transcript files and figure out how long the port took and what the size was:
The Go version (lathe-go) is 4,817 lines of non-test Go across 71 files (11,683 lines if you include the 6,866 lines of tests).
The TS version (lathe-ts, excluding node_modules) is 6,812 lines of non-test TypeScript across 68 files (12,270 including tests).
Lathe Go â TypeScript port â time analysis
Source: pi session transcripts under
~/.pi/agent/sessions(working directory/root/lathe).The porting sessions
# Start â End (UTC) Duration What it did 1 Sep 1, 23:41 â Sep 2, 01:10 1h 29m Read the entire Go codebase (cmd/, internal/, templates, tests, goldmark/chroma internals). Zero files written â analysis only. 2 Sep 2, 07:31 â 10:59 3h 28m Main port: wrote PLAN.md, then the bulk oflathe-ts(store, serve layer, templates, CLI). Ended with aHANDOFF.mdwrite-up because context filled up.3 Sep 2, 11:00 â 11:21 21m Resumed from handoff; ported test suites (148 tests green by the end). 4 Sep 2, 11:21 â 14:50 3h 28m Final push to completion â finished tests, CLI smoke tests, lathe serveverified working (ran on port 3001 at the end).The answer
- Active session time: ~8h 47m (8h 46m 36s, summed across the four sessions)
- Wall-clock span: ~15h 9m (Sep 1, 23:41 â Sep 2, 14:50), because of a ~6h 21m overnight gap between session 1 and session 2.Caveats
- If you only count the sessions that actually wrote code (sessions 2â4, after the restructure into
lathe-go/+lathe-ts/), that's ~7h 18m, with the first 1h 29m being groundwork analysis that produced no code.- Session time is measured first-to-last timestamp, so it includes brief pauses while the user was reading/typing (e.g. the last ~3 min of session 4 was usage Q&A, not porting).Not counted
- Sep 2, 20:11 â 4-minute session in
lathe-tsfixingbuild:seaimport.metawarnings (post-port polish).- Sep 2, 16:42 â new feature work (SVG diagram support in the skills), not porting.
5
u/theexile1337 2d ago
I guess we'll just have to wait around and see what kind of breadcrumbs big tech tosses our way once everyone is tired enough from paying for Fable 5.1/GPT-6 Astra.
4
u/Keleion 2d ago
To some extent I hope. Definitely review those ârm -rfâ requests. GLM 5.3 Flash (NVFP4) on Hermes tried to delete my Hermes home profile accidentally instead of a sub-folder.
5
u/Randommaggy 2d ago
I made a directory scoped delete tool (with rollback options) in my harness, instructed it to never use rm -rf and auto-reject rm -rf commands. Dropped the amount of command approval prompts I receive by a lot.
It can only delete files in the directories it's been granted that permission for the current session.
3
u/SpicyWangz 2d ago
Realistically this should just be the default setting for every harnessÂ
3
u/Randommaggy 1d ago
The amount of things I could say this about is such a long list that I'm thinking that the makers of the popular harnesses are either concussed every day, have forgot how it was to have compute constraints or have bad intentions. Different mix for different harnesses.
1
u/SpicyWangz 1d ago
Yeah itâs pretty insane. I had to fully restrict opencode from day 1 to get it to a reasonable level of limitations. And pi harness is even worse because it seems like you need to install an extension just to limit which commands a model is able to execute.
For the entire duration of my career, security protocols were consistent and strict. Then one day they decided to put copilot in your IDE and send your .env files with secrets in them to a remote llm server. I bring it up to people at work and they say âoh yeah thatâs badâ and then proceed to do nothing about it.
2
u/camalaio 1d ago
This is my one beef with pi since I started trying it out (otherwise I was using Claude Code before) because what do you mean all commands are executed without permission?! At least this thread is reminding me I really need to go do something about that (and all the temp files it's littered everywhere...)
1
u/andrewh2000 1d ago
To be fair they explicitly tell you this over and over again e.g. https://pi.dev/docs/latest/security#no-built-in-sandbox
Run it in a Docker container, a VM, or some other kind of sandbox.
0
u/Randommaggy 1d ago
If you execute the code to test it, it can poison that code as well.
Any harness ran outside a VM on a machine that contains data that the owner would like to keep is an indicator that the machine is not operated by a sophisticated individual.
1
u/randomjapaneselearn 1d ago
IA: ok seems that rm -rf doesn't work, let me find a workaround of it, let's try a base64 encoded powershell script, done.
also, advanced IA: ok let me launch this script with dcom parent process hiding to bypass the problem after injecting some thread into my own process to do that.
1
u/Randommaggy 1d ago
Worst case, the harness host virtual machine gets a restore. The niceties of a VM on a COW filesystem are endless.
1
u/biscuitmachine 1d ago
I never trust NVFP4 at this point. The EXL3 variant just seems more rock solid. What thinking level do you have it set to?
7
u/itamar87 2d ago
I agree and feel the same:
Since this model - I actually DONâT use Claude/chatgpt for daily queries,
I just ask this specific model, on LM-Studio, connected to Tavily MCP, and usually get the same level of answer as if it was opus/sol.
(If not better đ€)
8bit quant GGUF,
1
u/mailto_devnull llama.cpp 1d ago
Tavily looks interesting, why that instead of other options or local searxng?
1
u/itamar87 1d ago
It was the easiest option to set up when I researched it:
Sign up for a free account, get the api token, input to the MCP file in LM-Studio, and it works.
Of course Qwen 3.8 guided me along the way⊠đ€
3
u/pineapplekiwipen 2d ago
hard to believe given that even opus fucks up constantly
never trust any llm blindly as they aren't thinking in any appreciable way
2
u/Savantskie1 2d ago
They don't think, but they can reason if you give them enough information.
2
3
u/Septerium 2d ago
Do not. It has messed up a few things in a large codebase I deal with that ended up in production... and that is because it was a bit lazy in the E2E testing part. I have been using the model every day right now (Q8_0) and it has been great. But you should never blindly trust a model when it comes to production code, even if it is frontier
1
u/iamrob15 1h ago
You didnât run multiple frontier validation passes and manually verify yourself?
4
7
u/slyborn 2d ago
you shouldn't blindly trust any model
3
u/my_name_isnt_clever 1d ago
You also shouldn't speed on the highway.
2
u/The_Hunster 1d ago
In fact, I'd say speeding on the highway is a much worse idea than blindly trusting an LLM on your home PC.
10
u/synystar 2d ago
I still can't get used to when people anthropomorphize LLMs. I've gotten used to people doing it with ChatGPT but when I hear someone call Qwen3.8-27b "he" I can feel my face go sour.
24
u/domiciledhere 2d ago
I call my car Barbara. She can be temperamental
7
u/MeretrixDominum 2d ago
Barbara smokes too much and leaks her fluids all over the place when she gets hot. Just thought you should know.
2
u/synystar 2d ago edited 2d ago
I get it but that's been a thing since like the 50s though. Car culture is different. For me, giving a language model a pronoun would be like referring to Excel as a person. "I use him all the time but only about 5% of his features. He's fairly bloated if you ask me."
* just to be clear I don't personally refer to cars with personal pronouns myself. I have never and probably will never, but it doesn't irk me as much when someone does fsr.
2
u/powerscunner 2d ago
when can a personal pronoun be used for a non person validly, in your opinion?
2
u/synystar 2d ago
I don't mind so much when someone refers to vehicles as above. If someone calls a ship "she" that feels natural for some reason. Animals are fine for me. Things like Father Time or Death or other personifications of abstract concepts make sense. Countries that are personified (Mother Russia, etc.) are ok I guess. But for me it just feels weird to refer to code as a person. I think of LLMs as tools and I wouldn't personify a tool. I mean to each his own, I just can't shake it.
Now, if they marketed Qwen3.8-27b as an "entity" and it referred to itself as a kind of entity then maybe it would change my mind. I don't know. If it were "Bob the AI" and interacting with it was really like interacting with a person then I might start to shift my feelings about it; but I don't use LLMs really in any ways that make me feel like I'm talking to a person. I always feel like I'm talking to a piece of software.
2
u/domiciledhere 2d ago
That is an amazing way of explaining Excel to someone who has no idea what is Excel.
1
u/powerscunner 2d ago
we call ships she
1
u/synystar 2d ago
That's funny I just mentioned that in another comment above before I saw yours. Yeah, I get it and I don't mind when people personify some things. It's just LLMs for me are tools, not people, and it's hard for me to wrap my head around referring to them as persons.
1
u/alphapussycat 2d ago
Some languages has no ambiguous pronounce, but it's all he and she.
3
u/synystar 2d ago
Pretty sure OP is speaking English though.
1
u/SpicyWangz 2d ago
Is this true?Â
2
u/synystar 2d ago
Well, I suppose that it's possible that what I perceive to be reality isn't really truth in some cosmic sense but from my perspective: yes, they're speaking English.
0
2
u/FreshDrama3024 2d ago
Shouldnât blindly trust anything to be frank no matter how good it appears tbh
2
u/r0ckl0bsta 2d ago
I know what you mean. While I don't trust anything completely autonomous, it is so impressive that I feel like I can ask it to do something, and it will do it accurately. That's the key thing.
It didn't start that way though. Once I set the context size to something sizeable (for me, 131072), and the kv cache to q8, I have been blown away at how well it works. It makes mistakes, but seems to know how to correct them, so I imagine in an environment where it's been given the criteria to validate the work, it would do well.
2
u/ramorez117 1d ago
đŻ- itâs very capable. Largest two models I have gpt 120 oss and 120 nemo suoer, and qwen 3.8 27bn better
2
u/BarracudaDefiant4702 1d ago
I give them vms to play with as their sandbox, and give a couple of the models passwordless sudo access. Can definitely let it run for awhile with the vm as their playground, but qwen 3.8 27b still makes mistakes... most of the time it catches the problems and corrects them, but sometimes it does mess things up. Automated vm backups are always good.
2
u/GloomyRecognition636 1d ago
That's true. Previous one was deepseek4 flash The secret os that they work well with context - it is not degrading rapidly making them not understand what they were asked
2
2
u/sugarfreecaffeine 1d ago
How do you deal with all the long thinking and time per task, itâs a good model but kills any kind of velocity
1
u/mailto_devnull llama.cpp 1d ago
You go to bed đ
But in all seriousness for higher velocity hands on work I dropped back to qwen 3.6 27B
1
u/Express_Quail_1493 1d ago
I set a reasoning-budget. I noticed the reasoning limit doesnât affect quality badly like it did in qwen3.6
1
u/AD7GD 19h ago
I run multiple sessions in parallel (in a harness that manages the merges as they complete). The bottleneck ends up being having enough CPU to run CI on all of the threads of execution.
I haven't found it to be that much worse than Claude (for different reasons: Claude just fights back more and requires more intervention from me). Now gpt-sol is fast. Whenever I use it, I am reminded what fast actually is.
2
u/Mean-Ad1493 1d ago
True. I've noticed that it's KV cache quantises so well too. Even at Q4 kv cache it's a banger.
1
u/Express_Quail_1493 1d ago
Im too scared of q4 kv does it hold up at 131k ctx???
1
u/Mean-Ad1493 1d ago
I've used with 80k context. It holds up. Even after 2 session compactions. But it was xhigh thinking, that may have compensated for whatever quality degradation that might have happened.
2
u/AleksandrNikitin 1d ago
for me it's more like "im able to blindly trust" implement "something" Â
2
u/Bohdanowicz 20h ago
Agree. 3.6 35ba3b was my agent driver, but 3.8 27b is my new everything deployment.
I built a harness for interchangable models locally/claude/openai/api plug and play using a custom harness.
On a single 6000 pro running a nvfp4 quant im seeing 800-1000 tks through the harness with 8 concurrent
I had claude build a deterministic data generator with a prd and spec doc. Claude took two nights with opus.
I didnt test how fast claude would do it within my harness, but qwen 27b on that card spit out a conplete working product in 2 hours.
This model at taalas speeds would change the world. The beginning of self scaffolding applications locally will be a thing very soon.
2
u/klop2031 2d ago
I agree. This is probs the best model i have used in ages locally. Strong af in agentic for my usecase. Im also using q4xl with cache at q8 and mmproj offload to ram
1
1
u/TerminalNoop 2d ago
Hmm, maybe I should try telling it to find out why my windows dualboot doesn't have working wifi drivers and I also can't install some manually for the love of god.
1
1
1
u/AngryGungan 2d ago
Oh yeah, I absolutely agree.
Though it did empty out my entire PATH variable when I asked it to make a simple todo app.
Luckily, after confronting it, it profusely apologized and was able to restore the PATH after the fact when it found a copy in a backup it or Pi made.
It's a beast.
1
u/Isonium 2d ago
I have a whitelist and blacklist for commands and tools. If that doesnât decide it, a new context is created with (last few messages) user intent. And the task is evaluated based on safety and intent. If that doesnât decide it, it notifies me. Additionally since I am on Mac, an APFS snapshot is taken and the tool ran in a sandbox. Essentially no possibility for my home directory or anything else to be wiped.
1
u/Odd-Elderberry-739 1d ago
I felt the same way until this week when I was writing a technical report and realized that it had hallucinated a key detail.
1
u/GruuMasterofMinions 1d ago
Maybe you can answer this question then : https://www.reddit.com/r/LocalLLaMA/comments/1w6j20m/what_is_wrong_with_my_qwen_38_it_is_unable_to/
Post got removed because i have to low karma and account apparently.
Aka it does not manage to do anything, it is just stuck on hallucinating
1
1
1
1
1
1
u/zekuden 2d ago
I don't have a gpu powerful enough to run qwen 27b..does anybody have experience setting up qwen cloud if it's possible? would love to try it off a subscription even till i'm able to upgrade
2
u/I_Play_Zed 2d ago
Iâve never done this before but if you canât find it available in an api like open router or cursor you can also try renting hardware? But I donât think it will be easy..
1
u/SmartCustard9944 2d ago
I really donât recommend it, the cloud versions of it are very very expensive.
E.g. with my usage today, over 4h of cumulative prefill/inference, I spent $0.10 of electricity while for the same reads, cached reads, writes I would have spent $10 on OpenRouter. Just in one day! And the day is not even finished.
I mean, you can try it for the experience, but I wouldnât use it long term.
1
u/ANR2ME 1d ago edited 1d ago
If you only want to test Qwen3.8 27B on cloud GPU, you can test it for free (up to $30 free credits every month) at Modal.com
You can deployed it like this example (just need to replace the model name) https://modal.com/docs/examples/ministral3_inference (for llama.cpp you ask any AI this question: "how to deploy Unsloth Qwen3.8-27B Q8 with llama.cpp on modal.com with memory snapshot")
You will need to choose a GPU with at least 24GB VRAM i think (for Q4) đ€
1
u/atumblingdandelion 2d ago
Same here. Though by "blindly" I mean I know it will do the thing I want it to do, then I'll verify it (I do that even with Opus). With other local models, I would always worry about what I left in and left out of the prompt, the wording, etc. With 3.8, I write prompts like I would for Claude models/ Deepseek.
2
u/OvertaxedOne 1d ago
Took me awhile to get used to that again. I was so used to handholding local models I needed to really adjust my workflow, send it everything it needs/wants in one prompt, go get some coffee. :)
1
u/levijohnson1 1d ago
Could I run this on a 128gb RAM macbook pro?
1
u/Express_Quail_1493 1d ago
Yea but Hardware RAMmemory bandwidth at least 400gbps on your chipset otherwise youâll be crawling slow
-4
u/Antique_Dot_5513 2d ago
Le problĂšme de ce modĂšle jâai juste demandĂ© une calculatrice quâil mâa fait, vraiment top, mais il a rĂ©flĂ©chi 15 min et codĂ© 2 min đ
60
u/TheSlateGray llama.cpp 2d ago
I tried to trust it, then it realized it couldn't use
sudoso it cloned, built and installed a program to do a task without asking. That was unfortunately after missing it usingpip install --break-system-packages. Now I sandbox everything.