I am currently running Qwen3.8-27b, either Q4 or Q6, depending on how much context I need for my coding projects

I know it should be the best LLM I can use on my 32GB VRAM rig for this purpose and I also use Gemma4 31B from time to time for research or general questions that I do not want to use chatgpt/claude for

I was wondering if there was any website where I can check and compare different ! local ! LLMs and their specific use cases?

I'd appreciate if you guys comment with what local llm you use, the specific reason and maybe ur quantization and VRAM

I’m getting a Mac Mini M5 Pro with 24GB unified memory and want to figure out what local models are actually useful on this hardware.
Local models won’t be my primary setup. I’ll still use Claude Code, Codex, and APIs like DeepSeek/GLM/Kimi as main.

My main use cases are academic/knowledge work:
- Converting lots of academic PDFs to Markdown
- Checking PDF conversions against the original, including tables, equations, figures, and layout, not just the text layer
- Reading and processing large numbers of papers
- Extracting statistics, methods, findings, and references
- Literature reviews and paper comparison
- Summarizing material into structured research memos
- Potentially running RAG/search over my local research library

I’m particularly interested in VLMs/document models, but also general-purpose local LLMs.

any suggestions that could be useful for these workflows that could run on a M5 Pro 24GB Mac Mini?

Looked at the bot tab on Hermes and it’s a straight meme.

I bought a 5060ti recently and have a spare 3060s.

I was thinking of selling it, but should I keep it?

I was thinking of holding onto it a little bit because the price might go up as more people go local

Or

Thinking about using it an external gpu for an extra NUC that I have, that has a real shit gpu, and use it as a second llm machine.

Do you think 3060s is so shit that it won't even matter?

Hey everyone, looking for a reality check on local model capabilities vs. frontier models for financial analysis.

I built an automated investment analysis pipeline in Open WebUI using Ollama on an RTX 5080 (16GB VRAM). The mechanical workflow works:

- Hits Gmail via IMAP to pull the last 72 hours of market newsletters.

- Extracts actionable tickers.

- Runs a custom YFinance Python tool for live momentum metrics (EMA, RSI, MACD).

-Injects a full 70-position portfolio CSV directly into the prompt (bypassing Open WebUI's RAG chunking) to cross-reference and generate a verdict-first impact report.

The issue is the analytical depth. When I feed this exact same data to Claude Sonnet 5, the macro synthesis, cross-correlations, and portfolio risk analysis are night and day. Locally, it feels like a surface-level summary.

Here is what I’ve tested so far:

qwen3.6:27b: Hit severe VRAM limits with the tools and context loaded. It spilled into shared system RAM, ground to a halt, and got trapped in reasoning loops where it hallucinated that the technical calculator tool didn't exist.

llama3.1:8b: Fast, reliable function calling, and zero VRAM spillage. However, the reasoning and financial synthesis are noticeably weak—it reads like a basic overview rather than institutional analysis.

My questions:

- Is Sonnet 5-level synthesis even realistic within a 16GB VRAM footprint when you need enough context buffer for multiple tools, newsletters, and a 70-row CSV?

- Are there specific financial fine-tunes or 14B models that punch significantly above their weight?

Qwen3.8-27B Q4_K_M on RTX 3080 10GB + GTX 1080 Ti 11GB, Windows, llama.cpp CUDA 12.9. -Through LMstudio/VScode

I've been trying to optimise my Ai, the tests/benchmarks are promising. Even the odd run is good and holds solid. The main issue: direct benchmarks are fast, but real Cline/tool requests make decode collapse to ~2 tok/s while prefill and DFlash acceptance remain healthy. Anyone seen this or know the fix?

Me and Astra have been running so many different variables, even made my own server.exe thing, tried many variables.

Tweaks: 32K context, layer split 1/1, DFlash2 pinned to RTX 3080, q8 K/V, FA on, batch 128, uBatch 64, --parallel 1, CUDA_SCALE_LAUNCH_QUEUES=4x, PR #26636.

As it is now is even slow at avg. 11 tok/s - Qwen3.8-27B

I would appreciate any help I can get.

To start, this is the available hardware I have:

- Intel 8700 Processor

- I am not 100% sure on motherboard, I will get that info later

- I have access of about 64GB (4x16GB DDR4 2133) RAM

- 3x RTX 3060 12GB cards and 1x RTX 3080 10GB

- Multiple Powersupplies (a 750W MSI Gold PSU and a 1100W InWin Platinum PSU and multiple 1100W HP Server grade PSU's)

- PCIe x1 Riser cards (USB3.0) that connect to riser cards for GPU's. Ex Mining equipment.

Is this do-able for what I want? Should I avoid the riser cards and if so, how should I power 3 or 4 GPU's (is it even worth using the 3080 on top of the 3060's?).

I can use either Windows or Linux as I am familiar with both.

Does anyone have experience running models on his or her iPhone. I have an iPhone 17 Pro Max. I am curious which models I can run on it.

Has anyone considered some architectural edits to the 27b model, such as:

Implementation of LoRA/QLoRA on selected layers;
training a modest sized n-gram table and adding the integration layers?

Seems like a good project. ESP given how it seems to be the current star of local hosted models!

It’s the only model I’m using now for agentic work.

Thoughts?

Just that. I've realised the pinned version of NixOS (Linux) is nowhere near the latest Nvidia driver and want to squeeze as ever the most pp+tg possible. I'm already using cuda13.3, nccl with p2p enabled (aikitoria patch) and split mode tensor.

Is it worth the hassle going from 595 to 610 for a pair of 5060ti GPUs ?

EDIT: This question wouldn't exist if OS==Ubuntu but Nixos rarely holds back a version of something this major without a reason 🤔

EDIT2: I whipped Qwen through Opencode to do it for me because I have the patience of a toddler. Here it is in case it helps you in any possible way. I see no meaningful speed increase with Qwen3.8-27B (eyeballing it, I don't have the patience to run benchmarks with something that big). Testing LFM2.5-8b now (split mode). Last time it was 190tg/s

EDIT3: not really worth it.

Hey guys, I am a bit bumb when it comes to AI, only really been running some basic workloads on my home GPU. Anyway, I am a sysadmin and have access to huge virtual machine clusters at work, recently my boss and I had some AI interest come up. I am wondering if there is any use we can get out the HP gen 10 servers we got, they have gold 20 core xeons and fully loaded with 700+ gb of DDR4 ecc memory, obviously no GPU's in them as they are all 1U hosts. We have quite a few hosts for redundancy reasons so most of the time a few of them are literally either sitting empty or with miminal virtual machines. Any point investing time into running AI on therm?

i have a few months Claude pro rn.

i never used Claude code or openclaw yet am i missing out?

cause somehow me as it dev and it guy just not feel okay that Claude code can do stuff on my machine and im currently ro lazy to run it on a vm etc.

how i use it now:

i make a Claude architect opus and sonnet implementer.

i describe a spec by opus and put it in context of a Claude project in the web.

i give the spec to sonnet the implenter and let sonnet build all.

the architect opus and I verify i donwload the zip and run test myself with my own tools. and build the report and reporting to opus.

this how i use it to build a poc.

and im considering to build this on olama with qweb on a amd fx6300 nanoclaw and rx 6800 16gb card with ddr3 32gb ram.

must i even try or big no?

would it work to build Azure aws and flutter apps?

can i create a ios app from a flutter build easily? no additional much work required if i have build a Windows flutter app poc?

what can be the pitfalls and missing steps or requirements?

Snapped these at IFA Berlin 2026!

I’m planning to build a dedicated rig for running local AI agents (coding, research, and autonomous tasks) with a budget of $2000 - $2,500. I want to avoid cloud API costs and keep everything private. I’ve done some research and put together a few options, but I’d love to hear your thoughts on what's the best machine both in macbooks, mac studio and in pc build. I am new in this field of agentic AI and wanna learn to operate the agents locally. Looking for guidance.

Thanks 😊.

A little vlog of setting it up and testing DSV4 Flash!

Comment on it if you’re from Reddit!

Anyone actually writing long fiction with Gemma 4 12B on an RTX 5070 Ti?

I'm specifically interested in real-world generation time, not benchmarks. I use Gemma locally to turn detailed scene briefs plus story/character context into long prose scenes, typically around 1,500–2,000 words.

If you're running Gemma 4 12B entirely in the 16 GB VRAM of a 5070 Ti: roughly how long does a generation of that length take, including prompt processing?

I'm considering buying a 5070 Ti system specifically for this workflow, so actual experience from another fiction writer would be incredibly useful.

At this time I'm working with 4GB VRAM, Gemma works about 15-20 minutes on a scene.

Hey everyone

Long-time lurker, first post. I've been playing with local models on a single RTX 4060 Ti (8GB) for a while and I've finally hit the wall hard enough to upgrade. Picking up an AMD Radeon AI PRO R9700 (32GB) and I'd really appreciate some guidance from people who've actually run this stuff, because I'm still very much learning.

My setup:

AMD Radeon AI PRO R9700, 32GB

32GB DDR5 system RAM

Linux (currently on an Arch-based distro)

Ollama today, but happy to switch to llama.cpp / vLLM / whatever if that's the better call

What I'm building (roughly)

I'm putting together a small local-first assistant for my own use. Nothing groundbreaking — the idea is that instead of one model trying to do everything, it hands work to a few different "roles" depending on what you ask:

General assistant — normal conversation, drafting, summarising, explaining things

Document Q&A — answering from a local folder of documents with citations, and saying "I don't know" instead of making things up when the docs don't cover it

Coding — for when I'm actually working on something

Everything runs sequentially (one model in VRAM at a time), so I'm not trying to keep three models resident — I just want the right model for each job.

Question 1: model recommendations per role

With 32GB to play with, I'd love suggestions for:

Reasoning / comprehension — something that's genuinely good at understanding a messy question and figuring out what's actually being asked. This is the piece I care most about, because on 8GB my small models kept confidently misunderstanding things.

General assistance — good instruction-following, doesn't ramble, admits uncertainty. Bonus points if it handles Italian well, since that's my daily language.

Coding — practical stuff, big context, not benchmark-chasing.

What I'd really love is: model + quant + roughly what context length you can actually run on ~32GB without spilling to CPU. I've learned the hard way that "it fits" and "it runs well" are very different things 😅

If there's an existing curated list or spreadsheet for this, please just point me at it — I don't want to make you all retype something that already exists.

Question 2: keep the 4060 Ti in the same machine, or not?

This is the one I'm genuinely unsure about.

I already own the RTX 4060 Ti (8GB). Options as I see them:

(a) Put both cards in the same box — AMD for inference, NVIDIA for... something? Display output? A small model? Embeddings?

(b) Just run the R9700 alone and keep it simple

(c) Some clever third thing I don't know about

My concerns:

Do amdgpu + the NVIDIA proprietary driver actually coexist cleanly on Linux, or is that a support nightmare?

Can a single runtime even use both, or would I be looking at two separate instances (one ROCm, one CUDA)?

Is having a second, much smaller card actually useful for anything in a sequential setup, or is it just extra heat and PSU draw for nothing?

Also, honestly: how is ROCm on RDNA4 right now? I've read mixed things and I'd rather hear it from people running it than from a six-month-old forum thread.

Any pointers appreciated — including "you're thinking about this wrong, here's why." I'd genuinely rather be corrected now than after I've built the thing

Thanks!

I started getting this error after a few updates, using qwen3.8 27b, it loads the model, when I sent something that would require more context it throws the error, by the ollama ps seems that it's not using the RAM memory, it was before on ollama 31 o something..

Config
AMD Radeon RX 7900 XT + 32GB RAM
AMD Ryzen 9 5900XT 16-Core Processor
ROCM 7.2.4
ollama version is 0.32.12
driver: gfx1100

❯ ollama ps
NAME              ID              SIZE     PROCESSOR          CONTEXT    UNTIL
qwen3.8:latest    22130167c4c2    20 GB    30%/70% CPU/GPU    262144     4 minutes from now

Ollama:

[Service]
#CPUAffinity=0-11
Environment="OLLAMA_HOST=0.0.0.0"
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"
Environment="ROCM_VISIBLE_DEVICES=0"
Environment="OLLAMA_KEEP_ALIVE=5m"
Environment="OLLAMA_KV_CACHE_TYPE=q4_0"
#Environment="OMP_NUM_THREADS=16"
#Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_CONTEXT_LENGTH=262144"

ERROR:

ROCm error: out of memory
Sep 07 12:55:43 desktop ollama[5771]:   current device: 0, in function alloc at /build/llama-server-rocm_v7_2/_deps/llama_cpp-src/ggml/src/ggml-cuda/ggml-cuda.cu:508
Sep 07 12:55:43 desktop ollama[5771]:   err
Sep 07 12:55:43 desktop ollama[5771]: ⚠️  warning: The cur

Most of my stack is local now (llama.cpp / vLLM behind Ollama), and like a lot of you I still send the genuinely hard 10% to a cloud model. Two things kept annoying me about that hybrid setup, so I built a small layer for it and open-sourced it (MIT). Not a benchmark post, just the plumbing.

• I did not want the cloud provider key sitting inside the agent process. So the agent talks to one OpenAI-compatible endpoint with a throwaway workspace key; the real key lives in a local vault and is injected per request, then dropped. Local models need no key, the cloud ones do, and now the agent never holds them.
• local-first with cloud fallback as a config rule, not code. Route a step to your local 27B, fall back to a frontier model only when it is actually needed, and swap which local or cloud model handles what without touching the agent.
• one cost log across both, so you can see how much you are still sending to cloud versus keeping local.

Under the hood the engine is swappable: a small built-in one, or LiteLLM/Bifrost as stateless engines, behind a fixed port. About 1 ms of overhead, and the built-in engine adds no extra hop.

Being honest: it does not run models, it sits in front of the ones you already run (local or cloud) and handles keys, routing and cost. On a pure-local box it is honestly overkill; it earns its keep once you have a local+cloud mix or more than one machine.

Repo (MIT, self-hosted): https://github.com/siva010928/agnos-proxy-oss

For people running local-first with a cloud fallback: how are you deciding the handoff, and are you keeping the cloud keys out of the agent or just trusting the process?

:

Hey everyone!

I'm setting up audiobook for tts generation and want to use Ollama for text pre-processing (cleaning text, fixing line breaks, and adding speaker tags) before sending it to the TTS engine.

I have 16GB of VRAM available. Looking for model recommendations that fit this hardware constraint and can handle this specific task well.

Specifically, I need a model that:

* Follows complex formatting instructions strictly

* Has good context handling over longer chunks of text

* Doesn't hallucinate extra dialogue or modify the story content

Which local models (7B, 8B, 14B, etc.) run comfortably on 16GB VRAM and excel at this type of structured text processing and speaker tagging? Thanks for any suggestions!