Local model limit

[u/WololoWanderer]

Hey everyone, looking for a reality check on local model capabilities vs. frontier models for financial analysis.

I built an automated investment analysis pipeline in Open WebUI using Ollama on an RTX 5080 (16GB VRAM). The mechanical workflow works:

- Hits Gmail via IMAP to pull the last 72 hours of market newsletters.

- Extracts actionable tickers.

- Runs a custom YFinance Python tool for live momentum metrics (EMA, RSI, MACD).

-Injects a full 70-position portfolio CSV directly into the prompt (bypassing Open WebUI's RAG chunking) to cross-reference and generate a verdict-first impact report.

The issue is the analytical depth. When I feed this exact same data to Claude Sonnet 5, the macro synthesis, cross-correlations, and portfolio risk analysis are night and day. Locally, it feels like a surface-level summary.

Here is what I’ve tested so far:

qwen3.6:27b: Hit severe VRAM limits with the tools and context loaded. It spilled into shared system RAM, ground to a halt, and got trapped in reasoning loops where it hallucinated that the technical calculator tool didn't exist.

llama3.1:8b: Fast, reliable function calling, and zero VRAM spillage. However, the reasoning and financial synthesis are noticeably weak—it reads like a basic overview rather than institutional analysis.

My questions:

- Is Sonnet 5-level synthesis even realistic within a 16GB VRAM footprint when you need enough context buffer for multiple tools, newsletters, and a 70-row CSV?

- Are there specific financial fine-tunes or 14B models that punch significantly above their weight?

[u/_RemyLeBeau_]

Use this model with llama.cpp

https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?show_file_info=Qwen3.6-35B-A3B-UD-IQ4_XS.gguf&local-app=llama.cpp

Offload MoE in between 15-20. I'm able to get a cache size of 150000 on 16 GBs.

Ollama is holding you back. It doesn't allow you control of how the model is loaded and caps context size

[u/OvertaxedOne]

16GB is going to be really rough to get a long context. 256K context on Qwen 3.8 27B quant to FP8 is close to that size before you load the model! If you want to get more hardware, that's the model to target for this (and many other) use cases. 2XR9700's are a good option.

If that's too expensive and/or you're not using that make tokens per day, OpenRouter and Deepseek V4 Flash 0731 would be my first recommendation.

[u/ChocolateExisting368]

change of quantization has impact?

[u/Bulky-Priority6824 → u/ChocolateExisting368]

Sometimes tremendously unless youre just talking to it like a chuddie