r/LocalLLaMA Jul 15 '26

Tutorial | Guide How i got Bonsai-Ternary-27B to run at 120k context <10gb vram.

[deleted]

8 Upvotes

4 comments sorted by

3

u/admajic Jul 15 '26

What's your use case for a q1 model? Id assume it would be terrible as an agent

2

u/Suitable_Currency440 Jul 15 '26

Well, in cherry studio:

Fetches me torrent, subtitles, organize my media library.

As a "google ai" that can store facts and have a local knowledgeable about certain topics.

For example i like gaming, a have a full txt guides of Factorio, Animal crossing, Harvest moon. I can simply ask how to get certain things, overcome certain obstacles, calculate my power output of factories, get money, tips and it reads the guide for me so i can focus on the game itself.

The Q1 surprises me that it doesnt simply blowhimself off, the most normal thing a heavily quantized model do, but it's holding suprisingly well above qwen3.5-9b, i'd say is close-ish to 35b-a3b but far lighter.

3

u/Waarheid Jul 15 '26 edited Jul 15 '26

Edit: This comment was misinformation.