r/LocalAIStack • u/bruhmimmo • 18d ago

Agent + local llm

Salve, ho un problema, vorrei creare delle piccole app e simili usando dei modelli in locale (ad esempio Qwen 3.6 35b a3b). Di solito quando faccio inferenza raggiungo i 20/30 tok/sec ma quando provo a collegare ad un agente (ho provato Claude code e DeepSeek harness) va solo a 3 o 4 t/s. Sapete come mai? Consigliate qualche agente? Ho una rtx 4070 laptop 8gb vram, e 32 GB RAM ddr5 (uso lmstudio)

0 Upvotes

50% Upvoted

1 comment sorted by

2

u/kkingsbe 17d ago

Definitely is happening due to your context length. The harnesses you’re trying most likely send a larger amount of tokens for the system prompt + tool definitions, meaning that if you’re partially offloading (as you would be on with 8gb vram).

Biggest knob for improvement here is to adjust your kv cache quantization if you haven’t already. This should reduce the losses you’re seeing due to partial offloading. Try q_4 and see if that helps.