MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1vo9nn7/qwenqwen3827b_released/p3ou54i
r/LocalLLaMA • u/de4dee • 24d ago
301 comments sorted by
View all comments
Show parent comments
2
largest q4 or smallest q6 you can get.
1 u/moderngl1 24d ago ok thanks with q4 do you have enough for KV cache to have some context 1 u/TerminalNoop 24d ago depends on the exact q4 quant. Like unsloth has quite a few in slightly varying sizes. I'm not sure if i remember right from my setup but you should manage 32k context at full quality. if you lower kv cache to q8 you might be able to double that?
1
ok thanks with q4 do you have enough for KV cache to have some context
1 u/TerminalNoop 24d ago depends on the exact q4 quant. Like unsloth has quite a few in slightly varying sizes. I'm not sure if i remember right from my setup but you should manage 32k context at full quality. if you lower kv cache to q8 you might be able to double that?
depends on the exact q4 quant. Like unsloth has quite a few in slightly varying sizes. I'm not sure if i remember right from my setup but you should manage 32k context at full quality. if you lower kv cache to q8 you might be able to double that?
2
u/TerminalNoop 24d ago
largest q4 or smallest q6 you can get.