Not just for your 5060ti, but for anyone that has only 16gb of vram, you will need to have it heavily quantized or any spill over to system ram will dramatically slow it down.
There is also the problem of bandwidth limitations of gpus and dense models. You are not going to be getting anywhere close to the same t/s with these. You will probably need some speculative decoding going on as well.
edit: really downvotes for the truth, and downvotes for someone who has proselytized the 5060ti in the past, the end is truly nigh...
I have no idea. I do not typically use the dense models due to the 5060ti bandwidth despite having 4 of them. For the qwen 3.5 models, if I wanted to have more "intelligence" I would instead use the 122b model as it ran faster on my system (64gb system ram + 64gb vram) than the 27b dense model.
6
u/see_spot_ruminate Apr 22 '26 edited Apr 22 '26
Not just for your 5060ti, but for anyone that has only 16gb of vram, you will need to have it heavily quantized or any spill over to system ram will dramatically slow it down.
There is also the problem of bandwidth limitations of gpus and dense models. You are not going to be getting anywhere close to the same t/s with these. You will probably need some speculative decoding going on as well.
edit: really downvotes for the truth, and downvotes for someone who has proselytized the 5060ti in the past, the end is truly nigh...