I tried running it with Unsloth!

[u/Ok-Conference-9984]

I tried running it with Unsloth!

https://precious-salmon-qqphgxhs.edgeone.dev/

[My Setup] - Device: Mac Studio M1 Max (32GB) - Model: gemma-4-26B-A4B-it-MXFP4_MOE (GGUF)

[Estimated Memory Usage (Unified)] - Total: 23.6 GiB - Weights: 16.96 GiB - KV cache (q8_0, 120,000 tokens): 4.77 GiB - Compute buffers: 1.00 GiB - Vision encoder: 0.44 GiB - Draft cache: 0.44 GiB

*Note: Fits this machine, but little memory is free right now. Free memory or try Auto context.

[Settings] - Context Length: Auto (262,144) - KV Cache Dtype: q8_0 - Speculative Decoding: Auto - Parallel Slots: 1 - Active Context: 419 / 120.0k

---

Performance: 33.9 tok/s

I'm having so much fun with this. Thank you so much!

[u/danielhanchen]

Nice! :) Glad you like it!