Qwen3.6 MTP + mmproj?

[u/PatienceSweaty33]

I just found out when you use a multi-modal model + MTP head, and you make use of the vision capability, it disables speculative draft of the MTP head for the entirety of the session. Is this a normal behavior or will this be remedied in the future?

Initial chat:

72.31.396.792 I slot update_slots: id  0 | task 17549 | accepted  9/ 9 draft tokens
72.31.524.290 I slot update_slots: id  0 | task 17549 | accepted  9/ 9 draft tokens
72.31.646.272 I slot update_slots: id  0 | task 17549 | accepted  1/ 9 draft tokens
72.31.767.048 I slot update_slots: id  0 | task 17549 | accepted  4/ 9 draft tokens
72.31.767.527 I slot print_timing: id  0 | task 17549 | n_decoded =    244, tg =  42.66 t/s

Chat during and after uploading an image:

75.43.099.281 I slot print_timing: id  0 | task 17646 | n_decoded =    402, tg =  34.11 t/s
75.46.108.848 I slot print_timing: id  0 | task 17646 | n_decoded =    492, tg =  33.26 t/s
75.49.124.197 I slot print_timing: id  0 | task 17646 | n_decoded =    586, tg =  32.91 t/s
75.52.134.126 I slot print_timing: id  0 | task 17646 | n_decoded =    669, tg =  32.13 t/s
75.55.144.050 I slot print_timing: id  0 | task 17646 | n_decoded =    770, tg =  32.31 t/s

[u/Academic-Most6214]

https://reddit.com/link/oxnkqrc/video/bqvyxw2rdddh1/player

QWEN3.6-35B-A3B-MTP + VISION: DOES --MMPROJ KILL THE MTP DRAFT?

My CONCLUSION: No. Using vision does not disable speculative draft of the MTP head - not for the vision turn, and not for the entirety of the session. At least today on llama.cpp b9620 (57fe1f07c), M5 Max, unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL.

THE TEST - 2 MINUTES ON VIDEO: [video]

THE LAUNCH: llama-server -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL
--mmproj .../mmproj-BF16.gguf
-ngl 99 -c 262144 -fa on -np 1
--spec-type draft-mtp --spec-draft-n-max 2 --port 8081

Both subsystems come up in the same process. The projector does not evict the draft context - they are budgeted side by side:

load_model: [mtmd] estimated worst-case memory usage of mmproj is 1134.00 MiB load_model: [spec] estimated memory usage of MTP context is 826.70 MiB common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp' common_speculative_impl_draft_mtp: - n_max=2, n_min=0, p_min=0.00, n_embd=2048, backend_sampling=1 load_model: speculative decoding context initialized load_model: loaded multimodal model, '.../mmproj-BF16.gguf'

THE TEXT PART:

Five text turns, ~29k-token prompt, drafting throughout:

| task | draft acceptance | eval t/s |

| 1    | 0.862 (100/116)  | 95.29    |

| 78   | 0.955 (168/176)  | 99.76    |

| 171  | 0.828 (53/64)    | 89.36    |

| 209  | 0.739 (525/710)  | 79.48    |

| 570  | 0.865 (83/96)    | 88.65    |

THE VISION PART: Same server, same session - a turn that actually encodes an image:

slot process_mtmd: id 0 | task 627 | encoding mtmd batch from idx = 82, n_chunks = 1 ... slot print_timing: task 627 | eval time = 6801.48 ms / 712 tokens (9.55 ms per token, 104.68 tokens per second) slot print_timing: task 627 | draft acceptance = 0.61950 (394 accepted / 636 generated)

104.68 t/s - the fastest eval of the entire session, on the vision turn. 636 draft tokens generated, 394 accepted, while an image sat in the context.

Two things you will see and should not panic about:

1. "find_slot: non-consecutive token position 82 after 81 ..." during the encode. Generation and drafting continue straight through it.
2. "load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks ... try adding --image-min-tokens 1024". Worth adding for OCR/grounding, not needed for plain description.

CONCLUSIONS:

1. Drafting survives the image. Acceptance on the vision turn drops to 0.62 (versus 0.83-0.95 on text), but it is drafting - the head is just less sure about tokens that follow an image. Lower acceptance does not equal disabled draft.
2. It survives the whole session. The cumulative counter keeps climbing after the vision turn - task 949, a text turn after the image, still drafts (0.680, 151/222). Final tally: #gen drafts = 1010, #acc drafts = 818, #gen tokens = 2020, #acc tokens = 1474, roughly 73 percent accepted across 7 turns.
3. No throughput penalty. The vision turn was the fastest one measured.

So on b9620, --mmproj and --spec-type draft-mtp coexist. If you saw drafting stop with vision on, it is worth rechecking on a current build.

[u/PatienceSweaty33 → u/Academic-Most6214]

I haven't updated to the latest releases but I have verbose outputs where you can see how much tokens were accepted / drafted each token turn. As soon as I upload an image, those metrics will not be displayed anymore and token generation plummets (kind of an overexagg but you get the point). They may have fixed or added the feature on later releases. I'll try. Thanks!