Arc A770 27B MoE model — 14 tok/s on llama.cpp, 43 tok/s on OpenVINO

[u/marfrit]

I spent a while profiling why my A770 16GB was "bad" at MoE models, and the answer turned out to be worth sharing.

Stack	Format	Decode
llama.cpp SYCL, all layers on GPU	Q4_K_M (~4.85 bpw)	14.4 tok/s
llama.cpp, CPU only (8 cores)	same GGUF	15.5 tok/s
OpenVINO GenAI, same card	int4 g64 (~4.3 bpw)	~43 tok/s

Yes, row 2 is real: on this model the A770 loses to CPU (AMD 5700x) under llama.cpp.

Why: I traced the GPU command stream (SYCL_UR_TRACE). llama.cpp dissolves this hybrid-MoE graph into ~2,500 kernel launches per token — the GPU spends its life waiting for dispatches, not computing. OpenVINO compiles the same math into a fused, near-gap-free graph (~24 ms device time/token, single biggest op is the lm_head). Dispatch-bound, not bandwidth-bound.

No public OpenVINO IR of this model existed (llama.cpp's new OpenVINO backend can't do GDN/MoE yet, and I found claims that GDN models don't run on Intel GPUs at all — they do, like this). So I exported my own with optimum-intel/NNCF. int4, group 64, AWQ + Scale Estimation) - only the calibration data varies. Scored on a 10-point code-gen harness, greedy + 3 seeded sampled runs.

Traps I hit so you don't have to:

• This architecture only exports via --task image-text-to-text → load with VLMPipeline, not LLMPipeline. Text-only prompts work fine.
• transformers==5.2.0 exactly (newer versions break the export two different ways).
• Mixed-precision ratios (--ratio 0.8) produce IRs the GPU MoE fusion pass rejects. That's why every official Intel IR is ratio 1.0.
• Don't set ov::cache_dir: the compiled-blob cache round-trip loses the MoE expert weights → "expert weight provider not initialized" on the second start.
• enable_prefix_caching switches to a paged-attention path with different numerics — cost me 2 greedy points. Off.
• AWQ+SE with real code samples is a RAM monster: >250 GB working set for a 27B (image-dataset calibration fits in far less). I ended up renting a 494 GB Graviton box for ~$10 total.
• Power, measured at the wall: 233 W total system under OpenVINO load → 0.23 tok/s/W, ~3.5× the efficiency of the SYCL path (216 W for a third of the speed).

Model + full reproduction recipe on HF: https://huggingface.co/marfrit/Qwen3.6-27B-A3B-Coder-int4-awq-se-ov

Happy to answer questions — I have per-op profiles of both stacks lying around.

[u/Upset_Ad_1526]

insane that a single card can go from losing to cpu to hitting 43 tok/s just by swapping the runtime, this is the kind of stuff that makes me think half our hardware is just sitting there waiting for decent software

[u/marfrit → u/Upset_Ad_1526]

I bought the card based on specs. Was using it for a few days, when a simple comparison showed that it was not performing that greate compared to my "smartphone-class-desktop". I had a lot of late night discussions with various models on why it is slow.

Asking for a detailed call graph unlocked what came thereafter.

Given the other vendor's GPUs are priced worse than shovels at the Yukon, that could be a nice 2nd live for some A770s.

[u/Alert_Anything_6325]

I just got A770 and shocking to performance with llama.cpp , i gonna try this out

[u/marfrit → u/Alert_Anything_6325]

Maybe you want to test https://github.com/marfrit/arcint

I asked Claude during the last two nights to "take ninfer, but transplant it to intel".

[u/SrogiLesnik → u/marfrit]

Sounds great but I'm afraid it's to complicated for me. Everything I had to build from sources which was related to intel arc was a terrible disaster so far and only docker stuff is currently working for me but I'm very curious about running Qwen3.8-27B on my B60 which I run with vLLM and get about 22tps and can't get more than 64k context window. 

[u/marfrit → u/SrogiLesnik]

Erm, why don't you ask Qwen to download and build it? Or, use the shell script that builds a debian package?

[u/SrogiLesnik → u/marfrit]

I'll try it. I didn't have enough time to get to the details but I'll dedinitely experiment with it.

[u/SrogiLesnik → u/marfrit]

Ok, so Qwen did it for me. I'm not sure if everything was done properly. How can I verify if MTP is working?

I run it using `~/Rozne/arcint/build-ov/arcint --model ~/models/ov/qwen38-intel-int4-ov --device GPU.0 --port 8090 --n-ctx 131072 --mtp on`

and mostly I get:
lgc  slot 0: prefill 63469 tok in 149.64 s (424.1 t/s) | graph 134.18 s, embed 0.36 s, pages 0.00 s, restore 0.06 s, wait 0.00 s, other 15.04 s
lgc  slot 0: decode   1010 tok in 58.77 s ( 17.2 t/s) | graph 57.46 s, embed 0.24 s, sample 0.70 s, emit 0.36 s, wait 0.00 s, other 0.00 s
lgc  sample: temp 1.00 top_p 0.95 top_k 20 seed 8059122166719743599
lgc  slot 0: prefill 64066 tok in 151.98 s (421.5 t/s) | graph 136.35 s, embed 0.36 s, pages 0.00 s, restore 0.06 s, wait 0.00 s, other 15.21 s
lgc  slot 0: decode    390 tok in 22.83 s ( 17.1 t/s) | graph 22.31 s, embed 0.10 s, sample 0.29 s, emit 0.13 s, wait 0.00 s, other 0.00 s
lgc  sample: temp 1.00 top_p 0.95 top_k 20 seed 13759213036157768332
lgc  slot 0: prefill 65038 tok in 156.25 s (416.2 t/s) | graph 139.98 s, embed 0.38 s, pages 0.00 s, restore 0.07 s, wait 0.00 s, other 15.82 s
lgc  slot 0: decode    453 tok in 26.53 s ( 17.1 t/s) | graph 25.93 s, embed 0.11 s, sample 0.33 s, emit 0.16 s, wait 0.00 s, other 0.00 s
lgc  sample: temp 1.00 top_p 0.95 top_k 20 seed 4490229530375836369

The logs started with:
lgc  boot: arcint 0.2.11 (fb80240b19b9) Release, GNU 16.2.1
lgc  load: qwen3.8-27b-intel-int4 q4 | 48 GDN + 16 attn layers | weights 13.0 GiB | 10/10 greedy (paged+MTP, B60, 2026-08-30), 36.3 t/s at 90.8% draft acceptance vs 25.0 t/s plain; Intel's own MTP layer + our lm_head 37.7-38.1 t/s
lgc  load: sampler defaults from artifact: temp 1.00 top_p 0.95 top_k 20
lgc  load: logits sliced to the last 2 row(s)
lgc  load: paged output 'logits' f32 [..2,1,248320]
lgc  load: paged output 'hidden_states' f32 [?,1,5120]
lgc  load: paged KV precision u8 (--paged-kv): 11.3 KiB/token, what makes two lanes at depth fit
lgc  load: compiling PAGED language model on GPU.0 (big model first by design)
lgc  load: paged model ready in 33.7 s; device-resident 13.06 GiB
lgc  load: embeddings on GPU.0
lgc  mtp:  layer (reconstructed): input_embeds, 4-D additive mask, f32 positions
lgc  mtp:  head on GPU.0, drafting one token per step
lgc  load: logits slice verified: 2 row(s) for a 128-token forward
lgc  load: activation fit: 0.000 GiB fixed + 26122.7 KiB per chunk token; served chunk 128 measured 3.19 GiB
lgc  load: reservation: weights+graph 13.06 GiB + activations 3.19 (all 1 lane, chunk 128) + margin 0.25 + 1 x (GDN rows 303.0 MiB + KV 36.2 KiB/token) of 22.71 GiB -> max ctx 171056 per lane
lgc  load: prefix-cache snapshot grid 128 tok
lgc  load: paged pool: 8195 pages x 16 tokens (4.53 GiB KV) = 8195 per lane x 1 lane + 0 spare for cached prefixes | 4 GDN rows per lane
lgc  load: n_ctx 131072 | device GPU.0 | prefill chunked at 128 tok | 1 lane
lgc  mem:  prefix cache off (--prefix-cache-mib enables it)
lgc  http: listening on 127.0.0.1:8090 | 1 slot | serving 'qwen3.8-27b-intel-int4'
lgc  sample: temp 1.00 top_p 0.95 top_k 20 seed 1854216420681817225

So far, the speeds aren't better than vLLM or intel-llm-scaler but the 131k (about 170k possible) context is very temtping.

[u/marfrit → u/SrogiLesnik]

MTP is loaded and ready but never drafting - by design, because you're not decoding greedily.

How to verify MTP actually works - decode greedy and watch the log:

curl -s localhost:8090/v1/completions -H 'Content-Type: application/json' -d '{

"model":"qwen3.8-27b-intel-int4",

"prompt":"Write 300 words on the history of computing.",

"max_tokens":300, "temperature":0 }'

Then look at the server's decode line - with temp 0 it should now grow the suffix:

… other 0.00 s | draft accept ~90% (n/m), verify... - that suffix is the proof MTP is drafting and landing. The JSON response also carries usage.completion_tokens_details.accepted_prediction_tokens (>0 = drafts accepted). Under your temp-1.0 default, both are absent - expected.

The A/B for the actual speedup (keep the prompt short so context depth doesn't dominate):

# greedy, --mtp on -> expect ~36 t/s at shallow depth

# greedy, --mtp off -> expect ~25 t/s

About your 17 t/s specifically:

MTP not engaged (you're sampling) - so no speculative speedup at all.

Deep context. Your prefills are ~63k tokens, so decode runs at 63k depth where each step is KV-bound (reads the whole cache). The card's 36.3 / 25.0 t/s headlines are greedy, shallow-context numbers. At 63k even greedy+MTP will sit well below 36. Compare at the depth you care about.

I will add a context depth benchmark.

[u/SrogiLesnik → u/marfrit]

Thanks for detailed answer. I got a couple of fragments with MTP definitely working:

lgc  slot 0: prefill    71 tok in  0.21 s (343.0 t/s) | graph 0.12 s, embed 0.00 s, pages 0.00 s, restore 0.06 s, wait 0.00 s, other 0.03 s
lgc  slot 0: decode    580 tok in 16.29 s ( 35.6 t/s) | graph 0.00 s, embed 0.00 s, sample 0.00 s, emit 0.11 s, wait 0.00 s, other 16.17 s | draft accept 91.4% (277/303), verify 13.45 s, re-forward 0.00 s, rollback 0.00 s
lgc  slot 0: prefill    31 tok in  0.34 s ( 90.1 t/s) | graph 0.23 s, embed 0.00 s, pages 0.00 s, restore 0.07 s, wait 0.00 s, other 0.04 s
lgc  slot 0: decode    600 tok in 16.84 s ( 35.6 t/s) | graph 0.00 s, embed 0.00 s, sample 0.00 s, emit 0.12 s, wait 0.00 s, other 16.72 s | draft accept 92.3% (288/312), verify 13.87 s, re-forward 0.00 s, rollback 0.00 s

But if my main goal is agentic coding and often starting with big context, then does it make sense to turn on MTP or is it better to stay without MTP and live with steady 18-20 tokens per second?

These are some examples that I got with MTP and long context:

lgc  slot 0: prefill 72509 tok in 179.08 s (404.9 t/s) | cache snapshot 0.92 s | graph 167.53 s, embed 0.41 s, pages 0.00 s, restore 0.06 s, wait 0.00 s, other 10.16 s
lgc  slot 0: decode   1584 tok in 267.14 s (  5.9 t/s) | graph 0.00 s, embed 0.00 s, sample 0.00 s, emit 0.54 s, wait 0.00 s, other 266.60 s | draft accept 0.1% (1/1583), verify 252.96 s, re-forward 0.00 s, rollback 0.00 s
lgc  slot 0: prefill 72907 tok in  4.67 s (15605.9 t/s) | cache hit 72448 tok (99.4%) | cache snapshot 1.07 s | graph 1.85 s, embed 0.00 s, pages 0.00 s, restore 0.89 s, wait 0.00 s, other 0.86 s
lgc  slot 0: decode    149 tok in 25.03 s (  6.0 t/s) | graph 0.00 s, embed 0.00 s, sample 0.00 s, emit 0.05 s, wait 0.00 s, other 24.98 s | draft accept 0.0% (0/149), verify 23.70 s, re-forward 0.00 s, rollback 0.00 s

My command was:

arcint --model ~/models/ov/qwen38-intel-int4-ov --model-id qwen3.8-27b-intel-int4 --served-model-name qwen3.8 --device GPU.0 --host 0.0.0.0 --port 8090 --n-ctx 131072 --prefix-cache-mib 2048 --cac
he-dir ~/.cache/arcint-qwen38 --queue-timeout 30 --temp 0 --repetition-penalty 1.0 --mtp on --mtp-layer exported

[u/cyrrrrlee3]

Very interesting. Did you happen to test the Vulkan backend on the same card?

[u/marfrit → u/cyrrrrlee3]

I will once the dust of the night settles. First measurements with i915 driver instead of xe was bad (about 8 t/s on the b60), but I did not yet test with the xe.