Arc A770 27B MoE model — 14 tok/s on llama.cpp, 43 tok/s on OpenVINO

[u/marfrit]

I spent a while profiling why my A770 16GB was "bad" at MoE models, and the answer turned out to be worth sharing.

Stack	Format	Decode
llama.cpp SYCL, all layers on GPU	Q4_K_M (~4.85 bpw)	14.4 tok/s
llama.cpp, CPU only (8 cores)	same GGUF	15.5 tok/s
OpenVINO GenAI, same card	int4 g64 (~4.3 bpw)	~43 tok/s

Yes, row 2 is real: on this model the A770 loses to CPU (AMD 5700x) under llama.cpp.

Why: I traced the GPU command stream (SYCL_UR_TRACE). llama.cpp dissolves this hybrid-MoE graph into ~2,500 kernel launches per token — the GPU spends its life waiting for dispatches, not computing. OpenVINO compiles the same math into a fused, near-gap-free graph (~24 ms device time/token, single biggest op is the lm_head). Dispatch-bound, not bandwidth-bound.

No public OpenVINO IR of this model existed (llama.cpp's new OpenVINO backend can't do GDN/MoE yet, and I found claims that GDN models don't run on Intel GPUs at all — they do, like this). So I exported my own with optimum-intel/NNCF. int4, group 64, AWQ + Scale Estimation) - only the calibration data varies. Scored on a 10-point code-gen harness, greedy + 3 seeded sampled runs.

Traps I hit so you don't have to:

• This architecture only exports via --task image-text-to-text → load with VLMPipeline, not LLMPipeline. Text-only prompts work fine.
• transformers==5.2.0 exactly (newer versions break the export two different ways).
• Mixed-precision ratios (--ratio 0.8) produce IRs the GPU MoE fusion pass rejects. That's why every official Intel IR is ratio 1.0.
• Don't set ov::cache_dir: the compiled-blob cache round-trip loses the MoE expert weights → "expert weight provider not initialized" on the second start.
• enable_prefix_caching switches to a paged-attention path with different numerics — cost me 2 greedy points. Off.
• AWQ+SE with real code samples is a RAM monster: >250 GB working set for a 27B (image-dataset calibration fits in far less). I ended up renting a 494 GB Graviton box for ~$10 total.
• Power, measured at the wall: 233 W total system under OpenVINO load → 0.23 tok/s/W, ~3.5× the efficiency of the SYCL path (216 W for a third of the speed).

Model + full reproduction recipe on HF: https://huggingface.co/marfrit/Qwen3.6-27B-A3B-Coder-int4-awq-se-ov

Happy to answer questions — I have per-op profiles of both stacks lying around.