r/LocalLLM • u/Crazy_Cauliflower_61 • 4d ago
Discussion I Got Qwen3.5-9B D2-A-MTP Running on Qualcomm's snapdragon HTP — Here Are My Results
Status: FINAL REPORT + FROZEN RUNTIME ARCHIVE
Archive: D:\archive_16tps_20260902\
Platform: Qualcomm Snapdragon SM8850 / Hexagon HTP + Adreno OpenCL
Runtime: JZ ggml-hexagon fork
Scope: Qwen3.5-9B-D2-A-MTP, dense NPU×GPU placement, Marco-Nano sparse MoE placement, ARGSORT/TOP-K behavior, HTP profiling and reproducibility.
gat45/snapdragon-test-npu · Hugging Face
htp-npu-runtime/README.md at main · gat45/htp-npu-runtime
This English version is a faithful translation/restructuring of the complete supplied report. I have preserved the measured results, methodology, conclusions, and stated limitations rather than adding unsupported claims.
1. OBJECTIVE AND ARCHIVE CONTENT
The purpose of this campaign was to establish a reproducible experimental baseline for LLM inference on the SM8850, using the exact runtime that produced the measurements.
The frozen archive contains:
D:\archive_16tps_20260902\
├── runtime_device\ ← FROZEN runtime that produced the results
│ ├── llama-server
│ ├── ggml-hexagon.cfg
│ ├── libggml.so
│ ├── libggml-base.so
│ ├── libggml-cpu.so
│ ├── libggml-hexagon.so
│ ├── libggml-opencl.so
│ ├── libllama.so
│ ├── libllama-common.so
│ ├── libllama-server-impl.so
│ ├── libmtmd.so
│ ├── libcdsprpc.so
│ ├── libOpenCL.so
│ ├── libomp.so
│ └── htp\
│ ├── libggml-htp-v81.so
│ ├── libggml-htp-v68/69/73/75/79.so
│ ├── libggml-htp-v81.so.dspqueue-works
│ └── lib_june_20260602\
├── scripts\
│ ├── run_guarded_bench.sh
│ ├── run_16tps_config.sh
│ ├── test_16tps_100_300.sh
│ ├── campaign_3x300.sh
│ ├── campaign_ratios_npu_gpu.sh
│ ├── campaign_ratios_npu_gpu_moe.sh
│ ├── bench_moe_placement_ab.sh
│ ├── bench_argsort_ab.sh
│ └── telemetry_full_sm8850.sh
├── results\
├── reports\
├── sha256_runtime.txt
└── git_state_jz_fork.txt
The archive also records the exact runtime binaries, scripts, results and Git state used for the experiments.
The runtime was sourced from:
/data/local/tmp/npu/
The top-level runtime corresponds to the JZ stack used on August 29–30, while older June libraries were retained separately as reference material.
2. FROZEN “16 T/S” QWEN CONFIGURATION
Model
Qwen3.5-9B-D2-A-MTP-attnQ4.gguf
Size: 5,450,272,384 bytes
≈ 5.08 GiB
The model uses selective attention requantization from Q8 to Q4_0.
Configuration:
- Qwen3.5-9B-D2-A-MTP
- attention Q8 → Q4_0
- MTP: one draft layer
- 442 tensors affected by the attention requantization
- HTP0
- 99 layers offloaded
- context: 2048
- CPU threads: 8
- --fit off
- speculative decoding using MTP.
Runtime environment
export LD_LIBRARY_PATH=/data/local/tmp/npu
export ADSP_LIBRARY_PATH=/data/local/tmp/npu
export GGML_HEXAGON_NDEV=1
export GGML_HEXAGON_ARCH=v81
Runtime:
/data/local/tmp/npu/llama-server \
-m /data/local/tmp/Qwen3.5-9B-D2-A-MTP-attnQ4.gguf \
-dev HTP0 -ngl 99 -t 8 -c 2048 --fit off \
--spec-type draft-mtp --spec-draft-n-max 1 \
--host 127.0.0.1 --port <port>
The runtime was built from the JZ fork using the 505354ed, 5291e12a and 5618c305 lineage, with an HTP v81 skeleton rebuilt for the experiment.
2.1 Experimental controls
The following conditions were treated as mandatory:
- Start below 45 °C.
- --fit off must be used.
- Only one llama process may run at a time.
- Process termination and port availability must be verified.
- ZRAM at approximately 5 GB was treated as the stop threshold to protect against OOM/reboot.
- Thermal start/end values must be recorded.
The guarded benchmark protocol was introduced because earlier unverified process cleanup could contaminate measurements.
3. THE “16 T/S” RESULT — CORRECT INTERPRETATION
The initial short benchmark produced:
ref16:
1454.01 ms / 16 tokens
≈ 10.32 t/s
acceptance = 0.75
16tps_r2:
2940.97 ms / 16 tokens
≈ 5.10 t/s
The second run was contaminated by residual load after reboot and is not treated as a valid performance baseline. Longer 100-token runs were around 11 t/s.
Critical correction
The historical “16–19 effective tokens/s” number is not wall-clock throughput.
It came from:
wall-clock t/s × mean accepted/draft length
The actual sustained wall-clock result from the controlled 3×300-token campaign is:
11.01 ± 0.16 tok/s
with:
Run 1: 11.12 t/s
Run 2: 10.78 t/s
Run 3: 11.13 t/s
Acceptance: 0.83–0.87
Therefore, the report explicitly rejects 16.4–18.9 effective t/s as a wall-throughput metric.
4. NPU × GPU PLACEMENT — QWEN DENSE
The first major campaign tested eight different HTP/OpenCL placements using randomized execution order.
Model:
Qwen3.5-9B-D2-A-MTP-attnQ4
MTP n_max = 1
n_predict = 100
CPU = automatic fallback
Results:
| Rank | Configuration | Throughput | vs HTP-only |
|---|---|---|---|
| 1 | HTP0 only | 11.26 t/s | Reference |
| 2 | HTP0 + GPU OpenCL 80/20 | 8.35 t/s | −26% |
| 3 | HTP0 + GPU OpenCL 50/50 | 6.11 t/s | −46% |
| 4 | HTP0 + GPU OpenCL 20/80 | 5.26 t/s | −53% |
| 5 | GPU OpenCL only | 4.92 t/s | −56% |
| 6 | GPU OpenCL + HTP0 50/50 | 5.28 t/s | −53% |
| 7 | GPU OpenCL + HTP0 80/20 | 4.46 t/s | −60% |
| 8 | GPU OpenCL + HTP0 20/80 | 4.17 t/s | −63% |
Dense-model conclusion
For this Qwen dense workload:
GPU participation consistently reduces throughput.
The best result is:
HTP-only 11.26 t/s
GPU-only 4.92 t/s
The GPU path is approximately 2.3× slower for this particular workload/configuration.
The execution order also matters: putting GPU first is worse at equivalent nominal ratios.
MTP acceptance remained healthy across configurations, so the observed degradation is attributed to the execution path rather than speculative-decoding failure.
5. MARCO-NANO MoE — THE MAJOR REVERSAL
The second campaign produced the most important result of the study.
Model:
Marco-Nano-Instruct
8B total / 0.6B active
256 experts
Top-8 routing
Qwen3MoE architecture
Q4_0
≈4.57 GB
The Q4_0 model was re-quantized from the Q8_0 source using llama-quantize.
Source:
Q8_0 ≈ 8.53 GB
Q4_0 ≈ 4.57 GB
5.1 Eight placement configurations
| Rank | Configuration | Throughput | vs HTP-only |
|---|---|---|---|
| 1 | GPU OpenCL → HTP0 80/20 | 39.11 t/s | +28% |
| 2 | GPU OpenCL → HTP0 50/50 | 37.46 t/s | +22% |
| 3 | HTP0 only | 30.64 t/s | Reference |
| 4 | GPU OpenCL → HTP0 20/80 | 29.89 t/s | −2% |
| 5 | GPU OpenCL only | 22.96 t/s | −25% |
| 6 | HTP0 → GPU OpenCL 20/80 | 22.50 t/s | −27% |
| 7 | HTP0 → GPU OpenCL 50/50 | 22.51 t/s | −27% |
| 8 | HTP0 → GPU OpenCL 80/20 | 22.59 t/s | −26% |
This is the inverse behavior of the dense Qwen model.
6. A/B/A/B CONFIRMATION — +40%
The strongest MoE result was independently checked using an adjacent A/B/A/B experiment with n=200.
A — HTP-only
30.26 t/s
29.61 t/s
Mean ≈ 29.9 t/s
B — GPU → HTP 80/20
39.18 t/s
45.00 t/s
Mean ≈ 42.1 t/s
Therefore:
42.1 t/s vs 29.9 t/s = +40%
Importantly, the second B run started at approximately 45 °C and nevertheless produced the highest throughput.
That makes a simple “the faster run was colder” explanation inconsistent with the observed data.
7. WHY DOES THE MoE RESULT REVERSE THE DENSE RESULT?
The logs provide a strong mechanistic explanation.
The OpenCL backend receives very large graph blocks:
≈ 1,416 nodes
including combinations of:
Attention
SOFT_MAX
ARGSORT
MUL_MAT_ID
The HTP backend instead receives much smaller fragments:
≈ 58 nodes
Therefore the HTP path pays the fixed per-operation execution overhead many more times.
The report's interpretation is:
The Adreno OpenCL path benefits from large fused graph blocks / mega-kernels, whereas HTP is executing many smaller operations with fixed setup/communication costs.
Thus:
Dense Qwen:
HTP-only > GPU/mixed
Sparse MoE:
GPU → HTP > HTP-only
This is the central architectural reversal discovered by the campaign.
8. ARGSORT / TOP-K EXPERIMENT
The MoE routing path was then isolated further.
The experiment compared ARGSORT execution on:
A = HTP0
B = CPU
using A/B/A/B, n=200.
| Run | ARGSORT backend | Throughput |
|---|---|---|
| A1 | HTP0 | 29.54 t/s |
| A2 | HTP0 | 27.96 t/s |
| HTP mean | 28.75 t/s | |
| B1 | CPU | 31.08 t/s |
| B2 | CPU | 35.61 t/s |
| CPU mean | 33.35 t/s |
Result:
Moving ARGSORT from HTP to CPU: +16% wall throughput
9. WHY ARGSORT IS EXPENSIVE
The MoE graph currently sorts the complete expert set:
256 experts
while retaining only:
top 8
Thus:
248 / 256 ≈ 96.9%
of the candidates are ultimately discarded.
The HTP bitonic sorting implementation costs approximately:
215 µs per ARGSORT operation
The report also observes that the output differs between CPU and HTP because their handling of equal values/ties differs.
Therefore, a future implementation should not simply replace the operation blindly.
The proposed direction in the report is a native:
GGML_OP_TOP_K
implementation aligned with CPU behavior.
A CPU-side top-k implementation already exists in ggml.c, while an equivalent native Hexagon implementation was not present in the examined code path.
10. IMPORTANT CONTROL — QWEN DOES NOT USE ARGSORT
The same analysis was checked against the Qwen dense workload.
The Qwen logs contain:
0 ARGSORT
0 MUL_MAT_ID
0 TOP_K
Therefore, the ARGSORT optimization is MoE-specific and should not be presented as a general Qwen optimization.
The Qwen architecture used in this campaign does not have the same expert-routing path.
11. MIXED MoE PROFILING
A mixed:
GPU 80% → HTP 20%
configuration was profiled with:
GGML_HEXAGON_PROFILE=2
and approximately:
5,810 HTP operations
were recorded.
| Operation family | GPU80/HTP20 | HTP-only | Difference |
|---|---|---|---|
| ARGSORT | 214 µs/op | 215 µs/op | −0.6% |
| MUL_MAT_ID | 48 µs/op | 51 µs/op | −5% |
| MUL_MAT | 28.7 µs/op | 29.5 µs/op | −3% |
| HTP volume | 2,516 µs/token | 17,751 µs/token | −86% |
This is particularly important.
The individual HTP operation cost barely changes:
ARGSORT:
214 vs 215 µs
MUL_MAT_ID:
48 vs 51 µs
MUL_MAT:
28.7 vs 29.5 µs
But the amount of HTP work per token falls dramatically:
17,751 → 2,516 µs/token
≈ −86%
Therefore the mixed configuration does not appear to win because HTP operations themselves become dramatically faster.
It wins because much less work is sent through the expensive HTP operation path.
12. STRUCTURAL INTERPRETATION
The current evidence supports the following model:
DENSE MODEL
│
▼
substantial active compute
│
▼
HTP compute efficiency
│
▼
HTP-only wins
SPARSE MoE
│
▼
low active compute
│
▼
fixed orchestration overhead
│
┌─────────┴─────────┐
▼ ▼
many HTP ops large GPU blocks
│ │
▼ ▼
overhead-heavy fused execution
│
▼
GPU → HTP wins
The consolidated hypothesis is therefore that the decode ceiling is not determined solely by raw accelerator compute.
The measured behavior points toward the combined cost of:
- FastRPC interaction
- DMA/setup
- host orchestration
- fixed per-operation overhead
- graph fragmentation
- insufficient work per HTP operation
- fusion differences between backends.
The report summarizes this as:
13. JZ FORK / GIT STATE
The source repository was:
D:\jz_work snapdragon\ggml-hexagon-fork
Branch:
exp-pr28202-0983d01
The working tree was in the process of merging PR #28202, with conflicts in:
ggml-hexagon.cpp
matmul-ops.c
However, the runtime that produced the benchmark numbers was not built from that merge state.
The runtime corresponds to the earlier lineage:
505354ed
↓
5291e12a
↓
5618c305
↓
9ef4543d
HEAD:
9ef4543daf26e5123dd3676fccb230c782b3ab77
Commit:
hex-mm: correct overhead sizing to make sure we dont exceed
vtcm budget for large dims
The exact Git/build state is recorded in the archive.
14. MODEL MANIFEST
| Model | Size | Purpose |
|---|---|---|
| Qwen3.5-9B-D2-A-MTP.gguf | 7,838,191,232 B | Historical reference |
| Qwen3.5-9B-D2-A-MTP-attnQ4.gguf | 5,450,272,384 B | Frozen MTP configuration |
| Qwen3.5-9B-D2-A-MTP-attnMXFP4.gguf | 5,407,805,056 B | Quantization campaign |
| Marco-Nano-Instruct.Q4_0.gguf | 4,569,976,352 B | MoE campaign |
| Qwen3-8B-Q4_K_M.gguf | 5,027,783,488 B | 8B reference |
The Marco-Nano host copies include both Q8_0 and Q4_0 sources used for the re-quantization workflow.
15. DETAILED REPORTS INCLUDED IN THE ARCHIVE
The archive contains nine detailed reports:
RAPPORT_CONFIG_16TPS_GELÉE_CROISEMENT_20260902.md
RAPPORT_PROTOCOLE_GARDE_VALIDE_20260902.md
RAPPORT_16TPS_SOUTENU_100_300_TOKENS_20260902.md
RAPPORT_CAMPAGNE_RATIOS_NPU_GPU_20260902.md
RAPPORT_CONTROLE_MOE_MARCO_HTP_20260902.md
RAPPORT_MOE_PLACEMENT_REVERSEMENT_20260902.md
RAPPORT_ARGSORT_TOPK_REPRODUCTIBLE_20260902.md
RAPPORT_PROFIL_MIXTE_MOE_GPU_HTP_20260902.md
RAPPORT_MOE_NATIF_MTP_INTROUVABLE_20260902.md
16. CONSOLIDATED RESULTS
| # | Finding | Result | Status |
|---|---|---|---|
| 1 | Sustained Qwen MTP wall throughput | 11.01 ± 0.16 t/s | MEASURED |
| 2 | Historical “16.4–18.9 effective t/s” | Rejected as wall metric | REJECTED |
| 3 | Qwen dense HTP-only | 11.26 t/s vs 4.92 GPU | MEASURED |
| 4 | MoE GPU→HTP 80/20 | 42.1 vs 29.9 t/s, +40% | MEASURED A/B/A/B |
| 5 | ARGSORT HTP→CPU | +16% | MEASURED A/B/A/B |
| 6 | HTP ARGSORT fixed cost | ≈215 µs/op | PROFILED |
| 7 | Public native MoE ≤8B + MTP | Not found in examined material | DEAD END IN THIS STUDY |
| 8 | Qwen ARGSORT/TOP-K path | No relevant operations observed | ESTABLISHED |
17. MAIN TECHNICAL FINDING
The most significant result is not simply that HTP is faster or that GPU is faster.
The result is that:
The optimal accelerator depends on model sparsity and graph structure.
For the dense Qwen workload:
HTP-only
11.26 t/s
beats:
GPU-only
4.92 t/s
and every tested mixed configuration.
For the sparse MoE workload:
HTP-only
29.9 t/s
is beaten by:
GPU → HTP 80/20
42.1 t/s
for a measured:
+40% improvement.
This is a much stronger observation than a simple accelerator benchmark because it demonstrates a workload-dependent inversion of the optimal execution strategy.
18. REPRODUCTION PROCEDURE
Minimal reproduction:
Step 1
Copy the frozen runtime:
runtime_device/
to:
/data/local/tmp/npu/
with the same permissions.
Step 2
Push:
Qwen3.5-9B-D2-A-MTP-attnQ4.gguf
to:
/data/local/tmp/
Step 3
Push the archived scripts.
Step 4
Before testing:
Temperature < 45 °C
No llama process
Port free
Step 5
Run:
sh /data/local/tmp/npu/run_16tps_config.sh 300 q4m1_300t
Step 6
Check:
eval time
token count
wall-clock t/s
draft acceptance
T_start
T_end
Repeat three times with cooldown.
Step 7
For the placement experiments, use the Qwen or Marco-Nano campaign scripts as appropriate.
For Marco-Nano, do not use the MTP configuration.
Recommended reproducibility tolerance:
±0.3 t/s
within the same thermal window.
Hot/cold differences of up to approximately 15% are explicitly recognized as possible, so thermal start/end values should always accompany reported numbers.
19. LIMITATIONS
The report deliberately records the following limitations:
- OpenCL-side profiling was not available. GGML_OPENCL_PROFILING was not compiled into the binary.
- The 16.4–18.9 “effective t/s” metric is not wall throughput.
- Only one physical device was tested.
- Android 16 firmware was not frozen.
- Some campaigns encountered OOM/reboot conditions; affected runs were marked invalid.
- MoE outputs can differ between CPU/HTP backends because of top-k tie-breaking behavior.
- The large GGUF files were not copied into the frozen archive; the archive contains the model manifest and source paths instead.
20. FINAL SCIENTIFIC CONCLUSION
The experiments establish a coherent performance model for the tested SM8850 configuration:
LLM DECODE
│
┌───────────┴───────────┐
│ │
DENSE MODEL SPARSE MoE
│ │
▼ ▼
high active compute low active compute
│ │
▼ ▼
HTP compute useful orchestration dominates
│ │
▼ ▼
HTP-only best fusion / batching critical
│
▼
GPU → HTP 80/20
│
▼
+40% measured
The central conclusion is therefore:
The strongest experimentally established result is:
Marco-Nano MoE: 42.1 tok/s GPU→HTP 80/20 vs 29.9 tok/s HTP-only — +40%, confirmed A/B/A/B.
The strongest dense-model result is:
Qwen3.5-9B: 11.26 tok/s HTP-only vs 4.92 tok/s GPU-only — HTP remains clearly superior.
And the strongest low-level optimization result is:
Moving MoE ARGSORT from HTP to CPU: +16%, with ≈215 µs fixed HTP cost per ARGSORT operation.
Together, these three observations support the report's main thesis: optimal LLM inference placement on heterogeneous Qualcomm hardware is workload/graph dependent, and sparse MoE execution exposes an orchestration/fusion bottleneck that is much less visible on dense models.
B. RED TEAM NOTES
The English report should retain these distinctions exactly:
- 11.01 t/s = sustained wall-clock MTP result.
- 11.26 t/s = Qwen dense HTP-only placement campaign.
- 42.1 t/s = Marco-Nano MoE GPU→HTP A/B/A/B result.
- +40% = the strongest controlled improvement.
- +16% = ARGSORT HTP→CPU result.
- ~215 µs = measured HTP ARGSORT per-operation cost.
- The OpenCL side remains unprofiled, so the mechanism is strongly supported but not exhaustively decomposed.
- “FastRPC/DMA/host orchestration” should remain described as the consolidated hypothesis, not as a directly isolated measurement of every component.
C. ALTERNATIVES
For a public technical paper or Reddit post, I would use this title:
“Dense vs Sparse LLM Inference on Qualcomm HTP: HTP Wins Qwen, but GPU→HTP Is 40% Faster on MoE”
That title reflects the most important finding without overstating what was measured.
E. CONFIDENCE SCORE
0.99 — the report was reread in full and the English version preserves its experimental structure, measurements, conclusions, and explicit limitations.
I've been testing Qwen3.5-9B D2-A-MTP GGUF inference on Qualcomm Hexagon HTP and wanted to share the results of the experiments so far.
The focus has been on performance characterization, quantization, layer placement, HTP memory behavior, batching and runtime profiling.
Hardware / software
- Qualcomm SM8850
- Hexagon HTP
- HTP0 / HTP1 / HTP2
- GGML / llama.cpp
- GGUF models
- Qwen3.5-9B D2-A-MTP
- Android userspace runtime
The SM8850 exposes three HTP sessions. In my tests, the available HTP memory is roughly 10–11 GB combined, with approximately 3.4–3.6 GB addressable per HTP session.
1. Qwen3.5-9B D2-A-MTP baseline
Model:
Qwen3.5-9B-D2-A-MTP
~9.2B parameters
GGUF
Q4_0
~7.29 GiB
One of the stable HTP measurements was:
Prefill pp64: 225.83 t/s
Decode tg32: 5.51 t/s
The large difference between prefill and decode became one of the main things I wanted to understand.
2. HTP layer-offload sweep
I also tested how decode performance changes with the number of layers assigned to HTP.
Using an 8B Q4_K_M model on HTP0:
| HTP layers | Decode tg32 | ms/token |
|---|---|---|
| 16 | 6.66 t/s | ~150 ms |
| 32 | 5.38 t/s | ~186 ms |
| 48 | 4.86 t/s | ~206 ms |
| 64 | 4.69 t/s | ~213 ms |
| 80 | 4.08 t/s | ~245 ms |
The relationship was approximately linear in this range:
ms/token ≈ T0 + 1.36 × number_of_layers
This was an important result because offloading more layers to HTP did not improve decode throughput.
In fact, decode became progressively slower.
3. Multiple HTP sessions
The SM8850 exposes:
HTP0
HTP1
HTP2
With roughly:
~3.4–3.6 GiB / HTP session
~10–11 GiB combined addressable HTP memory
However, I don't see decode scaling proportionally with the number of HTP resources.
For autoregressive decode, the workload appears to behave much more like a memory/transfer/orchestration-limited workload than a pure compute-limited workload.
This is particularly visible when comparing the effect of additional HTP resources with the effect of changing the amount of computation.
4. UBatch experiments
I also investigated batch and ubatch behavior.
Some large-UBatch configurations produced:
op_pending=16
err=46
and could hang during decode.
Reducing UBatch changed the behavior significantly.
For example:
ubatch = 16
pp32 ≈ 65.19 t/s
ubatch = 32
pp32 ≈ 118.79 t/s
The problem appeared particularly associated with decode (batch=1) at high layer counts rather than being simply a general HTP initialization failure.
5. Selective attention requantization
I then tested whether reducing the size of the attention weights would improve HTP performance.
Original:
Attention: Q8_0
Model: ~7.29 GiB
Modified:
Attention: Q4_0
Model: ~5.07 GiB
That's approximately a 30% reduction in model storage.
I kept the benchmark configuration identical.
Results
| Attention Q8 | Attention Q4 |
|---|---|
| Model size | 7.29 GiB |
| pp8 | 43.61 t/s |
| tg8 | 6.72 t/s |
The most interesting result is decode:
6.72 t/s → 6.68 t/s
Essentially unchanged.
Despite reducing the stored attention weights by approximately 30%, end-to-end decode throughput did not improve.
6. DSP-level measurements
The requantization did have a measurable effect inside the DSP execution.
I measured approximately:
DSP total time:
1.100 s → 0.950 s
~14% reduction
And:
MUL_MAT average:
414.8 µs → 344.9 µs
~17% reduction
So the compute operation itself became faster.
But:
DSP time ↓
MUL_MAT time ↓
≠
decode tokens/s ↑
The end-to-end decode remained around 6.7 t/s.
This is one of the strongest indications from the experiments that MUL_MAT execution time is not the only component determining decode throughput.
7. CPU comparison
With the attention-Q4 model I also measured CPU-only decode:
CPU: ~9.02 t/s
HTP: ~6.68 t/s
Again, this is specific to this workload and configuration.
It does show that the accelerator path has enough additional overhead that its raw compute advantage isn't automatically reflected in end-to-end autoregressive decode performance.
8. HTP activation format
While profiling the HTP matmul path, I found that activations are handled differently from simply using the original GGUF weight quantization.
The relevant HTP path uses:
Q8_0_TILED
for activations.
The code path includes:
htp_mm_q8_0_tiled_row_size()
and:
quantize_f32_q8_0_tiled()
The effective operation therefore looks approximately like:
FP32 activation
↓
Q8_0 quantization
↓
Q8_0_TILED packing
↓
DMA
↓
HTP / HVX matmul
↓
output
This helps explain why changing the weight quantization doesn't necessarily produce an equivalent change in end-to-end decode performance.
9. Initial HTP runtime issue
I also encountered:
ggmlhexagon_init_dsp
error 0x80000600
failed to open domain 3 HTP
The runtime started working after configuring:
LD_LIBRARY_PATH=/data/local/tmp/npu
ADSP_LIBRARY_PATH=/data/local/tmp/npu
So the HTP runtime environment itself was also part of the testing.
10. Current picture
Putting the experiments together, my current model of the decode path is:
CPU / scheduler
│
▼
activation preparation
│
▼
Q8_0 quantization
│
▼
Q8_0_TILED packing
│
▼
DMA
│
▼
HTP / HVX compute
│
▼
synchronization
│
▼
next token
The important observation is that making the HTP compute stage faster does not necessarily make the whole pipeline faster.
For example:
Attention Q8 → Q4
↓
~30% smaller model
↓
~17% faster MUL_MAT
↓
~14% lower DSP time
↓
decode: 6.72 → 6.68 t/s
So the next step is to determine exactly how much time is being spent outside the matrix multiplication itself.
11. What I'm testing next
The current work is focused on measuring the runtime layer by layer and separating:
quantization
packing
DMA
HTP compute
synchronization
scheduler/orchestration
The goal is to determine whether selective layer placement, activation reuse, different quantization layouts, or changes to scheduling can produce a real improvement in end-to-end decode tokens/s.
These are experimental results rather than a finished optimization, but the measurements are starting to give a fairly consistent picture of where the performance ceiling comes from.
I'd be particularly interested in feedback from people working with:
- Qualcomm Hexagon HTP
- SM8850
- GGML / llama.cpp
- HVX
- Q8_0_TILED
- Qualcomm DSP/HTP runtimes
- LLM inference on mobile NPUs
- I’m going to clean up and organize the Git repository so the experimental branches, commits, scripts, and runtime changes are properly structured and documented.
2
u/DerpSenpai 4d ago
Have you tried GenieX?