On-Device Inference Benchmark — Results & Recommendation

Date: 2026-06-13 Device: Xiaomi 13 Pro (model 2210132G), Android 15, Snapdragon 8 Gen 2 (kalama), Adreno 740 GPU, Hexagon V73 NPU, 12 GB RAM. Thermally stable throughout (monitored). Harness: benchmarks/inference/ — one Rust binary per engine behind a shared Inference trait, cross-compiled to aarch64-linux-android (NDK 27.2, API 33), run via adb. Synthetic prompts only (no real user data — project privacy rule). Companion: desk research in the engine comparison.

This benchmark exists to settle the open decision from product-family-overview.md §6: which on-device inference engine for the substrate. It also corrected the desk research in two places (noted below).

TL;DR — decision

  • Production backend = llama.cpp (llama-cpp-2) on CPU. Fastest decode, lowest RAM, trivial Android build, and the only engine that can also reach the GPU/NPU.

  • Default bundled model = Qwen2.5-1.5B-Instruct, Q4_K_M. ~13 tok/s decode (interactive), ~1.0 GB RAM, Apache-2.0 (ship-clean). SmolLM2-1.7B (Apache) is the alternate.

  • GPU (Adreno) is not worth it for v1: it ~1.9× prefill but is slower than CPU at decode. Keep CPU; revisit GPU only if long-prompt prefill becomes a bottleneck.

  • NPU (Hexagon) is a future spike, now de-risked: the Qualcomm QNN/HTP-V73 libraries are present on the device, but reaching them needs the ort load issue fixed and a QNN-prepared model. Not an MVP dependency.

Results — all engines, on-device (Snapdragon 8 Gen 2)

Decode = token generation tok/s (what the user feels). Prefill = prompt processing tok/s. Median over reps × the 5 synthetic prompts. Higher = better; RSS lower = better.

llama.cpp (CPU, GGUF Q4_K_M) — full matrix, via harness

Model decode t/s prefill t/s TTFT ms peak RSS load ms

Qwen2.5-1.5B

13.4

27.9

1333

1.0 GB

1020

SmolLM2-1.7B

11.4

20.7

2117

1.4 GB

962

Qwen2.5-3B

7.7

12.7

3179

2.0 GB

1764

Llama-3.2-3B

7.3

11.9

3608

2.2 GB

2066

Phi-3.5-mini (3.8B)

6.2

8.0

5452

3.0 GB

1889

Cross-engine, Qwen2.5-1.5B (apples-to-apples, decode t/s)

Engine (CPU) on-device decode on-device prefill peak RSS notes

llama.cpp

13.4

27.9

1.0 GB

winner

mistral.rs

5.2

9.9

2.1 GB

beat candle on device; slow load (7.5s)

candle

3.9

6.2

2.1 GB

slowest; degrades with context length

ort (int4)

 — 

 — 

 — 

host-validated; on-device load blocked (see below)

 
llama.cpp   13.4  ▇▇▇▇▇▇▇▇▇▇▇▇▇
mistral.rs   5.2  ▇▇▇▇▇
candle       3.9  ▇▇▇▇

candle & mistral.rs across sizes (on-device, decode t/s)

Model candle mistral.rs

Qwen2.5-1.5B

3.9

5.2

Qwen2.5-3B

2.0

 — 

Llama-3.2-3B

1.5

3.0

Phi-3.5-mini

3.7

1.6

GPU vs CPU — llama.cpp (llama-bench, Qwen2.5-1.5B Q4_0, Adreno 740)

Backend prefill (pp512) decode (tg128)

GPU (OpenCL, ngl 99)

46.4

13.5

CPU (ngl 0)

24.9

16.4

GPU ~1.9× prefill, but slower decode (memory-bandwidth-bound). (Measured with upstream llama-bench, not the Rust harness — llama-cpp-2 doesn’t expose OpenCL; see Notes.)

Findings that corrected the desk research

  1. llama.cpp Android build was trivial, not hard. The cmake crate’s built-in Android support + ANDROID_NDK_HOME produced a working aarch64 binary in ~1m40s with zero build-script fighting. Only runtime need: push libc++_shared.so + LD_LIBRARY_PATH.

  2. mistral.rs DOES cross-compile to Android (research said "no Android story"). It built cleanly and ran on-device — and beat candle on the phone (5.2 vs 3.9 t/s), flipping the host ranking. It remains slowest-tier for throughput, with heavy RAM and slow load.

  3. candle scales worse with context length, not just "slower": Qwen-1.5B decode dropped from ~5.0 (short) to 3.9 t/s (96-token gen) — algorithmic (KV growth), confirmed not thermal. llama.cpp held steady.

  4. The Hexagon NPU is reachable in principle: /vendor/lib64/libQnnHtp.so, libQnnGpu.so, libQnnCpu.so, and /vendor/lib/rfsa/adsp/libQnnHtpV73{,Skel}.so are all present on-device — no QAIRT SDK download needed to attempt it.

Per-engine verdict

  • llama.cpp (llama-cpp-2) — ✅ Production backend. Fastest decode at every size, lowest RAM, all GGUF архs uniformly, one crate for desktop + Android, and the only engine with a mobile GPU (OpenCL/Vulkan) and NPU (forks) path. Adopt behind substrate::Inference.

  • mistral.rs — Viable mobile fallback + strong desktop backend (best API, widest quantization: ISQ/UQFF/GGUF/GPTQ/AWQ; CUDA/Metal on desktop). ~2.6× slower than llama.cpp on phone CPU. Keep as an optional desktop path; not the mobile default.

  • candle — Cleanest pure-Rust integration (no FFI), but slowest on-device and degrades with context. Reserve for a small non-LLM model if ever needed, not the LLM engine.

  • ort (ONNX Runtime) — Host-validated (int4 CPU ~11.5 t/s decode, 69 prefill, but 4 GB RSS + 9.4s load). On-device load is blocked with the stock onnxruntime-android 1.22 build: the process hangs at session init (~0% CPU, no output); ruled out missing deps, model size (0.5B also hangs), and graph optimization. Highest integration effort, as the research warned.

The NPU question (Hexagon V73) — scoped, not closed

The QNN/HTP libraries are on the device, so the NPU is tractable later. To actually measure it needs two things we deferred: (a) fix the ort-on-device load (or drive QNN via onnxruntime-genai/Qualcomm tooling instead of our Rust harness), and (b) a QNN-prepared model — our generic int4 MatMulNBits ONNX won’t map to HTP; QNN HTP needs INT8/INT4 QDQ graphs, typically per-model offline-compiled (Qualcomm AI Hub publishes some). Verdict unchanged from research: CPU is the pragmatic baseline; NPU is a post-MVP spike to validate whether it pays for the toolchain complexity.

Recommendation for the substrate (MVP)

  1. Engine: llama.cpp via llama-cpp-2, CPU, behind substrate::Inference.

  2. Model: Qwen2.5-1.5B-Instruct Q4_K_M bundled (Apache-2.0, ~1 GB, ~13 t/s decode — interactive). Allow Phi-3.5/3B-class as an optional "higher quality, slower" tier.

  3. Threads: 4 (used here); tune later.

  4. Defer: GPU offload (prefill-only win), NPU/QNN (post-MVP spike), ort entirely.

  5. Desktop: same llama-cpp-2 path (one codebase) — or mistral.rs if its quant breadth is wanted; both sit behind the same trait.

This keeps the inference layer one trait, one engine, swappable — satisfying the project-context.md rule that no backend type leaks above the substrate boundary.

Notes & caveats

  • GPU & QNN ran outside the Rust harness. llama-cpp-2 0.1.146 exposes only a cuda feature — no OpenCL/Vulkan — so the GPU number came from upstream llama.cpp llama-bench (built with -DGGML_OPENCL=ON + cross-compiled OpenCL-Headers/ICD-Loader). To ship GPU, vendor llama-cpp-sys-2 and add config.define("GGML_OPENCL","ON").

  • Quant differs in the GPU table (Q4_0 for the Adreno path vs Q4_K_M elsewhere) — Q4_0 is llama.cpp’s GPU-optimized format. Same-tool CPU baseline is included for fairness.

  • candle/mistral.rs runs: reps=2 (vs llama.cpp reps=3) to bound device time; medians still stable. candle covered 4 models, mistral.rs 3 (representative across архs/sizes).

  • Reproduce: benchmarks/inference/scripts/bench-device.sh <engine> <backend>; raw JSONL in benchmarks/inference/results/.