On-Device LLM Inference Engines for a Local-First Rust App — Comparison & Recommendation

Date: 2026-06-13 Author: research pass (Claude) Context: life — local-first, privacy-first app in Rust + Tauri v2 + Leptos. Inference sits behind an Inference trait in a shared substrate crate. Targets: Xiaomi 13 Pro (Snapdragon 8 Gen 2, Adreno 740, Hexagon NPU, 12 GB RAM) and desktop (Linux / macOS / Windows). Purpose: pick the top 2—​3 engines to actually benchmark on-device (Rust binary cross-compiled to aarch64-linux-android, pushed via adb), and which models/quantizations to test.

Honesty note on uncertainty. Several Android-specific claims below could not be confirmed end-to-end from a single authoritative "we built this on SD 8 Gen 2 and it worked" source. Where that is the case it is flagged explicitly with ⚠️. The benchmark itself is the way we resolve those unknowns — that is the whole point of shortlisting.

TL;DR

  • Benchmark these three: (1) llama-cpp-2 (utilityai bindings to llama.cpp) — the mobile perf leader, by a wide margin; (2) candle + candle-transformers — pure-Rust, cleanest Tauri/trait integration, CPU-only on Android but trivial to build; (3) ort (ONNX Runtime) with XNNPACK/NNAPI/QNN EPs — the only credible path to the Hexagon NPU, but the highest-risk/highest-effort option.

  • Models to test: Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct (Apache-2.0 for 1.5B; research-only license for 3B — flag), Llama-3.2-1B/3B-Instruct (Llama community license, attribution required), SmolLM2-1.7B-Instruct (Apache-2.0), Gemma-2-2B-it (Gemma license), Phi-3.5-mini (MIT). Quantize to Q4_K_M as the primary, with Q4_0 as the GPU-friendly variant for the llama.cpp Adreno/OpenCL backend.

  • Single biggest open question: Is the Qualcomm Hexagon NPU realistically reachable from a shippable indie Rust app on SD 8 Gen 2, and is it worth it? NPU paths (QNN EP, llama.cpp Hexagon backend) demand per-model offline compilation, INT4/INT8-only graphs, and heavy SDK/toolchain coupling. The pragmatic baseline is almost certainly CPU (ARM NEON / i8mm) via llama.cpp, with GPU (OpenCL on Adreno) as the second lever. NPU is a research spike, not a v1 dependency.

Candidate Matrix

Engine Latest ver (≈Jun 2026) License Android aarch64 build GPU on SD 8 Gen 2 NPU (Hexagon) Model formats / quant Tauri + trait fit Maturity for mobile LLM Verdict

llama.cpp via llama-cpp-2 (utilityai)

llama-cpp-2 0.1.133 / -sys-2 0.1.146

MIT (crate) + MIT (llama.cpp)

✅ Well-trodden (NDK + CMake)

✅ OpenCL (Adreno) ; Vulkan possible

⚠️ Experimental (QNN/Hexagon forks)

GGUF; full K-quant range incl. Q4_K_M, Q4_0, Q6_K, Q8_0

C FFI behind a Rust trait — clean

Highest (de-facto mobile leader)

Benchmark

candle + candle-transformers

active, 0.9.x line

Apache-2.0 / MIT

✅ Pure Rust, easy cross-compile

❌ No Vulkan/wgpu/Android GPU (CPU only)

❌

GGUF (llama.cpp quant types), safetensors

Native Rust — best fit

Medium (CPU-only on mobile)

Benchmark

ort (ONNX Runtime)

2.0.0-rc.12 (Mar 2026)

MIT/Apache-2.0 (crate); MIT (ORT)

✅ via prebuilt/ custom ORT

✅ via QNN-GPU; NNAPI

✅ QNN EP → Hexagon (the realistic NPU path)

ONNX (INT4/INT8 for NPU); needs ONNX export

FFI to ORT shared lib; trait OK

Medium; NPU = high effort

Benchmark (NPU spike)

mistral.rs

0.8.3 (Jun 1 2026)

MIT

⚠️ aarch64 builds, no Android story

CUDA/Metal only (no Vulkan)

❌

GGUF, ISQ, UQFF, GPTQ/AWQ/HQQ/FP8

Native Rust

Low for mobile

Desktop-only candidate

burn (+ burn-wgpu)

0.8.x line

Apache-2.0 / MIT

⚠️ wgpu→Vulkan could work; unproven for LLM

⚠️ wgpu/Vulkan in theory

❌

Own format; import via candle/onnx; no turnkey GGUF

Native Rust

Low (LLM inference immature)

Watch

tract (sonos)

active

Apache-2.0 / MIT

✅ pure Rust, embedded-friendly

❌ CPU only

❌

ONNX (~85% ops), NNEF

Native Rust

Not suited to autoregressive LLMs

Reject for LLM

ratchet

low activity

MIT-ish

⚠️ wgpu; web-first

⚠️ WebGPU/wgpu

❌

Own (GGUF-ish)

Native Rust

Low / early

Watch

wonnx

low activity

MIT/Apache

⚠️ web-first

⚠️ WebGPU

❌

ONNX

Native Rust

Low (1—​4 tok/s reported)

Reject for LLM

Per-Engine Detail

1. llama.cpp via llama-cpp-2 (utilityai/llama-cpp-rs) — mobile perf leader

  • Version / activity: llama-cpp-2 0.1.133 (2026-02-03), low-level llama-cpp-sys-2 0.1.146 (2026-04-30). Actively maintained, deliberately tracks upstream llama.cpp closely and "mimics its API as closely as possible while being safe." [crates.io/llama-cpp-2, deepwiki utilityai]

  • License: crate MIT; upstream llama.cpp MIT. Clean for shipping.

  • Android aarch64: llama.cpp itself has a first-class, documented Android NDK + CMake cross-compile path (arm64-v8a, platform 28). The Rust -sys crate vendors/builds llama.cpp via CMake, so the same NDK toolchain file should apply. ⚠️ Gotchas: NDK ships a limited native-lib set; the -sys crate’s build script may need CMAKE_TOOLCHAIN_FILE/ANDROID_ABI plumbing, and OpenMP-via-CMake is noted as unsupported in the NDK path. Expect to fight the build script once; it is a known-solvable problem (multiple projects do it). [llama.cpp docs/android.md, issue #5312]

  • GPU on SD 8 Gen 2 (Adreno 740): Qualcomm shipped an OpenCL backend for llama.cpp explicitly targeting Adreno, supporting Snapdragon 8 Gen 1/2/3/Elite, optimized for Q4_0 weights (FP16/Q6 in progress). Vulkan backend also exists but cross-compiling Vulkan for arm64 has reported friction. [Qualcomm Adreno OpenCL blog, llama.cpp OPENCL.md, issue #13068]

  • NPU (Hexagon): ⚠️ Experimental. Out-of-tree efforts (e.g. chraac/llama-cpp-qnn-builder, upstream PR-12326 Hexagon cDSP work) provide QNN-CPU/GPU/NPU and direct Hexagon cDSP offload. Real, but bleeding-edge, not in the stable Rust crate path. Treat as a separate spike. [llama.cpp discussion #14356, chraac/llama-cpp-qnn-builder]

  • Formats / quant: GGUF with the full K-quant family — Q4_K_M (primary), Q4_0 (GPU), Q5_K_M, Q6_K, Q8_0.

  • Perf / memory expectations: Best available on mobile CPU. Academic mobile benchmarking puts Llama-2 7B Q4 at ~12.7 tok/s prefill / ~8.9 tok/s decode on Snapdragon 8 Gen 3 (Xiaomi 14 Pro) CPU. A 1B—​3B model on the slightly older 8 Gen 2 should comfortably exceed that on decode (small models are memory-bandwidth bound; rough expectation 15—​40+ tok/s decode for 1B—​3B Q4 on CPU — to be confirmed by our benchmark). GPU OpenCL adds prefill throughput especially. RAM: a Q4_K_M model needs ≈ model-file size + KV cache; well within 12 GB for ≤3B. [arXiv 2410.03613]

  • Tauri + trait fit: C FFI wrapped by a safe Rust crate → implement the Inference trait directly over it. Same crate works on desktop (CPU/CUDA/Metal/Vulkan) and Android, which is exactly the cross-platform story we want.

  • Risks: build-script friction on Android; keeping -sys pinned to a compatible llama.cpp; GPU backend requires a separate compile flag + Q4_0 model; binary size.

2. candle + candle-transformers — cleanest Rust integration, CPU-only on Android

  • Version / activity: active 0.9.x line, ~20k stars, frequent commits; Feb 2026 brought Qwen3/Hunyuan optimizations (pre-allocated KV cache, GQA 4D matmul, fused RoPE, GGUF quant, batched decode). [github huggingface/candle]

  • License: dual Apache-2.0 / MIT. Clean.

  • Android aarch64: Pure Rust → cross-compiles cleanly to aarch64-linux-android with minimal build-script pain (this is candle’s strongest selling point for us). ⚠️ No official "we run on Android" doc, but no native-lib barrier either.

  • GPU/NPU on SD 8 Gen 2: ❌ None. Candle’s GPU backends are CUDA and Metal only — no Vulkan, no wgpu, no Android GPU. On Android it is CPU-only. (WebGPU has been a long-standing open issue, not delivered.) [candle README, issue #344]

  • Formats / quant: GGUF (same quant types as llama.cpp, incl. Q4_K_M), safetensors. Strong model coverage (Llama, Qwen, Mistral, Gemma, Phi).

  • Perf / memory: CPU decode generally below llama.cpp’s hand-tuned NEON kernels, but competitive for small models; expect to land within ~0.5—​0.8× of llama.cpp CPU — benchmark to confirm. Memory similar to llama.cpp for the same GGUF.

  • Tauri + trait fit: Best of all — no FFI, no external shared libs, one Cargo dependency, identical code desktop↔mobile. Lowest integration risk.

  • Risks: no GPU/NPU on Android caps peak performance; perf trails llama.cpp; you own more of the generation loop (sampling, KV cache management) than with llama.cpp’s batteries-included API.

3. ort (ONNX Runtime Rust) — the realistic NPU path, highest effort

  • Version / activity: 2.0.0-rc.12 (2026-03-05), active. [github pykeio/ort]

  • License: crate MIT/Apache-2.0; ONNX Runtime MIT. Clean.

  • Android aarch64: ✅ supported; ORT runs on-device (the Xybrid project uses ort for LLM/ASR/TTS on iOS/Android). Needs a prebuilt or custom-built ORT shared lib with the desired EPs.

  • Execution providers (the key feature): ort 2.x exposes CoreML, NNAPI, QNN, RKNPU, XNNPACK mobile/edge EPs, all "arbitrarily configurable." [ort discussion #301, ort.pyke.io]

    • XNNPACK EP — solid, well-optimized CPU path on ARM; lowest-risk ORT option.

    • NNAPI EP — routes to GPU/NPU via Android’s NNAPI, but falls back to CPU for unsupported ops (common for LLM graphs) and NNAPI is being de-emphasized industry-wide. Unreliable for full LLM acceleration. [onnxruntime NNAPI docs]

    • QNN EP → Hexagon NPU — the genuine NPU route. Uses Qualcomm AI Engine Direct (QNN/QAIRT) SDK. As of May 2026 Qualcomm shipped a "Plugin EP" that loads as a separate shared lib against stock ORT (no custom ORT build). Requires models quantized to INT8/INT4 and typically offline-compiled per model; Qualcomm AI Hub publishes pre-optimized Phi/Mistral/Llama variants. [Qualcomm Plugin EP blog May 2026, onnxruntime QNN docs, onnxruntime/onnxruntime-qnn]

  • Formats / quant: ONNX. You must export each model to ONNX (e.g. via Optimum / onnxruntime-genai), and for NPU, to INT4/INT8 ONNX — a real pipeline cost. LLM autoregressive support is via onnxruntime-genai patterns (KV-cache as graph I/O).

  • Perf / memory: NPU can deliver large energy/throughput wins for INT4 LLMs, but only for graphs that map cleanly; otherwise CPU fallback negates the benefit. Memory governed by the INT4/INT8 ONNX model.

  • Tauri + trait fit: FFI to an ORT shared lib; trait implementation is fine, but you ship + version native ORT libs per platform/EP, plus (for QNN) Qualcomm runtime libs.

  • Risks (high): ONNX export friction for newer architectures; per-model offline compilation for QNN; SDK/version coupling; large native dependency surface; NNAPI op-coverage gaps. This is a research spike, not a safe default.

4. mistral.rs — strong desktop, no mobile story

  • Version: 0.8.3 (2026-06-01), MIT, very active. Excellent quant breadth (GGUF, ISQ in-situ, UQFF, GPTQ/AWQ/HQQ/FP8). [github EricLBuehler/mistral.rs, docs.rs]

  • Backends: CUDA + Metal + CPU. No Vulkan, no Android/mobile story in docs. aarch64 (incl. Apple Silicon, Linux ARM servers) builds, but nothing demonstrates Android phone deployment. Could serve as the desktop backend behind the same trait, but not a mobile candidate. ⚠️

5. burn (+ burn-wgpu) — promising portability, immature for LLM

  • Version: 0.8.x line, Apache-2.0/MIT, active. Backends: ndarray (CPU), wgpu (→ Vulkan/Metal/DX12/WebGPU), libtorch, candle. no_std capable. [github tracel-ai/burn]

  • Mobile LLM: ⚠️ wgpu→Vulkan could in principle give Adreno GPU on Android, which would be unique among pure-Rust options — but there’s no turnkey GGUF loader and no demonstrated efficient autoregressive LLM inference path on mobile. High effort, unproven. Watch, don’t bet v1 on it.

6. tract (sonos) — not for autoregressive LLMs

  • Pure-Rust ONNX/NNEF, ~85% ONNX ops, CPU only, embedded-focused, excellent for small fixed-graph models (vision, keyword spotting). Architecturally wants to "flow Tensors and nothing else"; weak on the dynamic-shape / sequence / KV-cache patterns that autoregressive LLMs need. Reject for LLM use (great if life ever needs a small embedding/classifier model). [lib.rs tract-onnx, github sonos/tract]

7. ratchet / wonnx — web-first, not ready for mobile LLM

  • ratchet: wgpu-based, web-first inference toolkit (WebGPU + CPU); cross-platform in theory, but low activity and early for production mobile LLM. Watch.

  • wonnx: WebGPU ONNX runtime; reported ~1—​4 tok/s for quantized models — too slow. Reject for LLM. [github webonnx/wonnx, everyintel ratchet]

Qualcomm NPU reachability (summary judgment)

NPU access from a Rust app on SD 8 Gen 2 is technically reachable but operationally heavy:

  • Most realistic: ort + QNN Plugin EP (shipped May 2026) with INT4/INT8 ONNX models, ideally Qualcomm-AI-Hub-optimized.

  • Bleeding edge: llama.cpp Hexagon/QNN forks (not in the stable Rust crate).

  • Cost: per-model offline compilation, INT-only graphs, SDK/runtime-lib shipping, op-coverage cliffs. For an indie local-first app, CPU (llama.cpp NEON/i8mm) is the pragmatic baseline; Adreno OpenCL GPU is the second lever; NPU is a spike to validate, not a launch dependency. [Qualcomm Plugin EP blog]

Top 2—​3 to Benchmark — with rationale

  1. llama-cpp-2 (llama.cpp) — primary. Mobile perf leader; full GGUF/K-quant support; both CPU (NEON/i8mm) and Adreno OpenCL GPU reachable; MIT; one crate spans desktop + Android behind our Inference trait. This is almost certainly the production backend; benchmark it first and hardest.

  2. candle — fast-follow / fallback. Pure-Rust → near-zero Android build pain and the cleanest fit for substrate/Tauri. CPU-only on Android caps it, but if its decode is "good enough" for our small models, the integration simplicity is worth a lot. Benchmark head-to-head against llama.cpp CPU on identical GGUFs.

  3. ort + XNNPACK/QNN — NPU/ONNX spike (time-boxed). Only credible Hexagon-NPU path and a strong ONNX/CPU (XNNPACK) baseline. High effort (ONNX export, per-model QNN compilation), so treat as a bounded experiment to answer "is NPU worth it?" — not a v1 commitment.

Explicitly deferred: mistral.rs (desktop-only, behind same trait if useful), burn (watch wgpu/Vulkan-on-Android maturity), tract (reserve for small non-LLM models). Rejected for LLM: wonnx, ratchet (for now).

Recommended Models & Quantizations to Test

Primary quantization: Q4_K_M (best quality/size balance). For the llama.cpp Adreno OpenCL GPU path also build Q4_0 (its optimized format). Approx RAM = GGUF file size + KV cache; all comfortably fit 12 GB.

Model Params License / redistribution Notes Approx Q4_K_M file Test?

Qwen2.5-1.5B-Instruct

1.5B

Apache-2.0 ✅ ship freely

Strong reasoning/instruction for size

~1.0—​1.1 GB

✅ core

Qwen2.5-3B-Instruct

3B

⚠️ "Qwen Research License" — research-only, NOT free commercial (3B & 72B are the exceptions to Qwen2.5 Apache-2.0)

Best quality of the small set, but redistribution risk for a shipped app

~1.9—​2.0 GB

✅ test, but flag license

Llama-3.2-1B-Instruct

1B

Llama 3.2 Community License — usable <700M MAU; must show "Built with Llama" + name-prefix derivatives

Fast, decent quality

~0.8 GB

✅ core

Llama-3.2-3B-Instruct

3B

same Llama community license

Good quality

~2.0 GB

✅

SmolLM2-1.7B-Instruct

1.7B

Apache-2.0 ✅

Cleanest license; good warm/chatty "Guide voice" candidate

~1.0 GB

✅ core

Gemma-2-2B-it

2B

Gemma Terms of Use (permissive-ish, has use-policy + attribution; not OSI)

Strong 2B quality

~1.5 GB

✅

Phi-3.5-mini-instruct

3.8B

MIT ✅ ship freely

Strong reasoning; larger of the set

~2.2—​2.4 GB

✅

Capture / decomposition / scheduling reasoning: lean Qwen2.5-1.5B, Phi-3.5-mini, Llama-3.2-3B (reasoning-heavy). Warm "Guide voice": SmolLM2-1.7B-Instruct and Gemma-2-2B-it tend to be friendlier in tone; worth subjective comparison.

License recommendation for a shipped, redistributed model: prefer Apache-2.0 (Qwen2.5-1.5B, SmolLM2) or MIT (Phi-3.5) as the default bundled model — no redistribution strings. Use Llama-3.2 only with the required "Built with Llama" attribution and naming. Avoid bundling Qwen2.5-3B in the shipped app unless a commercial license is obtained (research-only). Verify every model’s current card on Hugging Face before shipping — licenses change. [Qwen2.5 blog, Llama 3.2 license, Phi MIT, bentoml/digitalapplied SLM guides]

Recommended Benchmark Methodology

Goal: apples-to-apples on the Xiaomi 13 Pro (SD 8 Gen 2), with desktop as a reference line, using synthetic prompts only (no real user data — project privacy rule).

Harness

  • Single Rust binary cross-compiled to aarch64-linux-android (target aarch64-linux-android, NDK r26+/clang, API 28+). One binary per engine (or feature-gated) implementing the same Inference trait and the same generation loop parameters.

  • Push via adb push to /data/local/tmp/; run via adb shell. Push GGUF/ONNX models the same way. Keep the device on a fixed governor / airplane mode / screen-on, plugged-or-unplugged consistently to reduce thermal/DVFS noise.

  • For each (engine × model × quant × backend) run N=5 reps; report median + IQR.

Metrics

  • Time-to-first-token (TTFT) (ms) — prompt processed → first decoded token.

  • Prefill throughput (tokens/s) — prompt tokens / prefill time. Measure at multiple prompt lengths (64, 256, 512, 1024).

  • Decode throughput (tokens/s) — generated tokens / decode time, at fixed output lengths (128, 256).

  • Model load time (cold + warm/mmap) (ms).

  • Peak RAM (RSS) — sample via adb shell dumpsys meminfo <pid> or /proc/<pid>/status (VmHWM) during a run.

  • Energy / thermal (best-effort, flag as noisy): battery delta via adb shell dumpsys batterystats / cat /sys/class/power_supply//current_now; SoC temps via adb shell cat /sys/class/thermal/thermal_zone/temp. Capture throttling onset by running a sustained 5-min decode loop and watching tok/s degrade.

  • Output sanity: log the generated text for a couple of prompts to confirm the engine produced coherent output (no silent garbage), but discard text from metrics.

Synthetic prompt set (no real user data)

Design prompts that exercise the actual workload classes without PII:

  • NL capture → structured: "Turn this note into a task with title, due date, priority: 'pick up dry cleaning before the weekend trip'" (fabricated).

  • Decomposition: "Break 'plan a small dinner party' into 5—​8 ordered subtasks."

  • Scheduling reasoning: "Given free slots Tue 2—​4pm and Thu 9—​11am, and a 90-min task, propose a schedule and explain."

  • Guide voice / tone: "Encourage someone who missed their morning routine, in two warm sentences."

  • Long-context prefill stress: a 1024-token synthetic filler doc + a question, to load prefill throughput. Fix temperature, top_p, max_tokens, and seed across engines so comparisons are fair. Keep the set in-repo under docs/research/ or a benches/ fixture so runs are reproducible.

Backends to sweep per engine

  • llama.cpp: CPU (NEON/i8mm), OpenCL (Adreno, Q4_0), optionally Vulkan if it builds.

  • candle: CPU only on Android.

  • ort: XNNPACK (CPU), NNAPI, QNN (NPU, INT4/INT8 ONNX) — the QNN run is the spike.

Decision criteria

Pick the production backend on: decode tok/s at acceptable quality, TTFT (interactivity), peak RAM headroom on a 12 GB device shared with the OS + app, build/integration cost, and license cleanliness of the bundled model. Sustained (thermally-throttled) tok/s matters more than burst for a real app.

Sources