Reports

Community-authored local inference reports

Published field reportsnewest

Zeus 2× Arc Pro B70: llama.cpp SYCL FP16 tuning across Qwen3.8 and nine other model routes

On two Intel Arc Pro B70s, a b11190 SYCL FP16 build substantially improved prompt processing across 10 production routes. This report gives matched serving results, copy-heavy generation gains from n-gram drafting, rejected tuning variants, and measurement limits.

Q6_K_XL focus; Q8_0, Q4_K_M, UD-Q4_K_XL llama.cpp b11190, SYCL / Level Zero

895 tok/s from a 27B dense model on one RTX 5080: DFlash2 + n-gram hybrid drafting on PQ2_0

Bonsai-2-27B-Ternary-CRACK-GGUF

How a 2.13-bpw ternary 27B, a self-speculative DFlash2 draft head, an n-gram drafting layer, and four small llama.cpp patches took a single RTX 5080 to 895 tok/s warm / 142 tok/s fresh - with verified runs, kernel-level analysis, and a full reproduction package.

PQ2_0 llama.cppRTX 5080 · 16 GB

7 benchmarks

bonsai rtx-5080 speculative-decoding pq2 llama.cpp dflash2 speed verified

Qwen3.6-35B-A3B on dual RTX 3060 12GB (llama.cpp tensor + MTP, 88k)

Qwen3.6-35B-A3B-GGUF

Two RTX 3060 12GB, no NVLink. llama.cpp b10682, Unsloth UD-Q4_K_XL, 88k. Tensor split + native MTP + q4_0 draft KV: 129 tok/s code / 104 tok/s prose. 96k+MTP does not fit.

1 benchmark

throughput multi-gpu rtx-3060 llama.cpp qwen3.6 moe

Qwen3.8-27B on dual RTX 3060 12GB (llama.cpp tensor + MTP)

Qwen3.8-27B-GGUF

Two RTX 3060 12GB, no NVLink. llama.cpp b10682, Unsloth UD-Q4_K_XL, 64k text-only. Tensor split + native MTP: 57 tok/s code / 46 tok/s prose. DFLASH2 on layer split lost.

1 benchmark

throughput multi-gpu rtx-3060 llama.cpp qwen3.8