Reports

Community-authored local inference reports

Published field reportsnewest

Zeus 2× Arc Pro B70: llama.cpp SYCL FP16 tuning across Qwen3.8 and nine other model routes

On two Intel Arc Pro B70s, a b11190 SYCL FP16 build substantially improved prompt processing across 10 production routes. This report gives matched serving results, copy-heavy generation gains from n-gram drafting, rejected tuning variants, and measurement limits.

Q6_K_XL focus; Q8_0, Q4_K_M, UD-Q4_K_XL llama.cpp b11190, SYCL / Level Zero

DeepSeek-V4.1-Flash on 4x DGX Spark: 76 tok/s verified

DeepSeek-V4.1-Flash

Verified single-stream run of deepseek-ai/DeepSeek-V4.1-Flash on 4x NVIDIA GB10 (DGX Spark, 512 GB unified, TP4, SGLang, DSPARK spec decoding): 76.0 tok/s decode, 887.6 tok/s prefill, 308.7 ms TTFT, 423 GB unified memory in use, 37.2% draft acceptance.

FP8+MXFP4 sglangGB10 Grace Blackwell · 512 GB

1 benchmark

throughput multi-gpu unified-memory sglang moe speculative-decoding dgx-spark fp8

895 tok/s from a 27B dense model on one RTX 5080: DFlash2 + n-gram hybrid drafting on PQ2_0

Bonsai-2-27B-Ternary-CRACK-GGUF

How a 2.13-bpw ternary 27B, a self-speculative DFlash2 draft head, an n-gram drafting layer, and four small llama.cpp patches took a single RTX 5080 to 895 tok/s warm / 142 tok/s fresh - with verified runs, kernel-level analysis, and a full reproduction package.

PQ2_0 llama.cppRTX 5080 · 16 GB

7 benchmarks

bonsai rtx-5080 speculative-decoding pq2 llama.cpp dflash2 speed verified

CMP 170HX: Qwen3.8-27B AutoRound + DFlash2 at 131k

Qwen3.8-27B-W4A16-AutoRound

vLLM 0.29 on a 64GB CMP 170HX. AutoRound W4A16 + DFlash2, production max-model-len 131072. 133 tok/s short decode, 98 tok/s with a 7.3k prefix. Not an 8k-ctx speed-run.

W4A16 vllmCMP 170HX · 64 GB

2 benchmarks

throughput autoround dflash2 long-context ampere

Qwen3.8-27B on RX 9070 XT: one 64K-context stream, 655 tok/s prefill, 23.5 tok/s decode on 16 GB

Qwen3.8-27B

A 65,100-token prompt on a single 16 GB card: 4-bit KV cache (1.1 GB for the whole context), host-staged embedding table, 99 s TTFT (was 146 s before the flash-attention fix), 23.46 tok/s decode, 14.24 GB peak. Speculative decoding is unavailable past 4K context by design.

long-context rocm amd-rdna4 qwen3.8 kv-cache-quantization throughput

Qwen3.8-27B on RX 9070 XT: 4 concurrent streams at 4K context, 98.3 tok/s aggregate (custom HIP engine)

Qwen3.8-27B

Four 4K-context streams on one 16 GB RDNA4 card: custom IQ4_S256 quant, 4-bit KV cache, 16K shared paged pool. Plain batched decode matches speculative decoding at this batch (98.2 vs 98.3 tok/s) with 1.65 GB more headroom.

throughput multi-stream rocm amd-rdna4 qwen3.8 batching speculative-decoding

Qwen3.8-27B on RX 9070 XT: custom HIP engine, 84.8 tok/s (MTP) / 80.8 tok/s (DFlash2 drafter)

Qwen3.8-27B

Single 16 GB RDNA4 card, custom IQ4_S256 quant, WY-form GDN prefill + WMMA flash attention; lossless speculative decoding verified byte-identical across three configs.

rocm rdna4 speculative-decoding custom-engine throughput quantization

Qwen3.6-35B-A3B on dual RTX 3060 12GB (llama.cpp tensor + MTP, 88k)

Qwen3.6-35B-A3B-GGUF

Two RTX 3060 12GB, no NVLink. llama.cpp b10682, Unsloth UD-Q4_K_XL, 88k. Tensor split + native MTP + q4_0 draft KV: 129 tok/s code / 104 tok/s prose. 96k+MTP does not fit.

1 benchmark

throughput multi-gpu rtx-3060 llama.cpp qwen3.6 moe

Qwen3.8-27B on dual RTX 3060 12GB (llama.cpp tensor + MTP)

Qwen3.8-27B-GGUF

Two RTX 3060 12GB, no NVLink. llama.cpp b10682, Unsloth UD-Q4_K_XL, 64k text-only. Tensor split + native MTP: 57 tok/s code / 46 tok/s prose. DFLASH2 on layer split lost.

1 benchmark

throughput multi-gpu rtx-3060 llama.cpp qwen3.8

FreeToken vs vLLM on one RTX 3090: BF16 MoE RAM offload compared

Qwen3.6-35B-A3B

An apples-to-apples single-RTX-3090 comparison of FreeToken expert offload and vLLM CPU offload with Qwen3.6-35B-A3B BF16. FreeToken reached 42.78 tok/s established decode versus 7.01 tok/s for tuned vLLM, while vLLM delivered lower TTFT. Includes sequential, prefill, concurrency, memory, tuning, and MTP/DFlash compatibility findings.

BF16 FreeToken,vLLM

freetoken vllm qwen3.6 rtx-3090 bf16 moe-offload cpu-offload benchmark

MTP vs DFlash: vLLM and SGLang on dual RTX 3090s

Qwen3.8-27B-GPTQ-4bit

Four-way Qwen3.8-27B GPTQ-4bit comparison: vLLM and SGLang with native MTP versus DFlash2. DFlash wins decode throughput; SGLang wins low-concurrency TTFT.

vLLM vs SGLang: Qwen3.8-27B Q4 DFlash2 on dual RTX 3090s

Qwen3.8-27B-GPTQ-4bit

Apples-to-apples DFlash2 comparison across concurrency 1, 2, 4, 8, and 16. SGLang wins at concurrency 4; vLLM reaches 430.67 tok/s at concurrency 16 and avoids SGLang's five-request Mamba cache cap.

Qwen3.8 27B GPTQ + MTP on 2× RTX 3090: the vLLM setup behind our Terminal-Bench run

Qwen3.8-27B-GPTQ-4bit

A reproducible analysis of our vLLM 0.26.0 stack: GPTQ-Marlin, TP2, FP8 KV cache, 262K context, native MTP, prefix caching, 96.6 tok/s single-stream decode, and 175.7 tok/s at concurrency 16.

1 benchmark

Qwen3.8 27B on Terminal-Bench 2.1: 69.7% xhigh, 78.7% adaptive

Qwen3.8-27B-GPTQ-4bit

A five-day, 89-task Terminal-Bench 2.1 run on two RTX 3090s: methodology, first-pass and adaptive scores, 880M processed tokens, failure analysis, and lessons for routing reasoning effort.