Community-authored local inference reports
Complete field report on Strata v0.1.42–v0.1.43: 15 verified speed/depth rows (canonical prompts, cold prefill, MTP), and three full quality evals — HellaSwag 92.66%, GSM8K 98.10%, ARC-Challenge 98.43% — all 13/13 shards approved. 180B MoE on 2x16 GB consumer GPUs via expert streaming.
Verified UD-IQ4_XS speed suite on Strata v0.1.43.1: reasoning 105.7 / code 107.6 tok/s (TTFT 649/589 ms), verified-at-depth decode 101.4/100.8/94.3 tok/s at 16k/65k/131k with cold prefill 3.2–3.7k tok/s. All five runs verifiedRun:true; full contamination-handling disclosure included.
Five weeks taking Qwen3.8-27B on a single Tenstorrent Blackhole P150 from 7.7 tok/s to ~50-55 tok/s single-stream decode (DFlash2), ~1,200 tok/s prefill and native 262K context, using a custom mixed BFP4/BFP8 quant (now GPTQ). Covers the quant, every speed-test milestone, quality checks, power and thermal data, the dead ends, lessons learned and what's next.
125B MoE model served locally on one 16 GB RTX 5080 plus 92 GB RAM: ~80-99 tok/s decode, prefill scaling to 140k tok/s at 32K context, MTP speculative decoding at ~80% acceptance.
Qwen3.8-27B on two RTX 3060 12GB cards, llama.cpp 0.4.1-dev. Measured 129,000-token synthetic prompt at 497.8 prompt tok/s and 42.7 output tok/s mean, with exact recall of 3/3 canary facts in two runs. Configured context 131,072; ngram-mod + MTP3; batch 2048, ubatch 512.
On two Intel Arc Pro B70s, a b11190 SYCL FP16 build substantially improved prompt processing across 10 production routes. This report gives matched serving results, copy-heavy generation gains from n-gram drafting, rejected tuning variants, and measurement limits.
Verified single-stream run of deepseek-ai/DeepSeek-V4.1-Flash on 4x NVIDIA GB10 (DGX Spark, 512 GB unified, TP4, SGLang, DSPARK spec decoding): 76.0 tok/s decode, 887.6 tok/s prefill, 308.7 ms TTFT, 423 GB unified memory in use, 37.2% draft acceptance.
How a 2.13-bpw ternary 27B, a self-speculative DFlash2 draft head, an n-gram drafting layer, and four small llama.cpp patches took a single RTX 5080 to 895 tok/s warm / 142 tok/s fresh - with verified runs, kernel-level analysis, and a full reproduction package.
vLLM 0.29 on a 64GB CMP 170HX. AutoRound W4A16 + DFlash2, production max-model-len 131072. 133 tok/s short decode, 98 tok/s with a 7.3k prefix. Not an 8k-ctx speed-run.
A 65,100-token prompt on a single 16 GB card: 4-bit KV cache (1.1 GB for the whole context), host-staged embedding table, 99 s TTFT (was 146 s before the flash-attention fix), 23.46 tok/s decode, 14.24 GB peak. Speculative decoding is unavailable past 4K context by design.
Four 4K-context streams on one 16 GB RDNA4 card: custom IQ4_S256 quant, 4-bit KV cache, 16K shared paged pool. Plain batched decode matches speculative decoding at this batch (98.2 vs 98.3 tok/s) with 1.65 GB more headroom.
Single 16 GB RDNA4 card, custom IQ4_S256 quant, WY-form GDN prefill + WMMA flash attention; lossless speculative decoding verified byte-identical across three configs.
Two RTX 3060 12GB, no NVLink. llama.cpp b10682, Unsloth UD-Q4_K_XL, 88k. Tensor split + native MTP + q4_0 draft KV: 129 tok/s code / 104 tok/s prose. 96k+MTP does not fit.
Two RTX 3060 12GB, no NVLink. llama.cpp b10682, Unsloth UD-Q4_K_XL, 64k text-only. Tensor split + native MTP: 57 tok/s code / 46 tok/s prose. DFLASH2 on layer split lost.
An apples-to-apples single-RTX-3090 comparison of FreeToken expert offload and vLLM CPU offload with Qwen3.6-35B-A3B BF16. FreeToken reached 42.78 tok/s established decode versus 7.01 tok/s for tuned vLLM, while vLLM delivered lower TTFT. Includes sequential, prefill, concurrency, memory, tuning, and MTP/DFlash compatibility findings.
Four-way Qwen3.8-27B GPTQ-4bit comparison: vLLM and SGLang with native MTP versus DFlash2. DFlash wins decode throughput; SGLang wins low-concurrency TTFT.
Apples-to-apples DFlash2 comparison across concurrency 1, 2, 4, 8, and 16. SGLang wins at concurrency 4; vLLM reaches 430.67 tok/s at concurrency 16 and avoids SGLang's five-request Mamba cache cap.
A reproducible analysis of our vLLM 0.26.0 stack: GPTQ-Marlin, TP2, FP8 KV cache, 262K context, native MTP, prefix caching, 96.6 tok/s single-stream decode, and 175.7 tok/s at concurrency 16.
A five-day, 89-task Terminal-Bench 2.1 run on two RTX 3090s: methodology, first-pass and adaptive scores, 880M processed tokens, failure analysis, and lessons for routing reasoning effort.
