Reports

Community-authored local inference reports

Published field reportsnewest

Qwen3.8-27B on one Tenstorrent P150: 7.7 to 50+ tok/s, native 262K context, and lessons learned

Five weeks taking Qwen3.8-27B on a single Tenstorrent Blackhole P150 from 7.7 tok/s to ~50-55 tok/s single-stream decode (DFlash2), ~1,200 tok/s prefill and native 262K context, using a custom mixed BFP4/BFP8 quant (now GPTQ). Covers the quant, every speed-test milestone, quality checks, power and thermal data, the dead ends, lessons learned and what's next.

BFP4 vllmTenstorrent P150 · 32 GB

3 speed tests

Qwen3.8-27B long-context test on dual RTX 3060 12GB

Qwen3.8-27B

Qwen3.8-27B on two RTX 3060 12GB cards, llama.cpp 0.4.1-dev. Measured 129,000-token synthetic prompt at 497.8 prompt tok/s and 42.7 output tok/s mean, with exact recall of 3/3 canary facts in two runs. Configured context 131,072; ngram-mod + MTP3; batch 2048, ubatch 512.

long-context throughput multi-gpu rtx-3060 llama.cpp qwen3.8

CMP 170HX: Qwen3.8-27B AutoRound + DFlash2 at 131k

Qwen3.8-27B-W4A16-AutoRound

vLLM 0.29 on a 64GB CMP 170HX. AutoRound W4A16 + DFlash2, production max-model-len 131072. 133 tok/s short decode, 98 tok/s with a 7.3k prefix. Not an 8k-ctx speed-run.

W4A16 vllmCMP 170HX · 64 GB

2 speed tests

throughput autoround dflash2 long-context ampere

Qwen3.8-27B on RX 9070 XT: one 64K-context stream, 655 tok/s prefill, 23.5 tok/s decode on 16 GB

Qwen3.8-27B

A 65,100-token prompt on a single 16 GB card: 4-bit KV cache (1.1 GB for the whole context), host-staged embedding table, 99 s TTFT (was 146 s before the flash-attention fix), 23.46 tok/s decode, 14.24 GB peak. Speculative decoding is unavailable past 4K context by design.

long-context rocm amd-rdna4 qwen3.8 kv-cache-quantization throughput