Reports

Community-authored local inference reports

Published field reportsnewest

Qwen3.8-27B long-context test on dual RTX 3060 12GB

Qwen3.8-27B on two RTX 3060 12GB cards, llama.cpp 0.4.1-dev. Measured 129,000-token synthetic prompt at 497.8 prompt tok/s and 42.7 output tok/s mean, with exact recall of 3/3 canary facts in two runs. Configured context 131,072; ngram-mod + MTP3; batch 2048, ubatch 512.

Qwen3.8-27B on RX 9070 XT: one 64K-context stream, 655 tok/s prefill, 23.5 tok/s decode on 16 GB

Qwen3.8-27B

A 65,100-token prompt on a single 16 GB card: 4-bit KV cache (1.1 GB for the whole context), host-staged embedding table, 99 s TTFT (was 146 s before the flash-attention fix), 23.46 tok/s decode, 14.24 GB peak. Speculative decoding is unavailable past 4K context by design.

long-context rocm amd-rdna4 qwen3.8 kv-cache-quantization throughput

Qwen3.8-27B on RX 9070 XT: 4 concurrent streams at 4K context, 98.3 tok/s aggregate (custom HIP engine)

Qwen3.8-27B

Four 4K-context streams on one 16 GB RDNA4 card: custom IQ4_S256 quant, 4-bit KV cache, 16K shared paged pool. Plain batched decode matches speculative decoding at this batch (98.2 vs 98.3 tok/s) with 1.65 GB more headroom.

throughput multi-stream rocm amd-rdna4 qwen3.8 batching speculative-decoding

Qwen3.8-27B on dual RTX 3060 12GB (llama.cpp tensor + MTP)

Qwen3.8-27B-GGUF

Two RTX 3060 12GB, no NVLink. llama.cpp b10682, Unsloth UD-Q4_K_XL, 64k text-only. Tensor split + native MTP: 57 tok/s code / 46 tok/s prose. DFLASH2 on layer split lost.

1 speed test

throughput multi-gpu rtx-3060 llama.cpp qwen3.8