Reports

Community-authored local inference reports

Published field reportsnewest

FreeToken vs vLLM on one RTX 3090: BF16 MoE RAM offload compared

An apples-to-apples single-RTX-3090 comparison of FreeToken expert offload and vLLM CPU offload with Qwen3.6-35B-A3B BF16. FreeToken reached 42.78 tok/s established decode versus 7.01 tok/s for tuned vLLM, while vLLM delivered lower TTFT. Includes sequential, prefill, concurrency, memory, tuning, and MTP/DFlash compatibility findings.

BF16 FreeToken,vLLM

MTP vs DFlash: vLLM and SGLang on dual RTX 3090s

Qwen3.8-27B-GPTQ-4bit

Four-way Qwen3.8-27B GPTQ-4bit comparison: vLLM and SGLang with native MTP versus DFlash2. DFlash wins decode throughput; SGLang wins low-concurrency TTFT.

vLLM vs SGLang: Qwen3.8-27B Q4 DFlash2 on dual RTX 3090s

Qwen3.8-27B-GPTQ-4bit

Apples-to-apples DFlash2 comparison across concurrency 1, 2, 4, 8, and 16. SGLang wins at concurrency 4; vLLM reaches 430.67 tok/s at concurrency 16 and avoids SGLang's five-request Mamba cache cap.

Qwen3.8 27B GPTQ + MTP on 2× RTX 3090: the vLLM setup behind our Terminal-Bench run

Qwen3.8-27B-GPTQ-4bit

A reproducible analysis of our vLLM 0.26.0 stack: GPTQ-Marlin, TP2, FP8 KV cache, 262K context, native MTP, prefix caching, 96.6 tok/s single-stream decode, and 175.7 tok/s at concurrency 16.

1 benchmark

Qwen3.8 27B on Terminal-Bench 2.1: 69.7% xhigh, 78.7% adaptive

Qwen3.8-27B-GPTQ-4bit

A five-day, 89-task Terminal-Bench 2.1 run on two RTX 3090s: methodology, first-pass and adaptive scores, 880M processed tokens, failure analysis, and lessons for routing reasoning effort.