Speed test matrix

Qwen3.8-27B-INT4 on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · INT4 W4A16 g1286 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 28, 2026, 20:22 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: INT4 W4A16 g128
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve RedHatAI/Qwen3.8-27B-INT4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}

Prefill

prompt tok/s by context length

8K context1,9224,261 ms TTFT

16K context1,8038,964 ms TTFT

32K context1,65819,374 ms TTFT

64K context1,42045,127 ms TTFT

128K context1,101116,235 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	67.9170 ms TTFT	136.01,512 ms TTFT	248.62,659 ms TTFT	419.12,777 ms TTFT	626.53,287 ms TTFT	713.12,250 ms TTFT	864.53,459 ms TTFT	823.2Requests queued4,652 ms TTFT
16K	69.11,198 ms TTFT	135.32,259 ms TTFT	217.93,844 ms TTFT	332.15,996 ms TTFT	466.110,335 ms TTFT	505.819,109 ms TTFT	Over capacity	Over capacity
32K	68.11,414 ms TTFT	121.82,628 ms TTFT	202.44,458 ms TTFT	279.36,954 ms TTFT	377.611,931 ms TTFT	Over capacity	Over capacity	Over capacity
64K	63.71,853 ms TTFT	105.33,422 ms TTFT	162.35,698 ms TTFT	216.18,787 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	57.54,856 ms TTFT	90.27,886 ms TTFT	128.312,971 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
248K	34.9Requests queued7,538 ms TTFT	44.7Requests queued12,147 ms TTFT	85.919,879 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	1,922.0	—	4,261 ms	—	—	1	—	—
Prefill	16K	1	Valid	1,803.0	—	8,964 ms	—	—	1	—	—
Prefill	32K	1	Valid	1,658.0	—	19,374 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,420.0	—	45,127 ms	—	—	1	—	—
Prefill	128K	1	Valid	1,101.0	—	116,235 ms	—	—	1	—	—
Sustained decode	0	1	Valid	67.9	68.5	170 ms	14.6 ms	1.0	0	2,037	30.0 s
Sustained decode	0	2	Valid	136.0	65.5	1,512 ms	15.3 ms	2.0	0	4,078	30.0 s
Sustained decode	0	4	Valid	248.6	61.5	2,659 ms	16.3 ms	4.0	0	7,456	30.0 s
Sustained decode	0	8	Valid	419.1	52.0	2,777 ms	19.2 ms	8.0	0	12,562	30.0 s
Sustained decode	0	16	Valid	626.5	40.2	3,287 ms	24.8 ms	16.0	0	18,782	30.0 s
Sustained decode	0	32	Valid	713.1	22.9	2,250 ms	43.7 ms	32.0	0	21,365	30.0 s
Sustained decode	0	64	Valid	864.5	13.8	3,459 ms	72.5 ms	64.0	0	25,846	29.9 s
Sustained decode	0	128	Requests queued	823.2	11.5	4,652 ms	86.8 ms	73.0	0	24,705	30.0 s
Sustained decode	16K	1	Valid	69.1	69.8	1,198 ms	14.3 ms	1.0	0	2,073	30.0 s
Sustained decode	16K	2	Valid	135.3	67.8	2,259 ms	14.7 ms	2.0	0	4,057	30.0 s
Sustained decode	16K	4	Valid	217.9	53.6	3,844 ms	18.7 ms	4.0	0	6,532	30.0 s
Sustained decode	16K	8	Valid	332.1	39.3	5,996 ms	25.4 ms	8.0	0	9,962	30.0 s
Sustained decode	16K	16	Valid	466.1	24.4	10,335 ms	41.0 ms	16.0	0	13,966	30.0 s
Sustained decode	16K	32	Valid	505.8	11.7	19,109 ms	85.5 ms	32.0	0	15,175	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	68.1	68.7	1,414 ms	14.6 ms	1.0	0	2,044	30.0 s
Sustained decode	32K	2	Valid	121.8	61.5	2,628 ms	16.3 ms	2.0	0	3,651	30.0 s
Sustained decode	32K	4	Valid	202.4	51.6	4,458 ms	19.4 ms	4.0	0	6,068	30.0 s
Sustained decode	32K	8	Valid	279.3	33.0	6,954 ms	30.3 ms	8.0	0	8,373	30.0 s
Sustained decode	32K	16	Valid	377.6	20.0	11,931 ms	50.0 ms	16.0	0	11,328	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	63.7	64.0	1,853 ms	15.6 ms	1.0	0	1,912	30.0 s
Sustained decode	64K	2	Valid	105.3	53.6	3,422 ms	18.7 ms	2.0	0	3,158	30.0 s
Sustained decode	64K	4	Valid	162.3	40.9	5,698 ms	24.5 ms	4.0	0	4,862	30.0 s
Sustained decode	64K	8	Valid	216.1	24.7	8,787 ms	40.6 ms	8.0	0	6,474	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	57.5	58.0	4,856 ms	17.2 ms	1.0	0	1,725	30.0 s
Sustained decode	128K	2	Valid	90.2	44.6	7,886 ms	22.5 ms	2.0	0	2,702	30.0 s
Sustained decode	128K	4	Valid	128.3	29.2	12,971 ms	34.2 ms	4.0	0	3,841	29.9 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Requests queued	34.9	45.5	7,538 ms	22.0 ms	0.8	0	1,045	30.0 s
Sustained decode	248K	2	Requests queued	44.7	36.6	12,147 ms	27.4 ms	1.4	0	1,341	30.0 s
Sustained decode	248K	4	Valid	85.9	18.8	19,879 ms	53.9 ms	4.0	0	2,573	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.