Speed test matrix

Qwen3.5-4B-FP8-dynamic on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · FP8 dynamic6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 30, 2026, 08:42 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: FP8 dynamic
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve RedHatAI/Qwen3.5-4B-FP8-dynamic --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}

Prefill

prompt tok/s by context length

8K context10.6K770 ms TTFT

16K context9,8411,638 ms TTFT

32K context9,0363,550 ms TTFT

64K context7,3558,704 ms TTFT

128K context5,30924,091 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	182.7128 ms TTFT	363.9225 ms TTFT	667.3214 ms TTFT	1,057.9279 ms TTFT	1,685.2347 ms TTFT	2,374.9501 ms TTFT	3,022.5721 ms TTFT	3,662.71,052 ms TTFT
16K	127.2224 ms TTFT	245.6373 ms TTFT	449.8541 ms TTFT	601.8721 ms TTFT	755.11,199 ms TTFT	905.92,072 ms TTFT	Over capacity	Over capacity
32K	90.9364 ms TTFT	181.3482 ms TTFT	328.9674 ms TTFT	398.81,085 ms TTFT	467.51,868 ms TTFT	547.33,374 ms TTFT	Over capacity	Over capacity
64K	58.1615 ms TTFT	117.1852 ms TTFT	218.31,144 ms TTFT	245.21,715 ms TTFT	271.72,745 ms TTFT	Over capacity	Over capacity	Over capacity
128K	35.71,064 ms TTFT	70.01,279 ms TTFT	126.71,785 ms TTFT	134.03,034 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	20.21,915 ms TTFT	38.12,702 ms TTFT	71.73,751 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	10.6K	—	770 ms	—	—	1	—	—
Prefill	16K	1	Valid	9,841.0	—	1,638 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,036.0	—	3,550 ms	—	—	1	—	—
Prefill	64K	1	Valid	7,355.0	—	8,704 ms	—	—	1	—	—
Prefill	128K	1	Valid	5,309.0	—	24,091 ms	—	—	1	—	—
Sustained decode	0	1	Valid	182.7	188.1	128 ms	5.3 ms	1.0	0	5,481	30.0 s
Sustained decode	0	2	Valid	363.9	185.6	225 ms	5.4 ms	2.0	0	10,917	30.0 s
Sustained decode	0	4	Valid	667.3	170.0	214 ms	5.9 ms	4.0	0	20,017	30.0 s
Sustained decode	0	8	Valid	1,057.9	136.8	279 ms	7.3 ms	8.0	0	31,738	30.0 s
Sustained decode	0	16	Valid	1,685.2	110.8	347 ms	9.0 ms	16.0	0	50,529	30.0 s
Sustained decode	0	32	Valid	2,374.9	78.0	501 ms	12.8 ms	32.0	0	71,223	30.0 s
Sustained decode	0	64	Valid	3,022.5	49.3	721 ms	20.3 ms	64.0	0	90,728	30.0 s
Sustained decode	0	128	Valid	3,662.7	29.9	1,052 ms	33.5 ms	128.0	0	109,990	30.0 s
Sustained decode	16K	1	Valid	127.2	129.0	224 ms	7.8 ms	1.0	0	3,815	30.0 s
Sustained decode	16K	2	Valid	245.6	124.5	373 ms	8.0 ms	2.0	0	7,368	30.0 s
Sustained decode	16K	4	Valid	449.8	114.1	541 ms	8.8 ms	4.0	0	13,490	30.0 s
Sustained decode	16K	8	Valid	601.8	76.8	721 ms	13.0 ms	8.0	0	18,054	30.0 s
Sustained decode	16K	16	Valid	755.1	47.5	1,199 ms	21.1 ms	16.0	0	22,642	30.0 s
Sustained decode	16K	32	Valid	905.9	28.4	2,072 ms	35.2 ms	32.0	0	27,157	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	90.9	92.4	364 ms	10.8 ms	1.0	0	2,728	30.0 s
Sustained decode	32K	2	Valid	181.3	92.1	482 ms	10.9 ms	2.0	0	5,436	30.0 s
Sustained decode	32K	4	Valid	328.9	83.1	674 ms	12.0 ms	4.0	0	9,866	30.0 s
Sustained decode	32K	8	Valid	398.8	51.1	1,085 ms	19.6 ms	8.0	0	11,964	30.0 s
Sustained decode	32K	16	Valid	467.5	29.4	1,868 ms	34.0 ms	16.0	0	14,009	30.0 s
Sustained decode	32K	32	Valid	547.3	17.1	3,374 ms	58.6 ms	32.0	0	16,402	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	58.1	59.6	615 ms	16.8 ms	1.0	0	1,741	30.0 s
Sustained decode	64K	2	Valid	117.1	59.8	852 ms	16.7 ms	2.0	0	3,513	30.0 s
Sustained decode	64K	4	Valid	218.3	55.5	1,144 ms	18.0 ms	4.0	0	6,545	30.0 s
Sustained decode	64K	8	Valid	245.2	31.3	1,715 ms	31.9 ms	8.0	0	7,356	30.0 s
Sustained decode	64K	16	Valid	271.7	17.3	2,745 ms	57.9 ms	16.0	0	8,131	29.9 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	35.7	36.3	1,064 ms	27.5 ms	1.0	0	1,069	30.0 s
Sustained decode	128K	2	Valid	70.0	35.8	1,279 ms	28.0 ms	2.0	0	2,097	30.0 s
Sustained decode	128K	4	Valid	126.7	32.5	1,785 ms	30.8 ms	4.0	0	3,801	30.0 s
Sustained decode	128K	8	Valid	134.0	17.5	3,034 ms	57.1 ms	8.0	0	4,012	29.9 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	20.2	21.0	1,915 ms	47.7 ms	1.0	0	607	30.0 s
Sustained decode	248K	2	Valid	38.1	19.8	2,702 ms	50.5 ms	2.0	0	1,142	30.0 s
Sustained decode	248K	4	Valid	71.7	18.5	3,751 ms	54.1 ms	4.0	0	2,147	29.9 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.