Speed test matrix

Qwen3.5-27B-GPTQ-Int4 on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 6, 2026, 15:08 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883643 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Prefill

prompt tok/s by context length

8K context10.1K815 ms TTFT

16K context9,7761,659 ms TTFT

32K context9,3343,459 ms TTFT

64K context8,5757,513 ms TTFT

128K context7,28417,669 ms TTFT

251K context5,62144,867 ms TTFT

256K context5,58946,018 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	148.2135 ms TTFT	281.6336 ms TTFT	538.0273 ms TTFT	994.8350 ms TTFT	1,674.31,673 ms TTFT	2,489.73,043 ms TTFT	3,104.64,275 ms TTFT	Over capacity
16K	142.5162 ms TTFT	277.2228 ms TTFT	526.9343 ms TTFT	965.0838 ms TTFT	1,561.11,622 ms TTFT	2,206.72,739 ms TTFT	2,676.14,372 ms TTFT	Over capacity
32K	141.0226 ms TTFT	275.5293 ms TTFT	505.7345 ms TTFT	903.61,199 ms TTFT	1,353.21,919 ms TTFT	1,805.42,878 ms TTFT	Over capacity	Over capacity
64K	135.7320 ms TTFT	261.1401 ms TTFT	470.3803 ms TTFT	765.71,368 ms TTFT	1,063.52,548 ms TTFT	Over capacity	Over capacity	Over capacity
128K	130.3531 ms TTFT	232.2700 ms TTFT	397.91,220 ms TTFT	572.32,242 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	116.7801 ms TTFT	194.91,045 ms TTFT	295.01,514 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	10.1K	—	815 ms	—	—	1	—	—
Prefill	16K	1	Valid	9,776.0	—	1,659 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,334.0	—	3,459 ms	—	—	1	—	—
Prefill	64K	1	Valid	8,575.0	—	7,513 ms	—	—	1	—	—
Prefill	128K	1	Valid	7,284.0	—	17,669 ms	—	—	1	—	—
Prefill	251K	1	Valid	5,621.0	—	44,867 ms	—	—	1	—	—
Prefill	256K	1	Valid	5,589.0	—	46,018 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	148.2	148.2	135 ms	6.7 ms	1.0	1	4,449	30.0 s
Sustained decode	8K	2	Valid	281.6	144.5	336 ms	6.9 ms	2.0	2	8,447	30.0 s
Sustained decode	8K	4	Valid	538.0	136.7	273 ms	7.3 ms	4.0	4	16,133	30.0 s
Sustained decode	8K	8	Valid	994.8	125.2	350 ms	8.0 ms	8.0	8	29,832	30.0 s
Sustained decode	8K	16	Valid	1,674.3	105.0	1,673 ms	9.5 ms	16.0	0	50,229	30.0 s
Sustained decode	8K	32	Valid	2,489.7	77.7	3,043 ms	12.9 ms	32.0	0	74,637	30.0 s
Sustained decode	8K	64	Valid	3,104.6	46.2	4,275 ms	21.6 ms	64.0	0	93,056	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	142.5	144.4	162 ms	6.9 ms	1.0	1	4,276	30.0 s
Sustained decode	16K	2	Valid	277.2	140.8	228 ms	7.1 ms	2.0	2	8,314	30.0 s
Sustained decode	16K	4	Valid	526.9	134.1	343 ms	7.5 ms	3.9	4	15,798	30.0 s
Sustained decode	16K	8	Valid	965.0	120.8	838 ms	8.3 ms	8.0	3	28,930	30.0 s
Sustained decode	16K	16	Valid	1,561.1	97.4	1,622 ms	10.3 ms	16.0	0	46,812	30.0 s
Sustained decode	16K	32	Valid	2,206.7	68.5	2,739 ms	14.6 ms	32.0	0	66,178	30.0 s
Sustained decode	16K	64	Valid	2,676.1	39.5	4,372 ms	25.3 ms	64.0	0	80,295	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	141.0	143.1	226 ms	7.0 ms	1.0	1	4,227	30.0 s
Sustained decode	32K	2	Valid	275.5	139.1	293 ms	7.2 ms	2.0	2	8,267	30.0 s
Sustained decode	32K	4	Valid	505.7	127.3	345 ms	7.9 ms	4.0	4	15,167	30.0 s
Sustained decode	32K	8	Valid	903.6	113.2	1,199 ms	8.8 ms	8.0	0	27,096	30.0 s
Sustained decode	32K	16	Valid	1,353.2	85.0	1,919 ms	11.8 ms	16.0	0	40,580	30.0 s
Sustained decode	32K	32	Valid	1,805.4	55.2	2,878 ms	18.1 ms	32.0	0	54,152	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	135.7	137.0	320 ms	7.3 ms	1.0	1	4,069	30.0 s
Sustained decode	64K	2	Valid	261.1	134.4	401 ms	7.4 ms	2.0	2	7,830	30.0 s
Sustained decode	64K	4	Valid	470.3	118.0	803 ms	8.5 ms	4.0	0	14,109	30.0 s
Sustained decode	64K	8	Valid	765.7	95.4	1,368 ms	10.5 ms	8.0	0	22,955	30.0 s
Sustained decode	64K	16	Valid	1,063.5	66.4	2,548 ms	15.1 ms	16.0	0	31,903	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	130.3	140.3	531 ms	7.1 ms	1.0	1	3,908	30.0 s
Sustained decode	128K	2	Valid	232.2	116.4	700 ms	8.6 ms	2.0	0	6,965	30.0 s
Sustained decode	128K	4	Valid	397.9	99.3	1,220 ms	10.1 ms	4.0	0	11,929	30.0 s
Sustained decode	128K	8	Valid	572.3	71.7	2,242 ms	13.9 ms	8.0	0	17,166	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	116.7	117.7	801 ms	8.5 ms	1.0	0	3,499	30.0 s
Sustained decode	251K	2	Valid	194.9	98.2	1,045 ms	10.2 ms	2.0	0	5,844	30.0 s
Sustained decode	251K	4	Valid	295.0	73.5	1,514 ms	13.6 ms	4.0	0	8,844	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.