Speed test matrix

DeepSeek-V4.1-Flash on H200 SXM 141GB ×4

vllm · 0.1.1.dev5+ge77daef89 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 14, 2026, 06:17 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.1.1.dev5+ge77daef89
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: e77daef89e18

Run

Served model: deepseek
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-deepseek-v41-vllm-e77daef89/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 4 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true}'

Prefill

prompt tok/s by context length

8K context14.8K555 ms TTFT

16K context16.1K1,010 ms TTFT

32K context16.2K1,993 ms TTFT

64K context15.9K4,043 ms TTFT

128K context15.2K8,466 ms TTFT

256K context13.7K18,801 ms TTFT

512K context11.4K45,246 ms TTFT

1019K context7,088144,197 ms TTFT

1024K context7,046145,770 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	124.553 ms TTFT	237.384 ms TTFT	433.1103 ms TTFT	759.8155 ms TTFT	1,225.8223 ms TTFT	1,936.4384 ms TTFT	2,821.9621 ms TTFT	4,258.21,169 ms TTFT
16K	124.265 ms TTFT	236.594 ms TTFT	431.0116 ms TTFT	752.0179 ms TTFT	1,219.0304 ms TTFT	1,917.4546 ms TTFT	2,811.1967 ms TTFT	4,215.61,907 ms TTFT
32K	123.9108 ms TTFT	236.4169 ms TTFT	429.0210 ms TTFT	755.6338 ms TTFT	1,216.9579 ms TTFT	1,916.91,007 ms TTFT	2,801.21,811 ms TTFT	4,183.03,547 ms TTFT
64K	123.4184 ms TTFT	235.7289 ms TTFT	428.4369 ms TTFT	748.4597 ms TTFT	1,216.71,059 ms TTFT	1,896.51,938 ms TTFT	2,757.13,713 ms TTFT	Over capacity
128K	123.5354 ms TTFT	234.5501 ms TTFT	424.9752 ms TTFT	742.61,241 ms TTFT	1,200.22,228 ms TTFT	1,857.14,231 ms TTFT	Over capacity	Over capacity
256K	122.7735 ms TTFT	232.71,075 ms TTFT	421.31,589 ms TTFT	725.12,738 ms TTFT	1,168.94,957 ms TTFT	Over capacity	Over capacity	Over capacity
512K	121.11,647 ms TTFT	228.22,355 ms TTFT	412.13,729 ms TTFT	708.26,473 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
1019K	117.93,733 ms TTFT	219.75,228 ms TTFT	387.88,539 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	14.8K	—	555 ms	—	—	1	—	—
Prefill	16K	1	Valid	16.1K	—	1,010 ms	—	—	1	—	—
Prefill	32K	1	Valid	16.2K	—	1,993 ms	—	—	1	—	—
Prefill	64K	1	Valid	15.9K	—	4,043 ms	—	—	1	—	—
Prefill	128K	1	Valid	15.2K	—	8,466 ms	—	—	1	—	—
Prefill	256K	1	Valid	13.7K	—	18,801 ms	—	—	1	—	—
Prefill	512K	1	Valid	11.4K	—	45,246 ms	—	—	1	—	—
Prefill	1019K	1	Valid	7,088.0	—	144,197 ms	—	—	1	—	—
Prefill	1024K	1	Valid	7,046.0	—	145,770 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	124.5	125.2	53 ms	8.0 ms	1.0	1	3,735	30.0 s
Sustained decode	8K	2	Valid	237.3	118.6	84 ms	8.4 ms	2.0	0	7,117	30.0 s
Sustained decode	8K	4	Valid	433.1	108.3	103 ms	9.2 ms	4.0	0	12,992	30.0 s
Sustained decode	8K	8	Valid	759.8	95.2	155 ms	10.5 ms	8.0	0	22,791	30.0 s
Sustained decode	8K	16	Valid	1,225.8	77.0	223 ms	13.0 ms	16.0	0	36,769	30.0 s
Sustained decode	8K	32	Valid	1,936.4	60.8	384 ms	16.4 ms	32.0	0	58,081	30.0 s
Sustained decode	8K	64	Valid	2,821.9	44.3	621 ms	22.6 ms	64.0	0	84,609	30.0 s
Sustained decode	8K	128	Valid	4,258.2	33.2	1,169 ms	30.1 ms	128.0	0	127,750	30.0 s
Sustained decode	16K	1	Valid	124.2	125.3	65 ms	8.0 ms	1.0	1	3,726	30.0 s
Sustained decode	16K	2	Valid	236.5	118.3	94 ms	8.5 ms	2.0	0	7,094	30.0 s
Sustained decode	16K	4	Valid	431.0	107.8	116 ms	9.3 ms	4.0	0	12,928	30.0 s
Sustained decode	16K	8	Valid	752.0	94.2	179 ms	10.6 ms	8.0	0	22,553	30.0 s
Sustained decode	16K	16	Valid	1,219.0	76.4	304 ms	13.1 ms	16.0	0	36,570	30.0 s
Sustained decode	16K	32	Valid	1,917.4	60.2	546 ms	16.6 ms	32.0	0	57,507	30.0 s
Sustained decode	16K	64	Valid	2,811.1	44.1	967 ms	22.7 ms	64.0	0	84,288	30.0 s
Sustained decode	16K	128	Valid	4,215.6	33.1	1,907 ms	30.2 ms	128.0	0	126,471	30.0 s
Sustained decode	32K	1	Valid	123.9	126.7	108 ms	7.9 ms	1.0	1	3,715	30.0 s
Sustained decode	32K	2	Valid	236.4	118.2	169 ms	8.5 ms	2.0	0	7,090	30.0 s
Sustained decode	32K	4	Valid	429.0	107.4	210 ms	9.3 ms	4.0	0	12,868	30.0 s
Sustained decode	32K	8	Valid	755.6	94.5	338 ms	10.6 ms	8.0	0	22,664	30.0 s
Sustained decode	32K	16	Valid	1,216.9	76.1	579 ms	13.1 ms	16.0	0	36,497	30.0 s
Sustained decode	32K	32	Valid	1,916.9	60.0	1,007 ms	16.7 ms	32.0	0	57,507	30.0 s
Sustained decode	32K	64	Valid	2,801.2	43.8	1,811 ms	22.8 ms	64.0	0	84,032	30.0 s
Sustained decode	32K	128	Valid	4,183.0	32.6	3,547 ms	30.7 ms	128.0	0	125,443	30.0 s
Sustained decode	64K	1	Valid	123.4	134.2	184 ms	7.5 ms	1.0	1	3,701	30.0 s
Sustained decode	64K	2	Valid	235.7	117.9	289 ms	8.5 ms	2.0	0	7,072	30.0 s
Sustained decode	64K	4	Valid	428.4	107.1	369 ms	9.3 ms	4.0	0	12,852	30.0 s
Sustained decode	64K	8	Valid	748.4	93.6	597 ms	10.7 ms	8.0	0	22,448	30.0 s
Sustained decode	64K	16	Valid	1,216.7	76.1	1,059 ms	13.1 ms	16.0	0	36,496	30.0 s
Sustained decode	64K	32	Valid	1,896.5	59.4	1,938 ms	16.8 ms	32.0	0	56,895	30.0 s
Sustained decode	64K	64	Valid	2,757.1	43.3	3,713 ms	23.1 ms	64.0	0	82,689	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	123.5	123.7	354 ms	8.1 ms	1.0	1	3,697	29.9 s
Sustained decode	128K	2	Valid	234.5	117.4	501 ms	8.5 ms	2.0	0	7,032	30.0 s
Sustained decode	128K	4	Valid	424.9	106.3	752 ms	9.4 ms	4.0	0	12,746	30.0 s
Sustained decode	128K	8	Valid	742.6	92.9	1,241 ms	10.8 ms	8.0	0	22,273	30.0 s
Sustained decode	128K	16	Valid	1,200.2	75.0	2,228 ms	13.3 ms	16.0	0	36,000	30.0 s
Sustained decode	128K	32	Valid	1,857.1	58.0	4,231 ms	17.3 ms	32.0	0	55,712	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	122.7	123.0	735 ms	8.1 ms	1.0	1	3,618	29.5 s
Sustained decode	256K	2	Valid	232.7	116.5	1,075 ms	8.6 ms	2.0	0	6,980	30.0 s
Sustained decode	256K	4	Valid	421.3	105.3	1,589 ms	9.5 ms	4.0	0	12,636	30.0 s
Sustained decode	256K	8	Valid	725.1	90.7	2,738 ms	11.0 ms	8.0	0	21,752	30.0 s
Sustained decode	256K	16	Valid	1,168.9	72.9	4,957 ms	13.7 ms	16.0	0	35,058	30.0 s
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	121.1	121.6	1,647 ms	8.2 ms	1.0	0	3,631	30.0 s
Sustained decode	512K	2	Valid	228.2	114.5	2,355 ms	8.7 ms	2.0	0	6,844	30.0 s
Sustained decode	512K	4	Valid	412.1	103.1	3,729 ms	9.7 ms	4.0	0	12,361	30.0 s
Sustained decode	512K	8	Valid	708.2	88.4	6,473 ms	11.3 ms	8.0	0	21,240	30.0 s
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	117.9	118.9	3,733 ms	8.4 ms	1.0	0	3,538	30.0 s
Sustained decode	1019K	2	Valid	219.7	110.5	5,228 ms	9.1 ms	2.0	0	6,588	30.0 s
Sustained decode	1019K	4	Valid	387.8	98.7	8,539 ms	10.1 ms	4.0	1	11,631	30.0 s
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.