Speed test matrix

DeepSeek-V4-Flash-Vision-Exp on H200 SXM 141GB ×4

vllm · 0.28.1rc1.dev317+g1356635d8 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 11, 2026, 15:09 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.28.1rc1.dev317+g1356635d8
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 1356635d837c

Run

Served model: deepseek
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-deepseek-vision-vllm-pr-54566/bin/vllm" serve deepseek-ai/DeepSeek-V4-Flash-Vision-Exp --served-model-name deepseek --trust-remote-code --tensor-parallel-size 4 --reasoning-parser deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --max-model-len 1048576 --gpu-memory-utilization 0.843993 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4 --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}'

Prefill

prompt tok/s by context length

8K context18.5K444 ms TTFT

16K context21.3K757 ms TTFT

32K context21.8K1,475 ms TTFT

64K context21.5K2,976 ms TTFT

128K context20.2K6,335 ms TTFT

256K context6,36740,096 ms TTFT

512K context8,54259,743 ms TTFT

1019K context8,740116,184 ms TTFT

1024K context8,452120,735 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	146.391 ms TTFT	277.8113 ms TTFT	512.9299 ms TTFT	895.621,557 ms TTFT	1,509.1516 ms TTFT	Looped	Looped	6,158.31,580 ms TTFT
16K	145.4134 ms TTFT	276.9141 ms TTFT	511.4172 ms TTFT	891.1273 ms TTFT	1,476.2385 ms TTFT	2,441.2674 ms TTFT	3,849.41,041 ms TTFT	6,099.91,897 ms TTFT
32K	144.5185 ms TTFT	274.9216 ms TTFT	506.8274 ms TTFT	882.3435 ms TTFT	1,489.7657 ms TTFT	2,408.21,054 ms TTFT	3,811.01,942 ms TTFT	6,000.43,410 ms TTFT
64K	142.8315 ms TTFT	271.1380 ms TTFT	501.1396 ms TTFT	876.8765 ms TTFT	1,462.21,128 ms TTFT	2,293.31,960 ms TTFT	3,652.63,579 ms TTFT	5,764.36,938 ms TTFT
128K	140.8509 ms TTFT	267.8657 ms TTFT	489.1730 ms TTFT	849.91,347 ms TTFT	1,417.52,312 ms TTFT	2,251.64,236 ms TTFT	3,472.38,042 ms TTFT	5,329.915,997 ms TTFT
256K	137.0916 ms TTFT	258.31,016 ms TTFT	480.21,642 ms TTFT	835.82,620 ms TTFT	1,373.54,569 ms TTFT	2,110.38,370 ms TTFT	3,222.116,319 ms TTFT	4,739.431,823 ms TTFT
512K	128.51,851 ms TTFT	240.9Requests queued1,909 ms TTFT	460.93,567 ms TTFT	790.55,825 ms TTFT	1,273.510,476 ms TTFT	1,863.819,316 ms TTFT	Looped	Looped
1019K	126.43,995 ms TTFT	234.05,232 ms TTFT	426.27,874 ms TTFT	694.011,743 ms TTFT	1,012.118,491 ms TTFT	1,359.9Requests queued8,602 ms TTFT	1,451.0Requests queued88,969 ms TTFT	Looped

Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	18.5K	—	444 ms	—	—	1	—	—
Prefill	16K	1	Valid	21.3K	—	757 ms	—	—	1	—	—
Prefill	32K	1	Valid	21.8K	—	1,475 ms	—	—	1	—	—
Prefill	64K	1	Valid	21.5K	—	2,976 ms	—	—	1	—	—
Prefill	128K	1	Valid	20.2K	—	6,335 ms	—	—	1	—	—
Prefill	256K	1	Valid	6,367.0	—	40,096 ms	—	—	1	—	—
Prefill	512K	1	Valid	8,542.0	—	59,743 ms	—	—	1	—	—
Prefill	1019K	1	Valid	8,740.0	—	116,184 ms	—	—	1	—	—
Prefill	1024K	1	Valid	8,452.0	—	120,735 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	146.3	146.9	91 ms	6.8 ms	1.0	1	4,390	30.0 s
Sustained decode	8K	2	Valid	277.8	139.4	113 ms	7.2 ms	2.0	2	8,334	30.0 s
Sustained decode	8K	4	Valid	512.9	129.4	299 ms	7.7 ms	4.0	4	15,386	30.0 s
Sustained decode	8K	8	Valid	895.6	111.8	21,557 ms	8.9 ms	8.0	0	26,865	30.0 s
Sustained decode	8K	16	Valid	1,509.1	94.9	516 ms	10.5 ms	16.0	0	45,266	30.0 s
Sustained decode	8K	32	Looped	—	—	—	—	—	0	0	—
Sustained decode	8K	64	Looped	—	—	—	—	—	0	0	—
Sustained decode	8K	128	Valid	6,158.3	47.7	1,580 ms	21.0 ms	128.0	0	184,706	30.0 s
Sustained decode	16K	1	Valid	145.4	146.2	134 ms	6.8 ms	1.0	1	4,362	30.0 s
Sustained decode	16K	2	Valid	276.9	139.1	141 ms	7.2 ms	2.0	2	8,306	30.0 s
Sustained decode	16K	4	Valid	511.4	128.6	172 ms	7.8 ms	3.9	4	15,340	30.0 s
Sustained decode	16K	8	Valid	891.1	111.6	273 ms	9.0 ms	8.0	0	26,728	30.0 s
Sustained decode	16K	16	Valid	1,476.2	92.5	385 ms	10.8 ms	16.0	0	44,287	30.0 s
Sustained decode	16K	32	Valid	2,441.2	76.6	674 ms	13.1 ms	32.0	0	73,248	30.0 s
Sustained decode	16K	64	Valid	3,849.4	60.2	1,041 ms	16.6 ms	64.0	0	115,454	30.0 s
Sustained decode	16K	128	Valid	6,099.9	47.4	1,897 ms	21.1 ms	128.0	0	183,016	30.0 s
Sustained decode	32K	1	Valid	144.5	146.0	185 ms	6.9 ms	1.0	1	4,333	30.0 s
Sustained decode	32K	2	Valid	274.9	139.3	216 ms	7.2 ms	2.0	2	8,248	30.0 s
Sustained decode	32K	4	Valid	506.8	128.0	274 ms	7.8 ms	4.0	4	15,204	30.0 s
Sustained decode	32K	8	Valid	882.3	110.4	435 ms	9.1 ms	8.0	0	26,464	30.0 s
Sustained decode	32K	16	Valid	1,489.7	93.1	657 ms	10.7 ms	16.0	0	44,689	30.0 s
Sustained decode	32K	32	Valid	2,408.2	75.4	1,054 ms	13.3 ms	32.0	0	72,256	30.0 s
Sustained decode	32K	64	Valid	3,811.0	59.5	1,942 ms	16.8 ms	64.0	0	114,368	30.0 s
Sustained decode	32K	128	Valid	6,000.4	46.7	3,410 ms	21.4 ms	128.0	0	179,963	30.0 s
Sustained decode	64K	1	Valid	142.8	144.8	315 ms	6.9 ms	1.0	1	4,283	30.0 s
Sustained decode	64K	2	Valid	271.1	137.3	380 ms	7.3 ms	2.0	2	8,133	30.0 s
Sustained decode	64K	4	Valid	501.1	126.9	396 ms	7.9 ms	3.9	4	15,032	30.0 s
Sustained decode	64K	8	Valid	876.8	109.6	765 ms	9.1 ms	8.0	0	26,306	30.0 s
Sustained decode	64K	16	Valid	1,462.2	91.3	1,128 ms	11.0 ms	16.0	0	43,855	30.0 s
Sustained decode	64K	32	Valid	2,293.3	71.4	1,960 ms	14.0 ms	32.0	0	68,801	30.0 s
Sustained decode	64K	64	Valid	3,652.6	56.2	3,579 ms	17.8 ms	64.0	0	109,601	30.0 s
Sustained decode	64K	128	Valid	5,764.3	43.9	6,938 ms	22.8 ms	128.0	0	173,192	30.0 s
Sustained decode	128K	1	Valid	140.8	144.2	509 ms	6.9 ms	1.0	1	4,222	30.0 s
Sustained decode	128K	2	Valid	267.8	136.8	657 ms	7.3 ms	2.0	2	8,034	30.0 s
Sustained decode	128K	4	Valid	489.1	124.7	730 ms	8.0 ms	4.0	4	14,627	29.9 s
Sustained decode	128K	8	Valid	849.9	105.8	1,347 ms	9.5 ms	8.0	0	25,497	30.0 s
Sustained decode	128K	16	Valid	1,417.5	87.9	2,312 ms	11.4 ms	16.0	0	42,528	30.0 s
Sustained decode	128K	32	Valid	2,251.6	69.6	4,236 ms	14.4 ms	32.0	0	67,555	30.0 s
Sustained decode	128K	64	Valid	3,472.3	53.3	8,042 ms	18.8 ms	64.0	0	104,191	30.0 s
Sustained decode	128K	128	Valid	5,329.9	40.8	15,997 ms	24.5 ms	128.0	0	159,918	30.0 s
Sustained decode	256K	1	Valid	137.0	142.7	916 ms	7.0 ms	1.0	1	4,110	30.0 s
Sustained decode	256K	2	Valid	258.3	133.7	1,016 ms	7.5 ms	1.9	2	7,747	30.0 s
Sustained decode	256K	4	Valid	480.2	120.3	1,642 ms	8.3 ms	4.0	2	14,403	30.0 s
Sustained decode	256K	8	Valid	835.8	103.5	2,620 ms	9.7 ms	8.0	0	25,073	30.0 s
Sustained decode	256K	16	Valid	1,373.5	84.3	4,569 ms	11.9 ms	16.0	0	41,200	30.0 s
Sustained decode	256K	32	Valid	2,110.3	64.7	8,370 ms	15.5 ms	32.0	0	63,298	30.0 s
Sustained decode	256K	64	Valid	3,222.1	49.2	16,319 ms	20.3 ms	64.0	0	96,639	30.0 s
Sustained decode	256K	128	Valid	4,739.4	36.9	31,823 ms	27.1 ms	128.0	2	142,171	30.0 s
Sustained decode	512K	1	Valid	128.5	142.0	1,851 ms	7.0 ms	0.9	1	3,855	30.0 s
Sustained decode	512K	2	Requests queued	240.9	129.1	1,909 ms	7.7 ms	1.9	2	7,162	29.7 s
Sustained decode	512K	4	Valid	460.9	116.7	3,567 ms	8.6 ms	3.9	2	13,823	30.0 s
Sustained decode	512K	8	Valid	790.5	97.8	5,825 ms	10.2 ms	8.0	0	23,712	30.0 s
Sustained decode	512K	16	Valid	1,273.5	78.8	10,476 ms	12.7 ms	16.0	0	38,209	30.0 s
Sustained decode	512K	32	Valid	1,863.8	58.3	19,316 ms	17.1 ms	31.9	2	55,905	30.0 s
Sustained decode	512K	64	Looped	—	—	—	—	—	0	0	—
Sustained decode	512K	128	Looped	—	—	—	—	—	0	0	—
Sustained decode	1019K	1	Valid	126.4	127.5	3,995 ms	7.8 ms	0.9	1	3,595	28.5 s
Sustained decode	1019K	2	Valid	234.0	121.8	5,232 ms	8.2 ms	1.9	2	7,000	29.9 s
Sustained decode	1019K	4	Valid	426.2	109.1	7,874 ms	9.2 ms	3.9	2	12,783	30.0 s
Sustained decode	1019K	8	Valid	694.0	88.9	11,743 ms	11.2 ms	7.8	2	20,817	30.0 s
Sustained decode	1019K	16	Valid	1,012.1	66.0	18,491 ms	15.1 ms	15.5	5	30,358	30.0 s
Sustained decode	1019K	32	Requests queued	1,359.9	45.1	8,602 ms	22.2 ms	30.7	42	40,773	30.0 s
Sustained decode	1019K	64	Requests queued	1,451.0	40.3	88,969 ms	24.8 ms	38.9	38	43,529	30.0 s
Sustained decode	1019K	128	Looped	—	—	—	—	—	0	0	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement.