Speed test matrix

gemma-4-E4B-it on H200 NVL 141GB

sglang · 0.5.6.post3.dev5791+gf7d4e44d8 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 19, 2026, 04:41 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev5791+gf7d4e44d8
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: f7d4e44d82ac

Run

Served model: gemma
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-gemma-sglang/bin/sglang" serve --model-path google/gemma-4-E4B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 131072 --mem-fraction-static 0.851725 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context27.5K299 ms TTFT

16K context22.6K719 ms TTFT

32K context16.4K1,972 ms TTFT

64K context10.4K6,231 ms TTFT

123K context6,11220,279 ms TTFT

128K context5,91321,811 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	131.987 ms TTFT	262.791 ms TTFT	518.9134 ms TTFT	1,014.0221 ms TTFT	1,954.3398 ms TTFT	3,295.9772 ms TTFT	5,060.81,453 ms TTFT	Over capacity
16K	113.6119 ms TTFT	228.1154 ms TTFT	455.8225 ms TTFT	896.2397 ms TTFT	1,734.2765 ms TTFT	2,713.71,460 ms TTFT	Over capacity	Over capacity
32K	88.9186 ms TTFT	182.4261 ms TTFT	366.3404 ms TTFT	734.6773 ms TTFT	1,429.31,512 ms TTFT	2,025.42,978 ms TTFT	Over capacity	Over capacity
64K	61.7331 ms TTFT	129.9478 ms TTFT	263.4813 ms TTFT	544.71,642 ms TTFT	1,047.33,154 ms TTFT	Over capacity	Over capacity	Over capacity
123K	39.4732 ms TTFT	83.91,157 ms TTFT	175.22,082 ms TTFT	366.44,116 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	27.5K	—	299 ms	—	—	1	—	—
Prefill	16K	1	Valid	22.6K	—	719 ms	—	—	1	—	—
Prefill	32K	1	Valid	16.4K	—	1,972 ms	—	—	1	—	—
Prefill	64K	1	Valid	10.4K	—	6,231 ms	—	—	1	—	—
Prefill	123K	1	Valid	6,112.0	—	20,279 ms	—	—	1	—	—
Prefill	128K	1	Valid	5,913.0	—	21,811 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	131.9	135.1	87 ms	7.4 ms	1.0	1	3,956	30.0 s
Sustained decode	8K	2	Valid	262.7	134.8	91 ms	7.4 ms	2.0	2	7,882	30.0 s
Sustained decode	8K	4	Valid	518.9	133.1	134 ms	7.5 ms	4.0	4	15,565	30.0 s
Sustained decode	8K	8	Valid	1,014.0	128.2	221 ms	7.8 ms	8.0	8	30,420	30.0 s
Sustained decode	8K	16	Valid	1,954.3	124.5	398 ms	8.0 ms	16.0	16	58,631	30.0 s
Sustained decode	8K	32	Valid	3,295.9	103.9	772 ms	9.6 ms	32.0	0	98,880	30.0 s
Sustained decode	8K	64	Valid	5,060.8	79.7	1,453 ms	12.5 ms	64.0	0	151,808	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	113.6	114.0	119 ms	8.8 ms	1.0	0	3,406	30.0 s
Sustained decode	16K	2	Valid	228.1	114.7	154 ms	8.7 ms	2.0	0	6,842	30.0 s
Sustained decode	16K	4	Valid	455.8	114.7	225 ms	8.7 ms	4.0	0	13,672	30.0 s
Sustained decode	16K	8	Valid	896.2	113.0	397 ms	8.9 ms	8.0	0	26,888	30.0 s
Sustained decode	16K	16	Valid	1,734.2	109.4	765 ms	9.1 ms	16.0	0	52,016	30.0 s
Sustained decode	16K	32	Valid	2,713.7	85.8	1,460 ms	11.7 ms	32.0	0	81,408	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	88.9	89.1	186 ms	11.2 ms	1.0	0	2,667	30.0 s
Sustained decode	32K	2	Valid	182.4	91.5	261 ms	10.9 ms	2.0	0	5,470	30.0 s
Sustained decode	32K	4	Valid	366.3	92.0	404 ms	10.9 ms	4.0	0	10,988	30.0 s
Sustained decode	32K	8	Valid	734.6	92.5	773 ms	10.8 ms	8.0	0	22,032	30.0 s
Sustained decode	32K	16	Valid	1,429.3	90.4	1,512 ms	11.1 ms	16.0	0	42,880	30.0 s
Sustained decode	32K	32	Valid	2,025.4	64.7	2,978 ms	15.5 ms	32.0	0	60,768	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	61.7	61.8	331 ms	16.2 ms	1.0	0	1,850	30.0 s
Sustained decode	64K	2	Valid	129.9	65.1	478 ms	15.4 ms	2.0	0	3,898	30.0 s
Sustained decode	64K	4	Valid	263.4	66.0	813 ms	15.1 ms	4.0	0	7,900	30.0 s
Sustained decode	64K	8	Valid	544.7	68.8	1,642 ms	14.5 ms	8.0	0	16,336	30.0 s
Sustained decode	64K	16	Valid	1,047.3	67.0	3,154 ms	14.9 ms	16.0	0	31,408	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	39.4	39.4	732 ms	25.4 ms	1.0	0	1,180	30.0 s
Sustained decode	123K	2	Valid	83.9	42.1	1,157 ms	23.7 ms	2.0	0	2,516	30.0 s
Sustained decode	123K	4	Valid	175.2	44.2	2,082 ms	22.6 ms	4.0	0	5,256	30.0 s
Sustained decode	123K	8	Valid	366.4	47.3	4,116 ms	21.2 ms	8.0	0	10,993	30.0 s
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.