Speed test matrix

Laguna-XS-2.1 on H200 NVL 141GB

vllm · 0.26.0 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 5, 2026, 23:12 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.0
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: poolside
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.876036 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1

Prefill

prompt tok/s by context length

8K context42.6K192 ms TTFT

16K context40K405 ms TTFT

32K context37.9K850 ms TTFT

64K context29.8K2,154 ms TTFT

128K context20.9K6,142 ms TTFT

251K context13.1K19,177 ms TTFT

256K context13K19,721 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	209.546 ms TTFT	356.362 ms TTFT	587.670 ms TTFT	903.6161 ms TTFT	1,424.0247 ms TTFT	2,135.8399 ms TTFT	3,178.6531 ms TTFT	Over capacity
16K	205.188 ms TTFT	348.2104 ms TTFT	572.1102 ms TTFT	865.4233 ms TTFT	1,353.9280 ms TTFT	1,939.2588 ms TTFT	2,754.9725 ms TTFT	Over capacity
32K	198.8104 ms TTFT	332.7156 ms TTFT	539.5204 ms TTFT	803.3326 ms TTFT	1,203.2499 ms TTFT	1,639.0798 ms TTFT	Over capacity	Over capacity
64K	186.1176 ms TTFT	307.6216 ms TTFT	484.6244 ms TTFT	687.9641 ms TTFT	981.7919 ms TTFT	Over capacity	Over capacity	Over capacity
128K	165.6294 ms TTFT	264.8399 ms TTFT	408.1611 ms TTFT	543.4994 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	136.0555 ms TTFT	213.3763 ms TTFT	308.71,203 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	42.6K	—	192 ms	—	—	1	—	—
Prefill	16K	1	Valid	40K	—	405 ms	—	—	1	—	—
Prefill	32K	1	Valid	37.9K	—	850 ms	—	—	1	—	—
Prefill	64K	1	Valid	29.8K	—	2,154 ms	—	—	1	—	—
Prefill	128K	1	Valid	20.9K	—	6,142 ms	—	—	1	—	—
Prefill	251K	1	Valid	13.1K	—	19,177 ms	—	—	1	—	—
Prefill	256K	1	Valid	13K	—	19,721 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	209.5	210.0	46 ms	4.8 ms	1.0	1	6,286	30.0 s
Sustained decode	8K	2	Valid	356.3	178.9	62 ms	5.6 ms	2.0	2	10,690	30.0 s
Sustained decode	8K	4	Valid	587.6	147.7	70 ms	6.8 ms	3.9	4	17,627	30.0 s
Sustained decode	8K	8	Valid	903.6	113.2	161 ms	8.8 ms	8.0	0	27,104	30.0 s
Sustained decode	8K	16	Valid	1,424.0	89.8	247 ms	11.1 ms	16.0	0	42,720	30.0 s
Sustained decode	8K	32	Valid	2,135.8	67.5	399 ms	14.8 ms	32.0	0	64,064	30.0 s
Sustained decode	8K	64	Valid	3,178.6	50.2	531 ms	19.9 ms	64.0	0	95,360	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	205.1	206.0	88 ms	4.9 ms	1.0	1	6,156	30.0 s
Sustained decode	16K	2	Valid	348.2	175.1	104 ms	5.7 ms	2.0	2	10,445	30.0 s
Sustained decode	16K	4	Valid	572.1	144.3	102 ms	6.9 ms	4.0	4	17,163	30.0 s
Sustained decode	16K	8	Valid	865.4	108.5	233 ms	9.2 ms	8.0	0	25,960	30.0 s
Sustained decode	16K	16	Valid	1,353.9	85.2	280 ms	11.7 ms	16.0	0	40,608	30.0 s
Sustained decode	16K	32	Valid	1,939.2	61.2	588 ms	16.3 ms	32.0	0	58,176	30.0 s
Sustained decode	16K	64	Valid	2,754.9	43.4	725 ms	23.0 ms	64.0	0	82,624	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	198.8	199.9	104 ms	5.0 ms	1.0	1	5,964	30.0 s
Sustained decode	32K	2	Valid	332.7	167.8	156 ms	6.0 ms	2.0	2	9,980	30.0 s
Sustained decode	32K	4	Valid	539.5	135.8	204 ms	7.4 ms	4.0	4	16,186	30.0 s
Sustained decode	32K	8	Valid	803.3	100.5	326 ms	9.9 ms	8.0	0	24,111	30.0 s
Sustained decode	32K	16	Valid	1,203.2	75.6	499 ms	13.2 ms	16.0	0	36,096	30.0 s
Sustained decode	32K	32	Valid	1,639.0	51.5	798 ms	19.4 ms	32.0	0	49,175	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	186.1	187.7	176 ms	5.3 ms	1.0	1	5,585	30.0 s
Sustained decode	64K	2	Valid	307.6	155.6	216 ms	6.4 ms	2.0	2	9,227	30.0 s
Sustained decode	64K	4	Valid	484.6	123.4	244 ms	8.1 ms	4.0	4	14,538	30.0 s
Sustained decode	64K	8	Valid	687.9	86.2	641 ms	11.6 ms	8.0	0	20,634	30.0 s
Sustained decode	64K	16	Valid	981.7	61.5	919 ms	16.3 ms	16.0	0	29,440	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	165.6	168.0	294 ms	6.0 ms	1.0	1	4,969	30.0 s
Sustained decode	128K	2	Valid	264.8	135.9	399 ms	7.4 ms	2.0	2	7,944	30.0 s
Sustained decode	128K	4	Valid	408.1	102.2	611 ms	9.8 ms	4.0	0	12,240	30.0 s
Sustained decode	128K	8	Valid	543.4	67.9	994 ms	14.7 ms	8.0	0	16,296	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	136.0	141.2	555 ms	7.1 ms	1.0	1	4,081	30.0 s
Sustained decode	251K	2	Valid	213.3	107.0	763 ms	9.3 ms	2.0	0	6,400	30.0 s
Sustained decode	251K	4	Valid	308.7	77.4	1,203 ms	12.9 ms	4.0	0	9,256	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.