Speed test matrix
Laguna-XS-2.1-NVFP4 on H200 NVL 141GB
vllm · 0.26.0 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1-NVFP4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.876095 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1
Prefill
prompt tok/s by context length8K context33.6K244 ms TTFT
16K context31.5K514 ms TTFT
32K context30.1K1,071 ms TTFT
64K context25.8K2,491 ms TTFT
128K context19.2K6,665 ms TTFT
251K context12.8K19,705 ms TTFT
256K context12.5K20,522 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 162.253 ms TTFT | 310.165 ms TTFT | 580.3103 ms TTFT | 1,007.886 ms TTFT | 1,625.1263 ms TTFT | 2,421.5402 ms TTFT | 3,375.3482 ms TTFT | 4,826.6792 ms TTFT |
| 16K | 160.087 ms TTFT | 305.389 ms TTFT | 559.6146 ms TTFT | 950.6158 ms TTFT | 1,498.4393 ms TTFT | 2,151.1490 ms TTFT | 2,835.8853 ms TTFT | 3,807.41,319 ms TTFT |
| 32K | 156.5112 ms TTFT | 294.0115 ms TTFT | 526.0194 ms TTFT | 861.0362 ms TTFT | 1,277.0497 ms TTFT | 1,734.0858 ms TTFT | 2,168.31,239 ms TTFT | 2,687.82,363 ms TTFT |
| 64K | 150.2172 ms TTFT | 274.1207 ms TTFT | 473.0363 ms TTFT | 722.2521 ms TTFT | 996.7918 ms TTFT | 1,260.81,557 ms TTFT | 1,467.22,340 ms TTFT | Over capacity |
| 128K | 139.4305 ms TTFT | 245.0366 ms TTFT | 389.3565 ms TTFT | 546.5955 ms TTFT | 694.41,620 ms TTFT | 807.22,984 ms TTFT | Over capacity | Over capacity |
| 251K | 123.9536 ms TTFT | 201.9791 ms TTFT | 291.61,078 ms TTFT | 371.61,770 ms TTFT | 436.63,065 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.6K | — | 244 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.5K | — | 514 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.1K | — | 1,071 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.8K | — | 2,491 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.2K | — | 6,665 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.8K | — | 19,705 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.5K | — | 20,522 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 162.2 | 162.6 | 53 ms | 6.2 ms | 1.0 | 1 | 4,867 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 310.1 | 155.7 | 65 ms | 6.4 ms | 2.0 | 2 | 9,303 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 580.3 | 146.4 | 103 ms | 6.8 ms | 4.0 | 4 | 17,405 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,007.8 | 128.3 | 86 ms | 7.8 ms | 8.0 | 8 | 30,249 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,625.1 | 102.3 | 263 ms | 9.8 ms | 16.0 | 0 | 48,752 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,421.5 | 76.4 | 402 ms | 13.1 ms | 32.0 | 0 | 72,640 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,375.3 | 53.2 | 482 ms | 18.8 ms | 64.0 | 0 | 101,248 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,826.6 | 37.9 | 792 ms | 26.4 ms | 128.0 | 0 | 144,768 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 160.0 | 160.7 | 87 ms | 6.2 ms | 1.0 | 1 | 4,799 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 305.3 | 153.4 | 89 ms | 6.5 ms | 2.0 | 2 | 9,160 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 559.6 | 141.3 | 146 ms | 7.1 ms | 4.0 | 4 | 16,786 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 950.6 | 119.6 | 158 ms | 8.4 ms | 8.0 | 8 | 28,515 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,498.4 | 94.2 | 393 ms | 10.6 ms | 16.0 | 0 | 44,944 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,151.1 | 67.6 | 490 ms | 14.8 ms | 32.0 | 0 | 64,512 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,835.8 | 44.5 | 853 ms | 22.5 ms | 64.0 | 0 | 85,056 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,807.4 | 29.8 | 1,319 ms | 33.6 ms | 128.0 | 0 | 114,176 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 156.5 | 157.6 | 112 ms | 6.3 ms | 1.0 | 1 | 4,695 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 294.0 | 148.0 | 115 ms | 6.8 ms | 2.0 | 2 | 8,819 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 526.0 | 133.0 | 194 ms | 7.5 ms | 4.0 | 4 | 15,777 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 861.0 | 108.0 | 362 ms | 9.3 ms | 8.0 | 0 | 25,824 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,277.0 | 80.1 | 497 ms | 12.5 ms | 16.0 | 0 | 38,304 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,734.0 | 54.3 | 858 ms | 18.4 ms | 32.0 | 0 | 52,000 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,168.3 | 34.0 | 1,239 ms | 29.4 ms | 64.0 | 0 | 65,024 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,687.8 | 21.1 | 2,363 ms | 47.4 ms | 128.0 | 0 | 80,640 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 150.2 | 151.7 | 172 ms | 6.6 ms | 1.0 | 1 | 4,504 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 274.1 | 138.6 | 207 ms | 7.2 ms | 2.0 | 2 | 8,220 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 473.0 | 118.4 | 363 ms | 8.4 ms | 4.0 | 0 | 14,188 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 722.2 | 90.5 | 521 ms | 11.0 ms | 8.0 | 0 | 21,664 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 996.7 | 62.5 | 918 ms | 16.0 ms | 16.0 | 0 | 29,904 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,260.8 | 39.5 | 1,557 ms | 25.3 ms | 32.0 | 0 | 37,824 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,467.2 | 23.2 | 2,340 ms | 43.1 ms | 64.0 | 0 | 43,968 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 139.4 | 142.1 | 305 ms | 7.0 ms | 1.0 | 1 | 4,185 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 245.0 | 122.7 | 366 ms | 8.1 ms | 2.0 | 0 | 7,348 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 389.3 | 97.5 | 565 ms | 10.3 ms | 4.0 | 0 | 11,684 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 546.5 | 68.3 | 955 ms | 14.6 ms | 8.0 | 0 | 16,400 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 694.4 | 43.5 | 1,620 ms | 23.0 ms | 16.0 | 0 | 20,832 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 807.2 | 25.3 | 2,984 ms | 39.5 ms | 32.0 | 0 | 24,192 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 123.9 | 124.3 | 536 ms | 8.0 ms | 1.0 | 1 | 3,704 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 201.9 | 101.3 | 791 ms | 9.9 ms | 2.0 | 0 | 6,056 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 291.6 | 73.1 | 1,078 ms | 13.7 ms | 4.0 | 0 | 8,749 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 371.6 | 46.5 | 1,770 ms | 21.5 ms | 8.0 | 0 | 11,152 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 436.6 | 27.3 | 3,065 ms | 36.6 ms | 16.0 | 0 | 13,104 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.