Speed test matrix
Laguna-S-2.1-NVFP4 on H200 NVL 141GB ×2
vllm · 0.27.1 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.1
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-vllm/bin/vllm" serve poolside/Laguna-S-2.1-NVFP4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 2 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 1048576 --gpu-memory-utilization 0.860202 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1
Prefill
prompt tok/s by context length8K context23.5K349 ms TTFT
16K context23.7K683 ms TTFT
32K context23.1K1,395 ms TTFT
64K context21.8K2,945 ms TTFT
128K context18.3K6,999 ms TTFT
256K context13.8K18,624 ms TTFT
512K context9,17955,843 ms TTFT
1019K context5,556183,589 ms TTFT
1024K context5,514185,880 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 152.652 ms TTFT | 276.257 ms TTFT | 471.293 ms TTFT | 758.0149 ms TTFT | 1,181.1200 ms TTFT | 1,793.0262 ms TTFT | 2,711.9443 ms TTFT | 3,800.8777 ms TTFT |
| 16K | 151.083 ms TTFT | 273.3106 ms TTFT | 466.5136 ms TTFT | 741.5162 ms TTFT | 1,140.3266 ms TTFT | 1,725.5448 ms TTFT | 2,518.7690 ms TTFT | 3,424.61,221 ms TTFT |
| 32K | 148.5109 ms TTFT | 266.9111 ms TTFT | 454.4184 ms TTFT | 708.5226 ms TTFT | 1,078.1391 ms TTFT | 1,575.9654 ms TTFT | 2,207.51,164 ms TTFT | Over capacity |
| 64K | 144.5159 ms TTFT | 255.6201 ms TTFT | 429.0306 ms TTFT | 654.6481 ms TTFT | 955.0872 ms TTFT | 1,333.21,265 ms TTFT | Over capacity | Over capacity |
| 128K | 136.1285 ms TTFT | 236.5365 ms TTFT | 387.0529 ms TTFT | 568.9883 ms TTFT | 790.91,473 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 124.1556 ms TTFT | 205.5745 ms TTFT | 325.81,133 ms TTFT | 449.91,808 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 104.11,147 ms TTFT | 163.31,660 ms TTFT | 247.72,500 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 75.0Requests queued2,405 ms TTFT | 107.2Requests queued3,480 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 23.5K | — | 349 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.7K | — | 683 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 23.1K | — | 1,395 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.8K | — | 2,945 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.3K | — | 6,999 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.8K | — | 18,624 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,179.0 | — | 55,843 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 5,556.0 | — | 183,589 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 5,514.0 | — | 185,880 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 152.6 | 153.2 | 52 ms | 6.5 ms | 1.0 | 1 | 4,578 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 276.2 | 138.8 | 57 ms | 7.2 ms | 2.0 | 2 | 8,286 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 471.2 | 118.0 | 93 ms | 8.5 ms | 4.0 | 0 | 14,132 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 758.0 | 95.2 | 149 ms | 10.5 ms | 8.0 | 0 | 22,736 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,181.1 | 74.2 | 200 ms | 13.5 ms | 16.0 | 0 | 35,423 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,793.0 | 56.5 | 262 ms | 17.7 ms | 32.0 | 0 | 53,788 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,711.9 | 42.7 | 443 ms | 23.4 ms | 64.0 | 0 | 81,347 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,800.8 | 29.9 | 777 ms | 33.5 ms | 128.0 | 0 | 114,048 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 151.0 | 151.8 | 83 ms | 6.6 ms | 1.0 | 1 | 4,531 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 273.3 | 137.8 | 106 ms | 7.3 ms | 2.0 | 2 | 8,197 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 466.5 | 116.9 | 136 ms | 8.6 ms | 4.0 | 0 | 13,992 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 741.5 | 93.0 | 162 ms | 10.7 ms | 8.0 | 0 | 22,240 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,140.3 | 71.7 | 266 ms | 13.9 ms | 16.0 | 0 | 34,208 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,725.5 | 54.3 | 448 ms | 18.4 ms | 32.0 | 0 | 51,744 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,518.7 | 39.7 | 690 ms | 25.2 ms | 64.0 | 0 | 75,520 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,424.6 | 26.9 | 1,221 ms | 37.1 ms | 128.0 | 0 | 102,786 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 148.5 | 149.5 | 109 ms | 6.7 ms | 1.0 | 1 | 4,456 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 266.9 | 134.9 | 111 ms | 7.4 ms | 2.0 | 2 | 8,011 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 454.4 | 113.8 | 184 ms | 8.8 ms | 4.0 | 0 | 13,632 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 708.5 | 88.9 | 226 ms | 11.2 ms | 8.0 | 0 | 21,248 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,078.1 | 67.7 | 391 ms | 14.8 ms | 16.0 | 0 | 32,334 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,575.9 | 49.6 | 654 ms | 20.2 ms | 32.0 | 0 | 47,264 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,207.5 | 34.7 | 1,164 ms | 28.8 ms | 64.0 | 0 | 66,240 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 144.5 | 145.6 | 159 ms | 6.9 ms | 1.0 | 1 | 4,333 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 255.6 | 130.7 | 201 ms | 7.7 ms | 1.9 | 2 | 7,666 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 429.0 | 107.5 | 306 ms | 9.3 ms | 4.0 | 0 | 12,877 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 654.6 | 82.2 | 481 ms | 12.2 ms | 8.0 | 0 | 19,632 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 955.0 | 60.0 | 872 ms | 16.7 ms | 16.0 | 0 | 28,639 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,333.2 | 41.9 | 1,265 ms | 23.9 ms | 32.0 | 0 | 39,999 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 136.1 | 138.9 | 285 ms | 7.2 ms | 1.0 | 1 | 4,084 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 236.5 | 118.6 | 365 ms | 8.4 ms | 2.0 | 0 | 7,094 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 387.0 | 97.1 | 529 ms | 10.3 ms | 4.0 | 0 | 11,608 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 568.9 | 71.4 | 883 ms | 14.0 ms | 8.0 | 0 | 17,064 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 790.9 | 49.7 | 1,473 ms | 20.1 ms | 16.0 | 0 | 23,729 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 124.1 | 124.5 | 556 ms | 8.0 ms | 1.0 | 1 | 3,702 | 29.8 s |
| Sustained decode | 256K | 2 | Valid | 205.5 | 103.1 | 745 ms | 9.7 ms | 2.0 | 0 | 6,164 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 325.8 | 81.8 | 1,133 ms | 12.2 ms | 4.0 | 0 | 9,776 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 449.9 | 56.5 | 1,808 ms | 17.7 ms | 8.0 | 0 | 13,504 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 104.1 | 104.7 | 1,147 ms | 9.6 ms | 1.0 | 0 | 3,123 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 163.3 | 82.1 | 1,660 ms | 12.2 ms | 2.0 | 0 | 4,898 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 247.7 | 62.4 | 2,500 ms | 16.0 ms | 4.0 | 0 | 7,428 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 75.0 | 80.0 | 2,405 ms | 12.5 ms | 0.9 | 0 | 2,250 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 107.2 | 59.5 | 3,480 ms | 16.8 ms | 1.8 | 0 | 3,216 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.