Speed test matrix
Qwen3.5-2B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-2B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883240 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context88.1K93 ms TTFT
16K context86.7K187 ms TTFT
32K context91.7K352 ms TTFT
64K context90K716 ms TTFT
128K context75.1K1,714 ms TTFT
251K context55.1K4,578 ms TTFT
256K context53.6K4,795 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 399.467 ms TTFT | 788.197 ms TTFT | 1,517.4133 ms TTFT | 2,861.7166 ms TTFT | 5,110.9195 ms TTFT | 8,254.3258 ms TTFT | 12.4K593 ms TTFT | 15.5K470 ms TTFT |
| 16K | 393.689 ms TTFT | 777.467 ms TTFT | 1,469.5147 ms TTFT | 2,716.5200 ms TTFT | 4,638.1252 ms TTFT | 7,197.3344 ms TTFT | 9,912.8377 ms TTFT | 12.1K1,371 ms TTFT |
| 32K | 381.8113 ms TTFT | 748.5177 ms TTFT | 1,392.1213 ms TTFT | 2,460.6221 ms TTFT | 3,959.3197 ms TTFT | 5,674.2263 ms TTFT | 7,154.31,169 ms TTFT | 8,082.22,168 ms TTFT |
| 64K | 362.3151 ms TTFT | 706.5164 ms TTFT | 1,254.1196 ms TTFT | 2,084.9197 ms TTFT | 3,071.3255 ms TTFT | 3,863.3541 ms TTFT | 4,525.32,303 ms TTFT | 4,795.84,731 ms TTFT |
| 128K | 326.2252 ms TTFT | 628.1304 ms TTFT | 1,038.2346 ms TTFT | 1,570.4350 ms TTFT | 1,973.4372 ms TTFT | 2,370.52,634 ms TTFT | 2,591.75,057 ms TTFT | Over capacity |
| 251K | 272.2526 ms TTFT | 510.7634 ms TTFT | 772.6580 ms TTFT | 1,019.8659 ms TTFT | 1,227.23,238 ms TTFT | 1,335.86,025 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 88.1K | — | 93 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 86.7K | — | 187 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 91.7K | — | 352 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 90K | — | 716 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 75.1K | — | 1,714 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 55.1K | — | 4,578 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 53.6K | — | 4,795 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 399.4 | 402.3 | 67 ms | 2.5 ms | 1.0 | 3 | 11,980 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 788.1 | 398.4 | 97 ms | 2.5 ms | 1.9 | 6 | 23,642 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,517.4 | 384.9 | 133 ms | 2.6 ms | 4.0 | 12 | 45,524 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,861.7 | 364.2 | 166 ms | 2.7 ms | 7.7 | 24 | 85,850 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,110.9 | 324.0 | 195 ms | 3.1 ms | 16.0 | 32 | 153,318 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 8,254.3 | 263.1 | 258 ms | 3.8 ms | 31.7 | 64 | 247,623 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 12.4K | 198.1 | 593 ms | 5.0 ms | 62.1 | 64 | 371,918 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15.5K | 123.8 | 470 ms | 8.1 ms | 126.3 | 128 | 463,575 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 393.6 | 397.3 | 89 ms | 2.5 ms | 1.0 | 3 | 11,808 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 777.4 | 391.5 | 67 ms | 2.6 ms | 2.0 | 6 | 23,320 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,469.5 | 373.2 | 147 ms | 2.7 ms | 4.0 | 12 | 44,085 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,716.5 | 344.5 | 200 ms | 2.9 ms | 8.0 | 16 | 81,497 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 4,638.1 | 294.9 | 252 ms | 3.4 ms | 16.0 | 32 | 139,147 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,197.3 | 227.9 | 344 ms | 4.4 ms | 31.9 | 32 | 215,912 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 9,912.8 | 155.6 | 377 ms | 6.4 ms | 63.6 | 64 | 297,382 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12.1K | 93.4 | 1,371 ms | 10.7 ms | 128.0 | 0 | 362,368 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 381.8 | 386.5 | 113 ms | 2.6 ms | 1.0 | 3 | 11,453 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 748.5 | 381.7 | 177 ms | 2.6 ms | 1.9 | 6 | 22,454 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,392.1 | 353.6 | 213 ms | 2.8 ms | 4.0 | 8 | 41,763 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,460.6 | 312.4 | 221 ms | 3.2 ms | 7.7 | 16 | 73,819 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,959.3 | 251.0 | 197 ms | 4.0 ms | 15.6 | 32 | 118,774 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 5,674.2 | 178.1 | 263 ms | 5.6 ms | 31.8 | 32 | 170,226 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,154.3 | 111.7 | 1,169 ms | 9.0 ms | 64.0 | 0 | 214,592 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,082.2 | 63.5 | 2,168 ms | 15.7 ms | 128.0 | 0 | 242,432 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 362.3 | 366.7 | 151 ms | 2.7 ms | 1.0 | 2 | 10,870 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 706.5 | 357.4 | 164 ms | 2.8 ms | 2.0 | 4 | 21,195 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,254.1 | 317.6 | 196 ms | 3.1 ms | 4.0 | 8 | 37,623 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,084.9 | 263.8 | 197 ms | 3.8 ms | 8.0 | 16 | 62,545 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,071.3 | 193.4 | 255 ms | 5.2 ms | 15.9 | 16 | 92,141 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,863.3 | 121.7 | 541 ms | 8.2 ms | 31.8 | 25 | 115,819 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,525.3 | 71.0 | 2,303 ms | 14.1 ms | 64.0 | 0 | 135,744 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,795.8 | 38.0 | 4,731 ms | 26.3 ms | 128.0 | 0 | 143,872 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 326.2 | 332.5 | 252 ms | 3.0 ms | 1.0 | 2 | 9,792 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 628.1 | 321.3 | 304 ms | 3.1 ms | 2.0 | 4 | 18,850 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,038.2 | 265.5 | 346 ms | 3.8 ms | 3.9 | 8 | 31,146 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,570.4 | 198.8 | 350 ms | 5.0 ms | 7.9 | 8 | 47,113 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,973.4 | 125.3 | 372 ms | 8.0 ms | 15.9 | 16 | 59,201 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,370.5 | 74.1 | 2,634 ms | 13.5 ms | 32.0 | 0 | 71,104 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,591.7 | 40.6 | 5,057 ms | 24.6 ms | 64.0 | 0 | 77,760 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 272.2 | 282.2 | 526 ms | 3.5 ms | 1.0 | 2 | 8,165 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 510.7 | 266.9 | 634 ms | 3.7 ms | 2.0 | 4 | 15,321 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 772.6 | 197.7 | 580 ms | 5.1 ms | 3.9 | 4 | 23,179 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,019.8 | 131.3 | 659 ms | 7.6 ms | 7.8 | 8 | 30,594 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,227.2 | 77.4 | 3,238 ms | 12.9 ms | 16.0 | 0 | 36,816 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,335.8 | 41.9 | 6,025 ms | 23.9 ms | 32.0 | 0 | 40,064 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.