Speed test matrix
Qwen3.5-122B-A10B-GPTQ-Int4 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.863309 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context22.3K367 ms TTFT
16K context21.9K742 ms TTFT
32K context22.6K1,431 ms TTFT
64K context21K3,073 ms TTFT
128K context17.4K7,381 ms TTFT
251K context12.9K19,499 ms TTFT
256K context12.8K20,161 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 153.5135 ms TTFT | 288.3183 ms TTFT | 511.6344 ms TTFT | 902.5457 ms TTFT | 1,474.8743 ms TTFT | 2,321.21,346 ms TTFT | 3,535.82,497 ms TTFT | 4,948.74,799 ms TTFT |
| 16K | 151.5171 ms TTFT | 285.5201 ms TTFT | 506.9330 ms TTFT | 886.1408 ms TTFT | 1,430.3911 ms TTFT | 2,242.31,244 ms TTFT | 3,337.52,340 ms TTFT | 4,552.44,303 ms TTFT |
| 32K | 148.5164 ms TTFT | 280.2169 ms TTFT | 495.9279 ms TTFT | 857.7455 ms TTFT | 1,363.0694 ms TTFT | 2,077.91,086 ms TTFT | 2,988.71,864 ms TTFT | 3,845.83,016 ms TTFT |
| 64K | 142.1243 ms TTFT | 267.9304 ms TTFT | 480.6516 ms TTFT | 804.1779 ms TTFT | 1,240.81,332 ms TTFT | 1,811.02,029 ms TTFT | 2,407.63,488 ms TTFT | Over capacity |
| 128K | 131.5354 ms TTFT | 248.1419 ms TTFT | 448.3569 ms TTFT | 720.11,097 ms TTFT | 1,056.31,829 ms TTFT | 1,411.53,215 ms TTFT | Over capacity | Over capacity |
| 251K | 116.2644 ms TTFT | 222.0879 ms TTFT | 394.91,194 ms TTFT | 593.81,939 ms TTFT | 804.53,368 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 22.3K | — | 367 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.9K | — | 742 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.6K | — | 1,431 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21K | — | 3,073 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.4K | — | 7,381 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.9K | — | 19,499 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.8K | — | 20,161 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 153.5 | 154.5 | 135 ms | 6.5 ms | 1.0 | 1 | 4,604 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 288.3 | 145.0 | 183 ms | 6.9 ms | 2.0 | 2 | 8,650 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 511.6 | 129.6 | 344 ms | 7.7 ms | 4.0 | 4 | 15,348 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 902.5 | 113.0 | 457 ms | 8.9 ms | 8.0 | 0 | 27,072 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,474.8 | 91.6 | 743 ms | 10.9 ms | 16.0 | 0 | 44,240 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,321.2 | 71.2 | 1,346 ms | 14.1 ms | 32.0 | 0 | 69,632 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,535.8 | 52.6 | 2,497 ms | 19.0 ms | 64.0 | 0 | 106,048 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,948.7 | 34.8 | 4,799 ms | 28.8 ms | 128.0 | 0 | 148,480 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 151.5 | 152.8 | 171 ms | 6.5 ms | 1.0 | 1 | 4,546 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 285.5 | 143.8 | 201 ms | 7.0 ms | 2.0 | 2 | 8,568 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 506.9 | 128.2 | 330 ms | 7.8 ms | 4.0 | 4 | 15,204 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 886.1 | 110.5 | 408 ms | 9.0 ms | 8.0 | 0 | 26,576 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,430.3 | 89.7 | 911 ms | 11.2 ms | 16.0 | 0 | 42,912 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,242.3 | 68.9 | 1,244 ms | 14.5 ms | 32.0 | 0 | 67,264 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,337.5 | 50.2 | 2,340 ms | 19.9 ms | 64.0 | 0 | 100,096 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,552.4 | 32.5 | 4,303 ms | 30.7 ms | 128.0 | 0 | 136,576 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 148.5 | 149.9 | 164 ms | 6.7 ms | 1.0 | 1 | 4,458 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 280.2 | 140.8 | 169 ms | 7.1 ms | 2.0 | 2 | 8,404 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 495.9 | 125.2 | 279 ms | 8.0 ms | 3.9 | 4 | 14,876 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 857.7 | 107.5 | 455 ms | 9.3 ms | 8.0 | 0 | 25,728 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,363.0 | 85.3 | 694 ms | 11.7 ms | 16.0 | 0 | 40,880 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,077.9 | 64.6 | 1,086 ms | 15.5 ms | 32.0 | 0 | 62,336 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,988.7 | 45.9 | 1,864 ms | 21.8 ms | 64.0 | 0 | 89,664 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,845.8 | 28.5 | 3,016 ms | 35.1 ms | 128.0 | 0 | 115,328 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 142.1 | 143.8 | 243 ms | 7.0 ms | 1.0 | 1 | 4,266 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 267.9 | 135.3 | 304 ms | 7.4 ms | 2.0 | 2 | 8,036 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 480.6 | 120.2 | 516 ms | 8.3 ms | 4.0 | 1 | 14,427 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 804.1 | 100.3 | 779 ms | 10.0 ms | 8.0 | 0 | 24,120 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,240.8 | 77.2 | 1,332 ms | 12.9 ms | 16.0 | 0 | 37,216 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,811.0 | 55.3 | 2,029 ms | 18.1 ms | 32.0 | 0 | 54,336 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,407.6 | 35.5 | 3,488 ms | 28.2 ms | 64.0 | 0 | 72,192 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 131.5 | 134.8 | 354 ms | 7.4 ms | 1.0 | 1 | 3,947 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 248.1 | 125.9 | 419 ms | 7.9 ms | 1.9 | 2 | 7,442 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 448.3 | 111.9 | 569 ms | 8.9 ms | 4.0 | 0 | 13,448 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 720.1 | 89.7 | 1,097 ms | 11.1 ms | 8.0 | 0 | 21,600 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,056.3 | 65.3 | 1,829 ms | 15.3 ms | 16.0 | 0 | 31,696 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,411.5 | 43.2 | 3,215 ms | 23.1 ms | 32.0 | 0 | 42,336 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 116.2 | 116.5 | 644 ms | 8.6 ms | 1.0 | 0 | 3,485 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 222.0 | 111.2 | 879 ms | 9.0 ms | 2.0 | 0 | 6,660 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 394.9 | 98.6 | 1,194 ms | 10.1 ms | 4.0 | 0 | 11,848 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 593.8 | 73.9 | 1,939 ms | 13.5 ms | 8.0 | 0 | 17,816 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 804.5 | 50.0 | 3,368 ms | 20.0 ms | 16.0 | 0 | 24,128 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.