Speed test matrix
Qwen3.5-4B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.872563 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-4B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context46.7K175 ms TTFT
16K context45.1K359 ms TTFT
32K context46K702 ms TTFT
64K context41.3K1,560 ms TTFT
128K context33.1K3,889 ms TTFT
251K context23.5K10,738 ms TTFT
256K context23.1K11,116 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 408.278 ms TTFT | 779.989 ms TTFT | 1,496.193 ms TTFT | 2,612.0103 ms TTFT | 3,975.7126 ms TTFT | 5,311.0215 ms TTFT | 6,018.41,027 ms TTFT | 6,429.21,800 ms TTFT |
| 16K | 375.6102 ms TTFT | 753.3119 ms TTFT | 1,419.8126 ms TTFT | 2,451.6122 ms TTFT | 3,765.5197 ms TTFT | 4,720.5225 ms TTFT | 5,342.71,085 ms TTFT | 5,690.72,592 ms TTFT |
| 32K | 368.5126 ms TTFT | 721.1148 ms TTFT | 1,331.4144 ms TTFT | 2,238.5163 ms TTFT | 3,280.0227 ms TTFT | 3,866.9632 ms TTFT | 4,299.41,455 ms TTFT | 4,483.92,891 ms TTFT |
| 64K | 360.8201 ms TTFT | 673.4223 ms TTFT | 1,176.2231 ms TTFT | 1,847.8254 ms TTFT | 2,462.9309 ms TTFT | 2,862.31,430 ms TTFT | 3,063.42,645 ms TTFT | 3,144.85,257 ms TTFT |
| 128K | 328.1339 ms TTFT | 536.5397 ms TTFT | 954.1421 ms TTFT | 1,344.4589 ms TTFT | 1,737.71,866 ms TTFT | 1,846.03,509 ms TTFT | 1,911.36,573 ms TTFT | 1,954.612,678 ms TTFT |
| 251K | 282.0656 ms TTFT | 470.75,791 ms TTFT | 676.6731 ms TTFT | 862.52,001 ms TTFT | 1,010.43,607 ms TTFT | 1,072.76,661 ms TTFT | 1,103.412,512 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 46.7K | — | 175 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 45.1K | — | 359 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 46K | — | 702 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.3K | — | 1,560 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 33.1K | — | 3,889 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 23.5K | — | 10,738 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 23.1K | — | 11,116 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 408.2 | 405.9 | 78 ms | 2.5 ms | 1.0 | 3 | 12,247 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 779.9 | 393.3 | 89 ms | 2.5 ms | 2.0 | 6 | 23,394 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,496.1 | 385.6 | 93 ms | 2.6 ms | 4.0 | 12 | 44,881 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,612.0 | 329.9 | 103 ms | 3.0 ms | 8.0 | 16 | 78,358 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,975.7 | 257.3 | 126 ms | 3.9 ms | 16.0 | 30 | 119,262 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,311.0 | 170.8 | 215 ms | 5.9 ms | 31.9 | 32 | 159,270 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,018.4 | 95.9 | 1,027 ms | 10.4 ms | 64.0 | 0 | 180,506 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,429.2 | 51.7 | 1,800 ms | 19.3 ms | 128.0 | 0 | 192,968 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 375.6 | 381.4 | 102 ms | 2.6 ms | 1.0 | 3 | 11,267 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 753.3 | 391.5 | 119 ms | 2.6 ms | 2.0 | 6 | 22,596 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,419.8 | 360.3 | 126 ms | 2.8 ms | 4.0 | 10 | 42,592 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,451.6 | 312.0 | 122 ms | 3.2 ms | 8.0 | 16 | 73,536 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,765.5 | 239.6 | 197 ms | 4.2 ms | 16.0 | 22 | 112,896 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,720.5 | 152.1 | 225 ms | 6.6 ms | 31.9 | 32 | 141,618 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,342.7 | 84.6 | 1,085 ms | 11.8 ms | 64.0 | 0 | 160,215 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,690.7 | 45.1 | 2,592 ms | 22.2 ms | 128.0 | 0 | 170,768 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 368.5 | 391.1 | 126 ms | 2.6 ms | 1.0 | 3 | 11,054 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 721.1 | 367.5 | 148 ms | 2.7 ms | 2.0 | 5 | 21,633 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,331.4 | 341.1 | 144 ms | 2.9 ms | 4.0 | 8 | 39,932 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,238.5 | 285.9 | 163 ms | 3.5 ms | 7.9 | 16 | 67,150 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,280.0 | 207.8 | 227 ms | 4.8 ms | 15.9 | 16 | 98,367 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,866.9 | 123.1 | 632 ms | 8.1 ms | 31.9 | 17 | 116,011 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,299.4 | 68.2 | 1,455 ms | 14.7 ms | 64.0 | 0 | 128,854 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,483.9 | 36.0 | 2,891 ms | 27.8 ms | 128.0 | 0 | 134,550 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 360.8 | 372.0 | 201 ms | 2.7 ms | 1.0 | 2 | 10,823 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 673.4 | 341.2 | 223 ms | 2.9 ms | 2.0 | 4 | 20,202 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,176.2 | 309.7 | 231 ms | 3.2 ms | 3.9 | 8 | 35,277 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,847.8 | 240.6 | 254 ms | 4.2 ms | 7.9 | 10 | 55,436 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,462.9 | 157.8 | 309 ms | 6.3 ms | 15.9 | 16 | 73,863 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,862.3 | 91.3 | 1,430 ms | 11.0 ms | 32.0 | 0 | 85,823 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,063.4 | 49.1 | 2,645 ms | 20.4 ms | 64.0 | 0 | 91,889 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,144.8 | 26.3 | 5,257 ms | 38.0 ms | 128.0 | 0 | 94,288 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 328.1 | 339.0 | 339 ms | 3.0 ms | 1.0 | 2 | 9,843 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 536.5 | 285.2 | 397 ms | 3.5 ms | 2.0 | 4 | 16,092 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 954.1 | 250.4 | 421 ms | 4.0 ms | 4.0 | 7 | 28,546 | 29.9 s |
| Sustained decode | 128K | 8 | Valid | 1,344.4 | 171.5 | 589 ms | 5.8 ms | 7.8 | 8 | 40,331 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,737.7 | 104.0 | 1,866 ms | 9.6 ms | 16.0 | 1 | 52,101 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,846.0 | 58.8 | 3,509 ms | 17.0 ms | 32.0 | 0 | 55,368 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,911.3 | 31.3 | 6,573 ms | 32.0 ms | 64.0 | 0 | 57,324 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 1,954.6 | 16.9 | 12,678 ms | 59.3 ms | 128.0 | 0 | 58,570 | 30.0 s |
| Sustained decode | 251K | 1 | Valid | 282.0 | 301.0 | 656 ms | 3.3 ms | 1.0 | 2 | 8,461 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 470.7 | 244.9 | 5,791 ms | 4.1 ms | 1.9 | 3 | 14,069 | 29.9 s |
| Sustained decode | 251K | 4 | Valid | 676.6 | 170.0 | 731 ms | 5.9 ms | 3.9 | 4 | 20,291 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 862.5 | 113.0 | 2,001 ms | 8.8 ms | 8.0 | 1 | 25,864 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,010.4 | 64.6 | 3,607 ms | 15.5 ms | 16.0 | 0 | 30,314 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,072.7 | 35.0 | 6,661 ms | 28.5 ms | 32.0 | 0 | 32,135 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 1,103.4 | 18.9 | 12,512 ms | 52.8 ms | 64.0 | 0 | 33,000 | 29.9 s |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.