Speed test matrix
Qwen3.5-35B-A3B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context38.8K211 ms TTFT
16K context37.8K429 ms TTFT
32K context39.7K814 ms TTFT
64K context36.5K1,763 ms TTFT
128K context28.3K4,553 ms TTFT
251K context19.6K12,899 ms TTFT
256K context19.3K13,347 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 204.598 ms TTFT | 360.5153 ms TTFT | 600.3932 ms TTFT | 922.7282 ms TTFT | 1,544.6360 ms TTFT | 2,309.3586 ms TTFT | 3,434.3936 ms TTFT | 4,919.31,411 ms TTFT |
| 16K | 201.499 ms TTFT | 358.1120 ms TTFT | 589.8216 ms TTFT | 899.4284 ms TTFT | 1,445.1364 ms TTFT | 2,177.6528 ms TTFT | 3,149.0838 ms TTFT | Over capacity |
| 32K | 197.0138 ms TTFT | 347.4216 ms TTFT | 565.7288 ms TTFT | 854.5360 ms TTFT | 1,347.7560 ms TTFT | 1,940.6795 ms TTFT | Over capacity | Over capacity |
| 64K | 188.0174 ms TTFT | 331.9198 ms TTFT | 527.6192 ms TTFT | 774.8553 ms TTFT | 1,168.9946 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 171.4322 ms TTFT | 303.6416 ms TTFT | 468.3630 ms TTFT | 655.41,001 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 147.0619 ms TTFT | 260.2665 ms TTFT | 382.21,172 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 38.8K | — | 211 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 37.8K | — | 429 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 39.7K | — | 814 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 36.5K | — | 1,763 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28.3K | — | 4,553 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.6K | — | 12,899 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.3K | — | 13,347 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 204.5 | 205.4 | 98 ms | 4.9 ms | 1.0 | 1 | 6,133 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 360.5 | 181.6 | 153 ms | 5.5 ms | 2.0 | 2 | 10,816 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 600.3 | 151.1 | 932 ms | 6.6 ms | 4.0 | 4 | 18,008 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 922.7 | 115.6 | 282 ms | 8.6 ms | 8.0 | 0 | 27,680 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,544.6 | 96.8 | 360 ms | 10.3 ms | 16.0 | 0 | 46,337 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,309.3 | 72.6 | 586 ms | 13.8 ms | 32.0 | 0 | 69,283 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,434.3 | 53.8 | 936 ms | 18.6 ms | 64.0 | 0 | 103,040 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,919.3 | 38.1 | 1,411 ms | 26.3 ms | 128.0 | 0 | 147,584 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 201.4 | 202.4 | 99 ms | 4.9 ms | 1.0 | 1 | 6,044 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 358.1 | 180.0 | 120 ms | 5.6 ms | 1.9 | 2 | 10,742 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 589.8 | 148.9 | 216 ms | 6.7 ms | 3.9 | 4 | 17,695 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 899.4 | 112.7 | 284 ms | 8.9 ms | 8.0 | 0 | 26,984 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,445.1 | 90.8 | 364 ms | 11.0 ms | 16.0 | 0 | 43,344 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,177.6 | 68.2 | 528 ms | 14.7 ms | 32.0 | 0 | 65,312 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,149.0 | 49.2 | 838 ms | 20.3 ms | 64.0 | 0 | 94,464 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 197.0 | 198.1 | 138 ms | 5.0 ms | 1.0 | 1 | 5,908 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 347.4 | 175.6 | 216 ms | 5.7 ms | 2.0 | 2 | 10,420 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 565.7 | 143.1 | 288 ms | 7.0 ms | 3.9 | 4 | 16,970 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 854.5 | 106.8 | 360 ms | 9.4 ms | 8.0 | 0 | 25,632 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,347.7 | 84.2 | 560 ms | 11.9 ms | 16.0 | 0 | 40,432 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,940.6 | 60.4 | 795 ms | 16.6 ms | 32.0 | 0 | 58,208 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 188.0 | 189.5 | 174 ms | 5.3 ms | 1.0 | 1 | 5,639 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 331.9 | 167.6 | 198 ms | 6.0 ms | 2.0 | 2 | 9,956 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 527.6 | 132.7 | 192 ms | 7.5 ms | 4.0 | 4 | 15,828 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 774.8 | 97.0 | 553 ms | 10.3 ms | 8.0 | 0 | 23,240 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,168.9 | 73.2 | 946 ms | 13.7 ms | 16.0 | 0 | 35,056 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 171.4 | 173.9 | 322 ms | 5.8 ms | 1.0 | 1 | 5,143 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 303.6 | 154.2 | 416 ms | 6.5 ms | 2.0 | 2 | 9,113 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 468.3 | 117.0 | 630 ms | 8.5 ms | 4.0 | 0 | 14,049 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 655.4 | 81.6 | 1,001 ms | 12.3 ms | 8.0 | 0 | 19,656 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 147.0 | 151.3 | 619 ms | 6.6 ms | 1.0 | 1 | 4,411 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 260.2 | 133.0 | 665 ms | 7.5 ms | 2.0 | 2 | 7,806 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 382.2 | 95.5 | 1,172 ms | 10.5 ms | 4.0 | 0 | 11,464 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.