Speed test matrix
Qwen3.5-35B-A3B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton
Prefill
prompt tok/s by context length8K context31.4K261 ms TTFT
16K context31.3K518 ms TTFT
32K context31.1K1,039 ms TTFT
64K context28.8K2,239 ms TTFT
128K context23.1K5,563 ms TTFT
251K context17.1K14,751 ms TTFT
256K context16.8K15,315 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 229.970 ms TTFT | 329.193 ms TTFT | 558.7148 ms TTFT | 880.6248 ms TTFT | 1,366.7503 ms TTFT | 2,057.4542 ms TTFT | 3,183.31,016 ms TTFT | 4,517.01,770 ms TTFT |
| 16K | 226.473 ms TTFT | 326.7102 ms TTFT | 557.3135 ms TTFT | 870.9198 ms TTFT | 1,321.0309 ms TTFT | 1,969.5498 ms TTFT | 2,990.9853 ms TTFT | 4,173.41,411 ms TTFT |
| 32K | 220.6112 ms TTFT | 315.0152 ms TTFT | 540.7201 ms TTFT | 842.8319 ms TTFT | 1,268.3483 ms TTFT | 1,842.9828 ms TTFT | 2,702.61,387 ms TTFT | 3,612.82,618 ms TTFT |
| 64K | 210.8117 ms TTFT | 299.1158 ms TTFT | 512.7132 ms TTFT | 791.0384 ms TTFT | 1,150.3681 ms TTFT | 1,624.01,272 ms TTFT | 2,255.52,513 ms TTFT | Over capacity |
| 128K | 191.7325 ms TTFT | 269.4453 ms TTFT | 463.6645 ms TTFT | 702.9996 ms TTFT | 982.21,762 ms TTFT | 1,301.93,252 ms TTFT | Over capacity | Over capacity |
| 251K | 163.2655 ms TTFT | 233.1869 ms TTFT | 377.31,206 ms TTFT | 582.22,019 ms TTFT | 761.03,500 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.4K | — | 261 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.3K | — | 518 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.1K | — | 1,039 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.8K | — | 2,239 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.1K | — | 5,563 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.1K | — | 14,751 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.8K | — | 15,315 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 229.9 | 230.5 | 70 ms | 4.3 ms | 1.0 | 1 | 6,896 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 329.1 | 165.2 | 93 ms | 6.1 ms | 2.0 | 2 | 9,872 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 558.7 | 141.4 | 148 ms | 7.1 ms | 4.0 | 4 | 16,760 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 880.6 | 110.8 | 248 ms | 9.0 ms | 8.0 | 0 | 26,416 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,366.7 | 86.4 | 503 ms | 11.6 ms | 16.0 | 0 | 40,992 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,057.4 | 64.7 | 542 ms | 15.4 ms | 32.0 | 0 | 61,696 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,183.3 | 49.9 | 1,016 ms | 20.0 ms | 64.0 | 0 | 95,488 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,517.0 | 34.9 | 1,770 ms | 28.7 ms | 128.0 | 0 | 135,424 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 226.4 | 227.0 | 73 ms | 4.4 ms | 1.0 | 1 | 6,792 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 326.7 | 164.3 | 102 ms | 6.1 ms | 2.0 | 2 | 9,802 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 557.3 | 140.3 | 135 ms | 7.1 ms | 4.0 | 4 | 16,715 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 870.9 | 109.3 | 198 ms | 9.1 ms | 8.0 | 0 | 26,120 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,321.0 | 83.2 | 309 ms | 12.0 ms | 16.0 | 0 | 39,616 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,969.5 | 62.1 | 498 ms | 16.1 ms | 32.0 | 0 | 59,072 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,990.9 | 47.0 | 853 ms | 21.3 ms | 64.0 | 0 | 89,729 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,173.4 | 32.3 | 1,411 ms | 30.9 ms | 128.0 | 0 | 125,184 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 220.6 | 221.4 | 112 ms | 4.5 ms | 1.0 | 1 | 6,617 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 315.0 | 158.3 | 152 ms | 6.3 ms | 2.0 | 2 | 9,448 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 540.7 | 136.1 | 201 ms | 7.3 ms | 4.0 | 4 | 16,219 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 842.8 | 105.4 | 319 ms | 9.5 ms | 8.0 | 0 | 25,280 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,268.3 | 79.3 | 483 ms | 12.6 ms | 16.0 | 0 | 38,048 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,842.9 | 57.4 | 828 ms | 17.4 ms | 32.0 | 0 | 55,264 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,702.6 | 41.6 | 1,387 ms | 24.0 ms | 64.0 | 0 | 81,080 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,612.8 | 27.1 | 2,618 ms | 36.9 ms | 128.0 | 0 | 108,288 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 210.8 | 211.7 | 117 ms | 4.7 ms | 1.0 | 1 | 6,324 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 299.1 | 150.5 | 158 ms | 6.6 ms | 2.0 | 2 | 8,972 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 512.7 | 130.0 | 132 ms | 7.7 ms | 4.0 | 4 | 15,380 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 791.0 | 99.1 | 384 ms | 10.1 ms | 8.0 | 0 | 23,728 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,150.3 | 72.0 | 681 ms | 13.9 ms | 16.0 | 0 | 34,496 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,624.0 | 50.9 | 1,272 ms | 19.6 ms | 32.0 | 0 | 48,704 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,255.5 | 35.6 | 2,513 ms | 28.1 ms | 64.0 | 0 | 67,648 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 191.7 | 194.0 | 325 ms | 5.2 ms | 1.0 | 1 | 5,749 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 269.4 | 136.5 | 453 ms | 7.3 ms | 2.0 | 2 | 8,081 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 463.6 | 115.7 | 645 ms | 8.6 ms | 4.0 | 0 | 13,904 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 702.9 | 87.2 | 996 ms | 11.5 ms | 8.0 | 0 | 21,080 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 982.2 | 59.7 | 1,762 ms | 16.8 ms | 16.0 | 0 | 29,456 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,301.9 | 38.6 | 3,252 ms | 25.9 ms | 32.0 | 0 | 39,040 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 163.2 | 167.4 | 655 ms | 6.0 ms | 1.0 | 1 | 4,896 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 233.1 | 116.7 | 869 ms | 8.6 ms | 2.0 | 0 | 6,994 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 377.3 | 93.9 | 1,206 ms | 10.6 ms | 4.0 | 0 | 11,316 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 582.2 | 71.8 | 2,019 ms | 13.9 ms | 8.0 | 0 | 17,464 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 761.0 | 45.7 | 3,500 ms | 21.9 ms | 16.0 | 0 | 22,816 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.