Speed test matrix
Qwen3.5-35B-A3B-Base on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.98 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context31.2K263 ms TTFT
16K context31.9K508 ms TTFT
32K context30K1,078 ms TTFT
64K context26.4K2,439 ms TTFT
128K context20.9K6,150 ms TTFT
251K context14.7K17,211 ms TTFT
256K context14.5K17,783 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 183.686 ms TTFT | 319.7101 ms TTFT | 547.5102 ms TTFT | 905.7326 ms TTFT | 1,434.5546 ms TTFT | 2,002.11,003 ms TTFT | 2,869.01,835 ms TTFT | 3,367.73,492 ms TTFT |
| 16K | 131.889 ms TTFT | 240.0110 ms TTFT | 446.5233 ms TTFT | 766.3273 ms TTFT | 1,252.6571 ms TTFT | 1,655.3814 ms TTFT | 2,365.61,506 ms TTFT | 2,724.22,792 ms TTFT |
| 32K | 93.3118 ms TTFT | 186.0160 ms TTFT | 330.9255 ms TTFT | 590.9403 ms TTFT | 998.1586 ms TTFT | 1,243.91,004 ms TTFT | 1,731.21,849 ms TTFT | Over capacity |
| 64K | 58.3183 ms TTFT | 115.5239 ms TTFT | 219.4391 ms TTFT | 390.7558 ms TTFT | 701.6916 ms TTFT | 818.91,619 ms TTFT | Over capacity | Over capacity |
| 128K | 30.6318 ms TTFT | 68.3437 ms TTFT | 123.6652 ms TTFT | 236.2993 ms TTFT | 442.91,723 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 17.3627 ms TTFT | 35.6876 ms TTFT | 70.91,383 ms TTFT | 132.72,238 ms TTFT | 260.93,751 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.2K | — | 263 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.9K | — | 508 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30K | — | 1,078 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.4K | — | 2,439 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.9K | — | 6,150 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.7K | — | 17,211 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.5K | — | 17,783 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 183.6 | 186.1 | 86 ms | 5.4 ms | 1.0 | 1 | 5,507 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 319.7 | 162.7 | 101 ms | 6.2 ms | 2.0 | 2 | 9,588 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 547.5 | 136.2 | 102 ms | 7.3 ms | 4.0 | 4 | 16,419 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 905.7 | 112.2 | 326 ms | 8.9 ms | 8.0 | 0 | 27,169 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,434.5 | 90.2 | 546 ms | 11.1 ms | 16.0 | 0 | 43,025 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,002.1 | 61.8 | 1,003 ms | 16.2 ms | 32.0 | 0 | 60,049 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,869.0 | 43.6 | 1,835 ms | 23.0 ms | 64.0 | 0 | 85,989 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,367.7 | 24.9 | 3,492 ms | 40.1 ms | 128.0 | 0 | 100,823 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 131.8 | 134.2 | 89 ms | 7.5 ms | 1.0 | 1 | 3,954 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 240.0 | 119.1 | 110 ms | 8.4 ms | 2.0 | 2 | 7,188 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 446.5 | 111.5 | 233 ms | 9.0 ms | 4.0 | 0 | 13,393 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 766.3 | 95.9 | 273 ms | 10.4 ms | 8.0 | 0 | 22,986 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,252.6 | 78.5 | 571 ms | 12.7 ms | 16.0 | 0 | 37,567 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,655.3 | 51.6 | 814 ms | 19.4 ms | 32.0 | 0 | 49,616 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,365.6 | 36.2 | 1,506 ms | 27.6 ms | 64.0 | 0 | 70,970 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,724.2 | 20.6 | 2,792 ms | 48.5 ms | 128.0 | 0 | 81,536 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 93.3 | 92.6 | 118 ms | 10.8 ms | 1.0 | 0 | 2,797 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 186.0 | 93.0 | 160 ms | 10.8 ms | 2.0 | 0 | 5,580 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 330.9 | 82.3 | 255 ms | 12.1 ms | 4.0 | 0 | 9,921 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 590.9 | 73.0 | 403 ms | 13.7 ms | 8.0 | 0 | 17,716 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 998.1 | 62.1 | 586 ms | 16.1 ms | 16.0 | 0 | 29,934 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,243.9 | 38.1 | 1,004 ms | 26.2 ms | 32.0 | 0 | 37,272 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,731.2 | 26.3 | 1,849 ms | 38.0 ms | 64.0 | 0 | 51,931 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 58.3 | 58.0 | 183 ms | 17.2 ms | 1.0 | 0 | 1,748 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 115.5 | 57.8 | 239 ms | 17.3 ms | 2.0 | 0 | 3,463 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 219.4 | 54.0 | 391 ms | 18.5 ms | 4.0 | 0 | 6,583 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 390.7 | 48.6 | 558 ms | 20.6 ms | 8.0 | 0 | 11,714 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 701.6 | 43.6 | 916 ms | 22.9 ms | 16.0 | 0 | 21,022 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 818.9 | 25.4 | 1,619 ms | 39.4 ms | 32.0 | 0 | 24,539 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 30.6 | 31.3 | 318 ms | 32.0 ms | 1.0 | 0 | 918 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 68.3 | 34.1 | 437 ms | 29.3 ms | 2.0 | 0 | 2,046 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 123.6 | 30.7 | 652 ms | 32.6 ms | 4.0 | 0 | 3,704 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 236.2 | 29.5 | 993 ms | 33.9 ms | 8.0 | 0 | 7,071 | 29.9 s |
| Sustained decode | 128K | 16 | Valid | 442.9 | 27.3 | 1,723 ms | 36.6 ms | 16.0 | 0 | 13,288 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 17.3 | 17.4 | 627 ms | 57.6 ms | 1.0 | 0 | 517 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 35.6 | 17.9 | 876 ms | 55.9 ms | 2.0 | 0 | 1,067 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 70.9 | 17.8 | 1,383 ms | 56.3 ms | 4.0 | 0 | 2,119 | 29.9 s |
| Sustained decode | 251K | 8 | Valid | 132.7 | 16.4 | 2,238 ms | 61.1 ms | 8.0 | 0 | 3,979 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 260.9 | 16.0 | 3,751 ms | 62.3 ms | 16.0 | 0 | 7,824 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.