Speed test matrix
Qwen3.5-122B-A10B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context15.6K524 ms TTFT
16K context15.9K1,018 ms TTFT
32K context15.4K2,103 ms TTFT
64K context14.4K4,466 ms TTFT
128K context12.5K10,297 ms TTFT
251K context9,84825,609 ms TTFT
256K context9,85426,103 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 171.3151 ms TTFT | 314.9169 ms TTFT | 518.7169 ms TTFT | 857.8599 ms TTFT | 1,299.21,057 ms TTFT | 1,874.21,947 ms TTFT | 2,638.23,759 ms TTFT | 3,050.47,461 ms TTFT |
| 16K | 134.2144 ms TTFT | 253.0174 ms TTFT | 435.3434 ms TTFT | 734.9615 ms TTFT | 1,160.1987 ms TTFT | 1,582.41,910 ms TTFT | 2,215.23,302 ms TTFT | 2,532.36,199 ms TTFT |
| 32K | 91.5205 ms TTFT | 184.3321 ms TTFT | 324.2831 ms TTFT | 572.9699 ms TTFT | 944.11,379 ms TTFT | 1,212.22,432 ms TTFT | 1,677.84,053 ms TTFT | Over capacity |
| 64K | 55.3238 ms TTFT | 115.7312 ms TTFT | 213.1449 ms TTFT | 397.2729 ms TTFT | 690.01,176 ms TTFT | 819.32,067 ms TTFT | Over capacity | Over capacity |
| 128K | 33.2658 ms TTFT | 65.8692 ms TTFT | 127.7958 ms TTFT | 247.41,568 ms TTFT | 444.92,735 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 18.1914 ms TTFT | 36.51,206 ms TTFT | 72.11,711 ms TTFT | 138.72,787 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.6K | — | 524 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.9K | — | 1,018 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.4K | — | 2,103 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.4K | — | 4,466 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.5K | — | 10,297 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,848.0 | — | 25,609 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,854.0 | — | 26,103 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 171.3 | 175.0 | 151 ms | 5.7 ms | 1.0 | 1 | 5,137 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 314.9 | 157.6 | 169 ms | 6.4 ms | 2.0 | 2 | 9,444 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 518.7 | 126.7 | 169 ms | 7.9 ms | 4.0 | 4 | 15,560 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 857.8 | 104.9 | 599 ms | 9.5 ms | 8.0 | 0 | 25,716 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,299.2 | 78.4 | 1,057 ms | 12.8 ms | 16.0 | 0 | 38,942 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,874.2 | 57.3 | 1,947 ms | 17.5 ms | 32.0 | 0 | 56,167 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,638.2 | 38.1 | 3,759 ms | 26.3 ms | 64.0 | 0 | 79,043 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,050.4 | 20.8 | 7,461 ms | 48.1 ms | 128.0 | 0 | 91,431 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 134.2 | 139.4 | 144 ms | 7.2 ms | 1.0 | 1 | 4,025 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 253.0 | 128.3 | 174 ms | 7.8 ms | 2.0 | 2 | 7,588 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 435.3 | 110.1 | 434 ms | 9.1 ms | 4.0 | 0 | 13,059 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 734.9 | 92.5 | 615 ms | 10.8 ms | 8.0 | 0 | 22,047 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,160.1 | 72.6 | 987 ms | 13.8 ms | 16.0 | 0 | 34,798 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,582.4 | 48.9 | 1,910 ms | 20.5 ms | 32.0 | 0 | 47,404 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,215.2 | 33.0 | 3,302 ms | 30.3 ms | 64.0 | 0 | 66,432 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,532.3 | 17.9 | 6,199 ms | 55.7 ms | 128.0 | 0 | 75,684 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 91.5 | 91.9 | 205 ms | 10.9 ms | 1.0 | 0 | 2,744 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 184.3 | 92.4 | 321 ms | 10.8 ms | 2.0 | 0 | 5,527 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 324.2 | 80.7 | 831 ms | 12.4 ms | 4.0 | 0 | 9,721 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 572.9 | 71.8 | 699 ms | 13.9 ms | 8.0 | 0 | 17,186 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 944.1 | 59.0 | 1,379 ms | 16.9 ms | 16.0 | 0 | 28,317 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,212.2 | 36.9 | 2,432 ms | 27.1 ms | 32.0 | 0 | 36,328 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,677.8 | 24.7 | 4,053 ms | 40.5 ms | 64.0 | 0 | 50,335 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 55.3 | 56.0 | 238 ms | 17.9 ms | 1.0 | 0 | 1,658 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 115.7 | 58.2 | 312 ms | 17.2 ms | 2.0 | 0 | 3,470 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 213.1 | 53.7 | 449 ms | 18.6 ms | 4.0 | 0 | 6,387 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 397.2 | 50.0 | 729 ms | 20.0 ms | 8.0 | 0 | 11,906 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 690.0 | 43.1 | 1,176 ms | 23.2 ms | 16.0 | 0 | 20,663 | 29.9 s |
| Sustained decode | 64K | 32 | Valid | 819.3 | 25.2 | 2,067 ms | 39.6 ms | 32.0 | 0 | 24,575 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 33.2 | 33.6 | 658 ms | 29.8 ms | 1.0 | 0 | 994 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 65.8 | 33.3 | 692 ms | 30.0 ms | 2.0 | 0 | 1,973 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 127.7 | 32.2 | 958 ms | 31.1 ms | 4.0 | 0 | 3,828 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 247.4 | 30.9 | 1,568 ms | 32.3 ms | 8.0 | 0 | 7,409 | 29.9 s |
| Sustained decode | 128K | 16 | Valid | 444.9 | 27.2 | 2,735 ms | 36.7 ms | 16.0 | 0 | 13,325 | 29.9 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 18.1 | 18.6 | 914 ms | 53.9 ms | 1.0 | 0 | 544 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 36.5 | 18.6 | 1,206 ms | 53.9 ms | 2.0 | 0 | 1,091 | 29.9 s |
| Sustained decode | 251K | 4 | Valid | 72.1 | 18.2 | 1,711 ms | 54.8 ms | 4.0 | 0 | 2,161 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 138.7 | 17.3 | 2,787 ms | 57.9 ms | 8.0 | 0 | 4,145 | 29.9 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.