Speed test matrix
Qwen3.5-27B-GPTQ-Int4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend marlin
Prefill
prompt tok/s by context length8K context6,2411,312 ms TTFT
16K context6,0792,669 ms TTFT
32K context5,7225,644 ms TTFT
64K context5,14412,524 ms TTFT
128K context4,24530,321 ms TTFT
251K context3,17079,563 ms TTFT
256K context3,14981,674 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 48.690 ms TTFT | 88.3138 ms TTFT | 170.3273 ms TTFT | 339.9456 ms TTFT | 606.6981 ms TTFT | 980.51,654 ms TTFT | 1,447.23,444 ms TTFT | Over capacity |
| 16K | 47.6146 ms TTFT | 86.1241 ms TTFT | 166.0437 ms TTFT | 326.5730 ms TTFT | 564.91,605 ms TTFT | 878.22,620 ms TTFT | Over capacity | Over capacity |
| 32K | 46.3115 ms TTFT | 83.2165 ms TTFT | 159.3235 ms TTFT | 303.6380 ms TTFT | 501.8685 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 43.5184 ms TTFT | 73.8242 ms TTFT | 141.4362 ms TTFT | 264.6536 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 38.9324 ms TTFT | 62.2434 ms TTFT | 118.7745 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 31.8862 ms TTFT | 47.61,104 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,241.0 | — | 1,312 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,079.0 | — | 2,669 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,722.0 | — | 5,644 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,144.0 | — | 12,524 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,245.0 | — | 30,321 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,170.0 | — | 79,563 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,149.0 | — | 81,674 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 48.6 | 48.6 | 90 ms | 20.6 ms | 1.0 | 0 | 1,456 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 88.3 | 44.1 | 138 ms | 22.7 ms | 2.0 | 0 | 2,648 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 170.3 | 42.6 | 273 ms | 23.5 ms | 4.0 | 0 | 5,104 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 339.9 | 42.4 | 456 ms | 23.6 ms | 8.0 | 0 | 10,192 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 606.6 | 38.0 | 981 ms | 26.3 ms | 16.0 | 0 | 18,192 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 980.5 | 30.4 | 1,654 ms | 32.9 ms | 32.0 | 0 | 29,408 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,447.2 | 22.4 | 3,444 ms | 44.6 ms | 64.0 | 0 | 43,392 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 47.6 | 47.7 | 146 ms | 21.0 ms | 1.0 | 0 | 1,429 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 86.1 | 43.0 | 241 ms | 23.2 ms | 2.0 | 0 | 2,582 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 166.0 | 41.5 | 437 ms | 24.1 ms | 4.0 | 0 | 4,980 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 326.5 | 40.7 | 730 ms | 24.6 ms | 8.0 | 0 | 9,792 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 564.9 | 35.4 | 1,605 ms | 28.3 ms | 16.0 | 0 | 16,944 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 878.2 | 27.1 | 2,620 ms | 36.9 ms | 32.0 | 0 | 26,336 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 46.3 | 46.3 | 115 ms | 21.6 ms | 1.0 | 0 | 1,387 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 83.2 | 41.6 | 165 ms | 24.0 ms | 2.0 | 0 | 2,496 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 159.3 | 39.8 | 235 ms | 25.1 ms | 4.0 | 0 | 4,780 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 303.6 | 37.9 | 380 ms | 26.4 ms | 8.0 | 0 | 9,104 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 501.8 | 31.3 | 685 ms | 32.0 ms | 16.0 | 0 | 15,040 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 43.5 | 43.5 | 184 ms | 23.0 ms | 1.0 | 0 | 1,305 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 73.8 | 36.9 | 242 ms | 27.1 ms | 2.0 | 0 | 2,215 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 141.4 | 35.3 | 362 ms | 28.3 ms | 4.0 | 0 | 4,240 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 264.6 | 32.8 | 536 ms | 30.5 ms | 8.0 | 0 | 7,936 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 38.9 | 38.9 | 324 ms | 25.7 ms | 1.0 | 0 | 1,166 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 62.2 | 31.1 | 434 ms | 32.1 ms | 2.0 | 0 | 1,866 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 118.7 | 29.6 | 745 ms | 33.8 ms | 4.0 | 0 | 3,560 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 31.8 | 31.8 | 862 ms | 31.4 ms | 1.0 | 0 | 954 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 47.6 | 23.7 | 1,104 ms | 42.2 ms | 2.0 | 0 | 1,428 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.