Speed test matrix
Qwen3.6-27B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass
Prefill
prompt tok/s by context length8K context9,942824 ms TTFT
16K context9,5461,699 ms TTFT
32K context8,7033,711 ms TTFT
64K context7,4048,700 ms TTFT
128K context5,68422,643 ms TTFT
251K context3,91264,478 ms TTFT
256K context3,86666,533 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 45.573 ms TTFT | 83.9109 ms TTFT | 162.7194 ms TTFT | 322.8310 ms TTFT | 581.1620 ms TTFT | 970.11,258 ms TTFT | 1,449.51,913 ms TTFT | Over capacity |
| 16K | 44.7124 ms TTFT | 82.3206 ms TTFT | 158.9317 ms TTFT | 310.8584 ms TTFT | 543.2916 ms TTFT | 869.71,928 ms TTFT | Over capacity | Over capacity |
| 32K | 43.4107 ms TTFT | 79.3147 ms TTFT | 152.2203 ms TTFT | 289.8336 ms TTFT | 483.4513 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 41.0174 ms TTFT | 70.8231 ms TTFT | 136.0337 ms TTFT | 254.0515 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 36.9318 ms TTFT | 60.1431 ms TTFT | 114.7597 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 30.5845 ms TTFT | 46.41,067 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,942.0 | — | 824 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,546.0 | — | 1,699 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,703.0 | — | 3,711 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,404.0 | — | 8,700 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,684.0 | — | 22,643 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,912.0 | — | 64,478 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,866.0 | — | 66,533 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 45.5 | 45.5 | 73 ms | 22.0 ms | 1.0 | 0 | 1,364 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 83.9 | 42.0 | 109 ms | 23.8 ms | 2.0 | 0 | 2,516 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 162.7 | 40.7 | 194 ms | 24.6 ms | 4.0 | 0 | 4,880 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 322.8 | 40.3 | 310 ms | 24.8 ms | 8.0 | 0 | 9,678 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 581.1 | 36.3 | 620 ms | 27.5 ms | 16.0 | 0 | 17,424 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 970.1 | 30.3 | 1,258 ms | 33.0 ms | 32.0 | 0 | 29,088 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,449.5 | 22.3 | 1,913 ms | 44.8 ms | 64.0 | 0 | 43,456 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 44.7 | 44.7 | 124 ms | 22.4 ms | 1.0 | 0 | 1,340 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 82.3 | 41.1 | 206 ms | 24.3 ms | 2.0 | 0 | 2,466 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 158.9 | 39.7 | 317 ms | 25.2 ms | 4.0 | 0 | 4,764 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 310.8 | 38.9 | 584 ms | 25.7 ms | 8.0 | 0 | 9,323 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 543.2 | 33.8 | 916 ms | 29.6 ms | 16.0 | 0 | 16,288 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 869.7 | 27.1 | 1,928 ms | 37.0 ms | 32.0 | 0 | 26,080 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 43.4 | 43.4 | 107 ms | 23.0 ms | 1.0 | 0 | 1,302 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 79.3 | 39.6 | 147 ms | 25.2 ms | 2.0 | 0 | 2,378 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 152.2 | 38.0 | 203 ms | 26.3 ms | 4.0 | 0 | 4,564 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 289.8 | 36.1 | 336 ms | 27.7 ms | 8.0 | 0 | 8,688 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 483.4 | 30.0 | 513 ms | 33.3 ms | 16.0 | 0 | 14,496 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 41.0 | 41.0 | 174 ms | 24.4 ms | 1.0 | 0 | 1,229 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 70.8 | 35.4 | 231 ms | 28.3 ms | 2.0 | 0 | 2,122 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 136.0 | 33.9 | 337 ms | 29.5 ms | 4.0 | 0 | 4,076 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 254.0 | 31.5 | 515 ms | 31.7 ms | 8.0 | 0 | 7,616 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 36.9 | 36.9 | 318 ms | 27.1 ms | 1.0 | 0 | 1,106 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 60.1 | 30.0 | 431 ms | 33.3 ms | 2.0 | 0 | 1,802 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 114.7 | 28.6 | 597 ms | 35.0 ms | 4.0 | 0 | 3,440 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 30.5 | 30.5 | 845 ms | 32.8 ms | 1.0 | 0 | 914 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 46.4 | 23.1 | 1,067 ms | 43.3 ms | 2.0 | 0 | 1,390 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.