Speed test matrix
Qwen3.6-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883421 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context10.4K789 ms TTFT
16K context10.1K1,609 ms TTFT
32K context9,6803,336 ms TTFT
64K context8,8377,289 ms TTFT
128K context7,47217,224 ms TTFT
251K context5,66244,540 ms TTFT
256K context5,63145,677 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 59.770 ms TTFT | 118.1125 ms TTFT | 229.21,810 ms TTFT | 432.6320 ms TTFT | 790.1684 ms TTFT | 1,370.21,217 ms TTFT | 2,067.62,350 ms TTFT | Over capacity |
| 16K | 59.2131 ms TTFT | 116.6180 ms TTFT | 224.1313 ms TTFT | 414.8490 ms TTFT | 734.6936 ms TTFT | 1,203.91,801 ms TTFT | Over capacity | Over capacity |
| 32K | 58.4117 ms TTFT | 113.6180 ms TTFT | 214.4250 ms TTFT | 383.9317 ms TTFT | 642.0494 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 56.6157 ms TTFT | 108.2257 ms TTFT | 197.5383 ms TTFT | 332.7612 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 53.7277 ms TTFT | 98.6396 ms TTFT | 169.3551 ms TTFT | 256.8942 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 48.6657 ms TTFT | 83.3875 ms TTFT | 129.61,196 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.4K | — | 789 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.1K | — | 1,609 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,680.0 | — | 3,336 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,837.0 | — | 7,289 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,472.0 | — | 17,224 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,662.0 | — | 44,540 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,631.0 | — | 45,677 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 59.7 | 59.7 | 70 ms | 16.7 ms | 1.0 | 0 | 1,791 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 118.1 | 59.1 | 125 ms | 16.9 ms | 2.0 | 0 | 3,542 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 229.2 | 57.3 | 1,810 ms | 17.4 ms | 4.0 | 0 | 6,872 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 432.6 | 54.1 | 320 ms | 18.5 ms | 8.0 | 0 | 12,984 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 790.1 | 49.5 | 684 ms | 20.2 ms | 16.0 | 0 | 23,696 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,370.2 | 43.0 | 1,217 ms | 23.3 ms | 32.0 | 0 | 41,088 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,067.6 | 32.4 | 2,350 ms | 30.8 ms | 64.0 | 0 | 62,016 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 59.2 | 59.2 | 131 ms | 16.9 ms | 1.0 | 0 | 1,777 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 116.6 | 58.3 | 180 ms | 17.1 ms | 2.0 | 0 | 3,498 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 224.1 | 56.1 | 313 ms | 17.8 ms | 4.0 | 0 | 6,724 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 414.8 | 51.8 | 490 ms | 19.3 ms | 8.0 | 0 | 12,443 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 734.6 | 45.9 | 936 ms | 21.8 ms | 16.0 | 0 | 22,032 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,203.9 | 37.6 | 1,801 ms | 26.6 ms | 32.0 | 0 | 36,096 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 58.4 | 58.4 | 117 ms | 17.1 ms | 1.0 | 0 | 1,751 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 113.6 | 56.9 | 180 ms | 17.6 ms | 2.0 | 0 | 3,408 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 214.4 | 53.6 | 250 ms | 18.6 ms | 4.0 | 0 | 6,432 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 383.9 | 48.0 | 317 ms | 20.8 ms | 8.0 | 0 | 11,520 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 642.0 | 40.1 | 494 ms | 25.0 ms | 16.0 | 0 | 19,248 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 56.6 | 56.6 | 157 ms | 17.7 ms | 1.0 | 0 | 1,697 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 108.2 | 54.1 | 257 ms | 18.5 ms | 2.0 | 0 | 3,244 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 197.5 | 49.4 | 383 ms | 20.2 ms | 4.0 | 0 | 5,924 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 332.7 | 41.6 | 612 ms | 24.0 ms | 8.0 | 0 | 9,984 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 53.7 | 53.8 | 277 ms | 18.6 ms | 1.0 | 0 | 1,611 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 98.6 | 49.4 | 396 ms | 20.3 ms | 2.0 | 0 | 2,958 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 169.3 | 42.4 | 551 ms | 23.6 ms | 4.0 | 0 | 5,076 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 256.8 | 32.1 | 942 ms | 31.1 ms | 8.0 | 0 | 7,704 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 48.6 | 48.7 | 657 ms | 20.5 ms | 1.0 | 0 | 1,458 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 83.3 | 41.7 | 875 ms | 24.0 ms | 2.0 | 0 | 2,498 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 129.6 | 32.3 | 1,196 ms | 30.9 ms | 4.0 | 0 | 3,884 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.