Speed test matrix
Qwen3.5-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.860078 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context9,631846 ms TTFT
16K context9,5311,692 ms TTFT
32K context9,2193,482 ms TTFT
64K context8,3307,690 ms TTFT
128K context7,05418,142 ms TTFT
251K context5,35946,800 ms TTFT
256K context5,32648,022 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 148.499 ms TTFT | 285.3128 ms TTFT | 504.9888 ms TTFT | 799.7205 ms TTFT | 1,559.6349 ms TTFT | 1,843.2Requests queued4,363 ms TTFT | Over capacity | Over capacity |
| 16K | 141.2117 ms TTFT | 203.4209 ms TTFT | 523.0180 ms TTFT | 733.3342 ms TTFT | 1,502.3657 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 144.4173 ms TTFT | 265.2223 ms TTFT | 439.5413 ms TTFT | 840.8803 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 135.2297 ms TTFT | 268.3414 ms TTFT | 402.3949 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 134.6532 ms TTFT | 240.9998 ms TTFT | 345.51,876 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 115.61,022 ms TTFT | 189.22,041 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,631.0 | — | 846 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,531.0 | — | 1,692 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,219.0 | — | 3,482 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,330.0 | — | 7,690 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,054.0 | — | 18,142 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,359.0 | — | 46,800 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,326.0 | — | 48,022 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 148.4 | 150.4 | 99 ms | 6.6 ms | 1.0 | 1 | 4,453 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 285.3 | 146.7 | 128 ms | 6.8 ms | 1.9 | 2 | 8,557 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 504.9 | 136.4 | 888 ms | 7.4 ms | 5.3 | 4 | 15,139 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 799.7 | 101.1 | 205 ms | 9.9 ms | 12.0 | 0 | 23,991 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,559.6 | 97.1 | 349 ms | 10.3 ms | 16.0 | 0 | 46,790 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,843.2 | 83.8 | 4,363 ms | 11.9 ms | 22.0 | 22 | 55,241 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 141.2 | 144.5 | 117 ms | 6.9 ms | 1.0 | 1 | 4,234 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 203.4 | 101.9 | 209 ms | 9.8 ms | 12.0 | 0 | 6,103 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 523.0 | 132.6 | 180 ms | 7.5 ms | 4.0 | 4 | 15,681 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 733.3 | 92.7 | 342 ms | 10.8 ms | 12.0 | 0 | 21,987 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,502.3 | 90.4 | 657 ms | 11.1 ms | 16.0 | 0 | 45,032 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 144.4 | 147.3 | 173 ms | 6.8 ms | 1.0 | 1 | 4,331 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 265.2 | 134.2 | 223 ms | 7.5 ms | 2.3 | 2 | 7,954 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 439.5 | 111.7 | 413 ms | 9.0 ms | 5.8 | 0 | 13,185 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 840.8 | 106.6 | 803 ms | 9.4 ms | 8.0 | 0 | 25,223 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 135.2 | 142.4 | 297 ms | 7.0 ms | 1.0 | 1 | 4,055 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 268.3 | 135.7 | 414 ms | 7.4 ms | 1.9 | 2 | 8,050 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 402.3 | 101.7 | 949 ms | 9.8 ms | 5.9 | 0 | 12,061 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 134.6 | 135.7 | 532 ms | 7.4 ms | 1.0 | 1 | 4,037 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 240.9 | 123.9 | 998 ms | 8.1 ms | 2.0 | 1 | 7,225 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 345.5 | 88.7 | 1,876 ms | 11.3 ms | 5.0 | 0 | 10,363 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 115.6 | 115.5 | 1,022 ms | 8.7 ms | 1.0 | 0 | 3,466 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 189.2 | 94.2 | 2,041 ms | 10.6 ms | 2.8 | 0 | 5,672 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.