Speed test matrix
Qwen3.8-Flash-Next on H200 SXM 141GB ×4
vllm · 0.1.dev1+g50a061f79 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g50a061f79
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
50a061f792f3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_MMAP=1 "$HOME/env_h200-qwen-flash-next-vllm-pr-54129/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-expert-parallel --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context24.8K331 ms TTFT
16K context27.7K584 ms TTFT
32K context29.5K1,091 ms TTFT
64K context30.3K2,116 ms TTFT
128K context30.1K4,256 ms TTFT
251K context28.4K8,847 ms TTFT
256K context26K9,857 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 157.0359 ms TTFT | 266.5291 ms TTFT | 446.7541 ms TTFT | 709.7585 ms TTFT | 1,199.01,489 ms TTFT | 2,025.01,074 ms TTFT | 3,383.41,633 ms TTFT | 5,292.72,732 ms TTFT |
| 16K | 159.7490 ms TTFT | 268.9308 ms TTFT | 458.6577 ms TTFT | 723.9638 ms TTFT | 1,213.6857 ms TTFT | 2,054.11,331 ms TTFT | 3,413.51,864 ms TTFT | 5,358.83,022 ms TTFT |
| 32K | 155.5761 ms TTFT | 269.9348 ms TTFT | 458.0644 ms TTFT | 721.3903 ms TTFT | 1,214.21,300 ms TTFT | 2,063.11,842 ms TTFT | 3,431.02,659 ms TTFT | 5,325.64,289 ms TTFT |
| 64K | 156.41,323 ms TTFT | 269.41,419 ms TTFT | 455.0920 ms TTFT | 732.81,337 ms TTFT | 1,205.21,844 ms TTFT | 2,061.52,781 ms TTFT | 3,407.14,508 ms TTFT | 5,268.38,187 ms TTFT |
| 128K | 157.32,499 ms TTFT | Looped | 452.91,089 ms TTFT | 729.01,806 ms TTFT | 1,194.82,780 ms TTFT | 2,025.24,788 ms TTFT | 3,327.38,778 ms TTFT | 5,159.117,683 ms TTFT |
| 251K | 156.34,999 ms TTFT | 265.05,254 ms TTFT | 446.19,626 ms TTFT | 715.03,099 ms TTFT | 1,189.75,598 ms TTFT | 1,984.910,111 ms TTFT | 3,187.618,760 ms TTFT | 4,811.636,546 ms TTFT |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24.8K | — | 331 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 27.7K | — | 584 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.5K | — | 1,091 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.3K | — | 2,116 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 30.1K | — | 4,256 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 28.4K | — | 8,847 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 26K | — | 9,857 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 157.0 | 160.3 | 359 ms | 6.2 ms | 1.0 | 1 | 4,709 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 266.5 | 133.3 | 291 ms | 7.5 ms | 2.0 | 2 | 7,994 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 446.7 | 112.6 | 541 ms | 8.9 ms | 4.0 | 0 | 13,396 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 709.7 | 81.7 | 585 ms | 12.2 ms | 8.0 | 0 | 21,284 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,199.0 | 74.3 | 1,489 ms | 13.5 ms | 16.0 | 0 | 35,967 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,025.0 | 63.5 | 1,074 ms | 15.8 ms | 32.0 | 0 | 60,752 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,383.4 | 51.6 | 1,633 ms | 19.4 ms | 64.0 | 0 | 101,432 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,292.7 | 39.4 | 2,732 ms | 25.4 ms | 128.0 | 0 | 158,691 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 159.7 | 162.5 | 490 ms | 6.2 ms | 1.0 | 1 | 4,790 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 268.9 | 134.5 | 308 ms | 7.4 ms | 2.0 | 2 | 8,065 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 458.6 | 115.1 | 577 ms | 8.7 ms | 4.0 | 0 | 13,758 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 723.9 | 90.7 | 638 ms | 11.0 ms | 8.0 | 0 | 21,711 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,213.6 | 75.7 | 857 ms | 13.2 ms | 16.0 | 0 | 36,391 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,054.1 | 64.2 | 1,331 ms | 15.6 ms | 32.0 | 0 | 61,603 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,413.5 | 52.0 | 1,864 ms | 19.2 ms | 64.0 | 0 | 102,402 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,358.8 | 40.7 | 3,022 ms | 24.6 ms | 128.0 | 0 | 160,631 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 155.5 | 159.7 | 761 ms | 6.3 ms | 1.0 | 1 | 4,664 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 269.9 | 136.7 | 348 ms | 7.3 ms | 2.0 | 2 | 8,097 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 458.0 | 114.3 | 644 ms | 8.7 ms | 4.0 | 0 | 13,736 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 721.3 | 89.8 | 903 ms | 11.1 ms | 8.0 | 0 | 21,636 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,214.2 | 75.7 | 1,300 ms | 13.2 ms | 16.0 | 0 | 36,418 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,063.1 | 63.6 | 1,842 ms | 15.7 ms | 32.0 | 0 | 61,855 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,431.0 | 52.2 | 2,659 ms | 19.2 ms | 64.0 | 0 | 102,930 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,325.6 | 39.5 | 4,289 ms | 25.3 ms | 128.0 | 0 | 159,696 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 156.4 | 159.2 | 1,323 ms | 6.3 ms | 1.0 | 1 | 4,690 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 269.4 | 133.8 | 1,419 ms | 7.5 ms | 2.0 | 2 | 8,081 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 455.0 | 113.2 | 920 ms | 8.8 ms | 4.0 | 0 | 13,648 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 732.8 | 89.7 | 1,337 ms | 11.1 ms | 8.0 | 0 | 21,978 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,205.2 | 74.3 | 1,844 ms | 13.5 ms | 16.0 | 0 | 36,147 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,061.5 | 62.4 | 2,781 ms | 16.0 ms | 32.0 | 0 | 61,842 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,407.1 | 49.2 | 4,508 ms | 20.3 ms | 64.0 | 0 | 102,144 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 5,268.3 | 35.4 | 8,187 ms | 28.2 ms | 128.0 | 0 | 157,978 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 157.3 | 162.0 | 2,499 ms | 6.2 ms | 1.0 | 1 | 4,718 | 30.0 s |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Valid | 452.9 | 112.3 | 1,089 ms | 8.9 ms | 4.0 | 0 | 13,585 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 729.0 | 88.9 | 1,806 ms | 11.2 ms | 8.0 | 0 | 21,866 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,194.8 | 71.0 | 2,780 ms | 14.1 ms | 16.0 | 0 | 35,828 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,025.2 | 58.0 | 4,788 ms | 17.2 ms | 32.0 | 0 | 60,752 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 3,327.3 | 44.1 | 8,778 ms | 22.7 ms | 64.0 | 0 | 99,737 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 5,159.1 | 30.3 | 17,683 ms | 33.0 ms | 128.0 | 0 | 154,782 | 30.0 s |
| Sustained decode | 251K | 1 | Valid | 156.3 | 164.5 | 4,999 ms | 6.1 ms | 1.0 | 1 | 4,690 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 265.0 | 139.2 | 5,254 ms | 7.2 ms | 1.9 | 2 | 7,950 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 446.1 | 112.5 | 9,626 ms | 8.9 ms | 4.0 | 0 | 13,383 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 715.0 | 87.7 | 3,099 ms | 11.4 ms | 8.0 | 0 | 21,443 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,189.7 | 70.6 | 5,598 ms | 14.2 ms | 16.0 | 0 | 35,685 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,984.9 | 55.1 | 10,111 ms | 18.2 ms | 32.0 | 0 | 59,511 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 3,187.6 | 41.4 | 18,760 ms | 24.2 ms | 64.0 | 0 | 95,592 | 30.0 s |
| Sustained decode | 251K | 128 | Valid | 4,811.6 | 29.1 | 36,546 ms | 34.3 ms | 128.0 | 0 | 144,197 | 30.0 s |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement.