Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 NVL 141GB ×2
sglang · 0.0.0.dev0 · NVFP47 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
4a0c6f2fe049
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-glm53flash-dflash2-sglang-4a0c6f2fe/bin/sglang" serve --model-path nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --trust-remote-code --tp 2 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 524288 --mem-fraction-static 0.89 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --attention-backend dsa --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --fp4-gemm-backend marlin --moe-runner-backend marlin --speculative-algorithm DFLASH --speculative-draft-model-path incoai/GLM-5.3-Flash-DFlash2 --speculative-draft-attention-backend fa4 --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context9,773839 ms TTFT
16K context10.1K1,609 ms TTFT
32K context10.2K3,168 ms TTFT
64K context10.2K6,302 ms TTFT
128K context10.1K12,725 ms TTFT
256K context9,85426,143 ms TTFT
507K context9,39354,298 ms TTFT
512K context9,40554,765 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 204.430 ms TTFT | 294.056 ms TTFT | 442.574 ms TTFT | 645.571 ms TTFT | 724.0Requests queued115 ms TTFT | 739.6Requests queued92 ms TTFT | Not run | Over capacity |
| 16K | 195.762 ms TTFT | 301.194 ms TTFT | 439.1134 ms TTFT | 649.4178 ms TTFT | 712.3Requests queued218 ms TTFT | 710.9Requests queued232 ms TTFT | Over capacity | Over capacity |
| 32K | 185.252 ms TTFT | 298.377 ms TTFT | 443.6112 ms TTFT | 670.6139 ms TTFT | 752.8Requests queued168 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 196.885 ms TTFT | 311.5129 ms TTFT | 432.9206 ms TTFT | 645.5326 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 186.666 ms TTFT | 290.198 ms TTFT | 423.5153 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 172.866 ms TTFT | 279.596 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 196.989 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,773.0 | — | 839 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.1K | — | 1,609 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.2K | — | 3,168 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.2K | — | 6,302 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.1K | — | 12,725 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,854.0 | — | 26,143 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 9,393.0 | — | 54,298 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,405.0 | — | 54,765 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 204.4 | 205.1 | 30 ms | 4.9 ms | 1.0 | 1 | 6,131 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 294.0 | 153.8 | 56 ms | 6.5 ms | 2.0 | 2 | 8,820 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 442.5 | 114.0 | 74 ms | 8.8 ms | 4.0 | 0 | 13,265 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 645.5 | 83.5 | 71 ms | 12.0 ms | 8.0 | 0 | 19,367 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 724.0 | 71.0 | 115 ms | 14.1 ms | 10.2 | 11 | 21,701 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 739.6 | 69.8 | 92 ms | 14.3 ms | 10.0 | 11 | 22,181 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 195.7 | 199.5 | 62 ms | 5.0 ms | 1.0 | 1 | 5,871 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 301.1 | 156.0 | 94 ms | 6.4 ms | 1.9 | 2 | 9,031 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 439.1 | 110.3 | 134 ms | 9.1 ms | 4.0 | 1 | 13,170 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 649.4 | 84.8 | 178 ms | 11.8 ms | 8.0 | 0 | 19,471 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 712.3 | 68.7 | 218 ms | 14.6 ms | 10.0 | 11 | 21,357 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 710.9 | 71.0 | 232 ms | 14.1 ms | 10.0 | 11 | 21,322 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 185.2 | 189.2 | 52 ms | 5.3 ms | 1.0 | 1 | 5,556 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 298.3 | 157.7 | 77 ms | 6.3 ms | 2.0 | 2 | 8,948 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 443.6 | 114.3 | 112 ms | 8.8 ms | 4.0 | 0 | 13,304 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 670.6 | 86.6 | 139 ms | 11.5 ms | 8.0 | 0 | 20,107 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 752.8 | 69.0 | 168 ms | 14.5 ms | 10.0 | 11 | 22,570 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 196.8 | 201.1 | 85 ms | 5.0 ms | 1.0 | 1 | 5,902 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 311.5 | 157.8 | 129 ms | 6.3 ms | 2.0 | 2 | 9,341 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 432.9 | 112.1 | 206 ms | 8.9 ms | 4.0 | 1 | 12,984 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 645.5 | 82.4 | 326 ms | 12.1 ms | 8.0 | 0 | 19,364 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 186.6 | 193.2 | 66 ms | 5.2 ms | 1.0 | 1 | 5,595 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 290.1 | 146.2 | 98 ms | 6.8 ms | 2.0 | 2 | 8,699 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 423.5 | 112.3 | 153 ms | 8.9 ms | 4.0 | 1 | 12,700 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 172.8 | 177.6 | 66 ms | 5.6 ms | 1.0 | 1 | 5,184 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 279.5 | 150.7 | 96 ms | 6.7 ms | 2.0 | 2 | 8,383 | 30.0 s |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 196.9 | 200.4 | 89 ms | 5.0 ms | 1.0 | 1 | 5,906 | 30.0 s |
| Sustained decode | 507K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.