Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 SXM 141GB ×4
sglang · 0.5.6.post3.dev10338+g787299797 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev10338+g787299797
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
787299797931
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-glm53flash-dflash2-sglang-pr-37818/bin/sglang" serve --model-path RadixArk/GLM-5.3-Flash-NVFP4 --served-model-name glm --tp 4 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.70 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --kv-cache-dtype bfloat16 --moe-runner-backend marlin --speculative-algorithm NEXTN --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 --speculative-adaptive --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context12.5K657 ms TTFT
16K context16.3K994 ms TTFT
32K context16.5K1,955 ms TTFT
64K context16.5K3,902 ms TTFT
128K context16.3K7,907 ms TTFT
256K context15.6K16,530 ms TTFT
512K context14.6K35,350 ms TTFT
1019K context12.5K81,790 ms TTFT
1024K context12.1K85,157 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 205.7183 ms TTFT | 396.9255 ms TTFT | 620.8263 ms TTFT | 1,013.2272 ms TTFT | 1,686.0619 ms TTFT | 2,467.21,009 ms TTFT | Not run | Not run |
| 16K | 204.6227 ms TTFT | 359.2284 ms TTFT | 660.8299 ms TTFT | 1,097.1450 ms TTFT | 1,680.41,082 ms TTFT | 2,452.42,018 ms TTFT | Not run | Over capacity |
| 32K | 206.6440 ms TTFT | 370.6654 ms TTFT | 638.7434 ms TTFT | 1,043.3771 ms TTFT | 1,662.52,165 ms TTFT | 2,480.04,307 ms TTFT | Over capacity | Over capacity |
| 64K | 204.8432 ms TTFT | 362.5516 ms TTFT | 619.1844 ms TTFT | 1,004.62,218 ms TTFT | 1,655.54,463 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 199.8708 ms TTFT | 365.7920 ms TTFT | 614.41,467 ms TTFT | 993.92,788 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 196.61,223 ms TTFT | 364.21,855 ms TTFT | 603.7Requests queued3,506 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 184.52,425 ms TTFT | 315.8Requests queued4,264 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 149.66,113 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.5K | — | 657 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 16.3K | — | 994 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.5K | — | 1,955 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.5K | — | 3,902 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.3K | — | 7,907 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15.6K | — | 16,530 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 14.6K | — | 35,350 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 12.5K | — | 81,790 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 12.1K | — | 85,157 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 205.7 | 210.6 | 183 ms | 4.7 ms | 1.0 | 1 | 6,170 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 396.9 | 196.6 | 255 ms | 5.1 ms | 1.9 | 2 | 11,906 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 620.8 | 160.7 | 263 ms | 6.2 ms | 4.0 | 4 | 18,624 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,013.2 | 126.1 | 272 ms | 7.9 ms | 8.0 | 8 | 30,395 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,686.0 | 106.2 | 619 ms | 9.4 ms | 16.0 | 0 | 50,580 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,467.2 | 77.0 | 1,009 ms | 13.0 ms | 32.0 | 0 | 73,964 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 204.6 | 210.3 | 227 ms | 4.8 ms | 1.0 | 1 | 6,135 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 359.2 | 181.8 | 284 ms | 5.5 ms | 2.0 | 2 | 10,774 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 660.8 | 168.2 | 299 ms | 5.9 ms | 4.0 | 4 | 19,821 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,097.1 | 140.5 | 450 ms | 7.1 ms | 7.9 | 8 | 32,900 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,680.4 | 105.5 | 1,082 ms | 9.5 ms | 16.0 | 0 | 50,397 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,452.4 | 76.9 | 2,018 ms | 13.0 ms | 32.0 | 0 | 73,585 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 206.6 | 210.4 | 440 ms | 4.8 ms | 1.0 | 1 | 6,198 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 370.6 | 194.0 | 654 ms | 5.2 ms | 1.9 | 2 | 11,113 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 638.7 | 165.5 | 434 ms | 6.0 ms | 3.9 | 4 | 19,152 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,043.3 | 129.8 | 771 ms | 7.7 ms | 7.9 | 8 | 31,281 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,662.5 | 105.7 | 2,165 ms | 9.5 ms | 16.0 | 0 | 49,874 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,480.0 | 77.2 | 4,307 ms | 12.9 ms | 32.0 | 0 | 74,363 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 204.8 | 208.4 | 432 ms | 4.8 ms | 1.0 | 1 | 6,142 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 362.5 | 186.6 | 516 ms | 5.4 ms | 1.9 | 2 | 10,873 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 619.1 | 161.2 | 844 ms | 6.2 ms | 3.9 | 4 | 18,565 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,004.6 | 132.8 | 2,218 ms | 7.5 ms | 7.9 | 7 | 30,139 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,655.5 | 109.1 | 4,463 ms | 9.2 ms | 16.0 | 0 | 49,650 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 199.8 | 206.6 | 708 ms | 4.8 ms | 1.0 | 1 | 5,992 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 365.7 | 191.6 | 920 ms | 5.2 ms | 2.0 | 2 | 10,969 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 614.4 | 161.9 | 1,467 ms | 6.2 ms | 3.9 | 4 | 18,428 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 993.9 | 134.4 | 2,788 ms | 7.4 ms | 7.8 | 8 | 29,811 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 196.6 | 206.9 | 1,223 ms | 4.8 ms | 1.0 | 1 | 5,899 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 364.2 | 198.1 | 1,855 ms | 5.0 ms | 1.9 | 2 | 10,922 | 30.0 s |
| Sustained decode | 256K | 4 | Requests queued | 603.7 | 165.4 | 3,506 ms | 6.0 ms | 3.8 | 4 | 18,102 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 184.5 | 199.8 | 2,425 ms | 5.0 ms | 0.9 | 1 | 5,532 | 30.0 s |
| Sustained decode | 512K | 2 | Requests queued | 315.8 | 192.8 | 4,264 ms | 5.2 ms | 1.8 | 2 | 9,470 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 149.6 | 193.2 | 6,113 ms | 5.2 ms | 0.9 | 1 | 4,486 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.