Speed test matrix
Muse-Glimmer-30B-FP8-block on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/Muse-Glimmer-30B-FP8-block --served-model-name muse-glimmer --tp 1 --reasoning-parser muse --tool-call-parser muse --context-length 131072 --mem-fraction-static 0.858257 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DFLASH --speculative-draft-model-path meta-models/Muse-Glimmer-30B-assistant --speculative-num-draft-tokens 16
Prefill
prompt tok/s by context length8K context14.8K553 ms TTFT
16K context15K1,080 ms TTFT
32K context14.6K2,200 ms TTFT
64K context13.5K4,755 ms TTFT
123K context11.6K10,639 ms TTFT
128K context11.5K11,145 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 456.7106 ms TTFT | 926.0118 ms TTFT | 1,737.7Requests queued115 ms TTFT | 2,552.2130 ms TTFT | 3,440.3183 ms TTFT | 3,420.3749 ms TTFT | Not run | Over capacity |
| 16K | 545.7140 ms TTFT | 788.6148 ms TTFT | 1,569.1148 ms TTFT | 3,028.3152 ms TTFT | 3,285.5199 ms TTFT | 3,629.71,502 ms TTFT | Not run | Over capacity |
| 32K | 478.3192 ms TTFT | 811.3198 ms TTFT | 1,486.0201 ms TTFT | 2,392.5223 ms TTFT | 3,137.01,030 ms TTFT | 3,896.23,292 ms TTFT | Over capacity | Over capacity |
| 64K | 432.0300 ms TTFT | 540.7331 ms TTFT | 1,673.2296 ms TTFT | 2,348.8381 ms TTFT | 2,859.41,994 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 412.8512 ms TTFT | 822.6516 ms TTFT | 1,580.0508 ms TTFT | 2,105.8866 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.8K | — | 553 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15K | — | 1,080 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.6K | — | 2,200 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.5K | — | 4,755 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 11.6K | — | 10,639 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.5K | — | 11,145 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 456.7 | 389.6 | 106 ms | 2.6 ms | 0.9 | 3 | 13,702 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 926.0 | 431.3 | 118 ms | 2.3 ms | 2.0 | 6 | 27,767 | 30.0 s |
| Sustained decode | 8K | 4 | Requests queued | 1,737.7 | 430.7 | 115 ms | 2.3 ms | 3.8 | 12 | 52,124 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,552.2 | 302.7 | 130 ms | 3.3 ms | 8.1 | 16 | 76,544 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,440.3 | 205.5 | 183 ms | 4.9 ms | 16.3 | 18 | 103,167 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,420.3 | 99.8 | 749 ms | 10.0 ms | 35.5 | 7 | 102,493 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 545.7 | 543.9 | 140 ms | 1.8 ms | 0.9 | 4 | 16,367 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 788.6 | 404.0 | 148 ms | 2.5 ms | 4.7 | 5 | 23,595 | 29.9 s |
| Sustained decode | 16K | 4 | Valid | 1,569.1 | 388.4 | 148 ms | 2.6 ms | 4.1 | 11 | 47,057 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,028.3 | 349.7 | 152 ms | 2.9 ms | 7.8 | 22 | 90,801 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,285.5 | 190.6 | 199 ms | 5.2 ms | 18.2 | 17 | 98,509 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,629.7 | 112.1 | 1,502 ms | 8.9 ms | 33.8 | 11 | 108,778 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 478.3 | 456.8 | 192 ms | 2.2 ms | 0.9 | 3 | 14,345 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 811.3 | 364.4 | 198 ms | 2.9 ms | 6.4 | 6 | 24,338 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,486.0 | 408.4 | 201 ms | 2.4 ms | 4.0 | 10 | 44,577 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,392.5 | 268.1 | 223 ms | 3.7 ms | 8.0 | 16 | 71,762 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,137.0 | 187.4 | 1,030 ms | 5.3 ms | 16.5 | 16 | 94,108 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,896.2 | 112.7 | 3,292 ms | 8.9 ms | 32.9 | 21 | 116,838 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 432.0 | 408.8 | 300 ms | 2.5 ms | 0.9 | 3 | 12,958 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 540.7 | 213.6 | 331 ms | 4.7 ms | 13.8 | 4 | 16,220 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,673.2 | 404.9 | 296 ms | 2.5 ms | 4.0 | 12 | 50,185 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,348.8 | 292.4 | 381 ms | 3.4 ms | 7.9 | 16 | 70,494 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,859.4 | 168.8 | 1,994 ms | 5.9 ms | 16.3 | 16 | 85,670 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 412.8 | 419.4 | 512 ms | 2.4 ms | 0.9 | 3 | 12,380 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 822.6 | 396.6 | 516 ms | 2.6 ms | 2.6 | 6 | 24,667 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 1,580.0 | 404.1 | 508 ms | 2.5 ms | 3.9 | 11 | 47,386 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 2,105.8 | 250.5 | 866 ms | 4.0 ms | 8.0 | 15 | 62,530 | 29.7 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.