Speed test matrix
Laguna-XS.2-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-poolside-sglang/bin/sglang" serve --model-path poolside/Laguna-XS.2-FP8 --served-model-name poolside --trust-remote-code --tp 1 --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --context-length 262144 --mem-fraction-static 0.850588 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context40.7K202 ms TTFT
16K context41.6K390 ms TTFT
32K context37.9K849 ms TTFT
64K context29.8K2,155 ms TTFT
128K context20.8K6,161 ms TTFT
251K context13.1K19,213 ms TTFT
256K context13K19,794 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 187.379 ms TTFT | 226.4107 ms TTFT | 600.5119 ms TTFT | 865.7132 ms TTFT | 1,573.9197 ms TTFT | 2,419.2348 ms TTFT | Over capacity | Over capacity |
| 16K | 184.392 ms TTFT | 212.5141 ms TTFT | 580.2139 ms TTFT | 773.1185 ms TTFT | 1,448.2328 ms TTFT | 2,051.5617 ms TTFT | Over capacity | Over capacity |
| 32K | 178.6118 ms TTFT | 126.1162 ms TTFT | 491.3216 ms TTFT | 663.1366 ms TTFT | 1,120.4686 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 168.9175 ms TTFT | 158.8243 ms TTFT | 445.3390 ms TTFT | 725.9801 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 152.2292 ms TTFT | 221.3425 ms TTFT | 414.1860 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 127.5498 ms TTFT | 206.8862 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.7K | — | 202 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 41.6K | — | 390 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 37.9K | — | 849 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.8K | — | 2,155 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.8K | — | 6,161 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.1K | — | 19,213 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13K | — | 19,794 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 187.3 | 188.0 | 79 ms | 5.3 ms | 1.0 | 1 | 5,619 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 226.4 | 112.7 | 107 ms | 8.9 ms | 2.9 | 0 | 6,792 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 600.5 | 151.1 | 119 ms | 6.6 ms | 4.0 | 4 | 18,012 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 865.7 | 107.9 | 132 ms | 9.3 ms | 10.8 | 0 | 25,968 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,573.9 | 98.6 | 197 ms | 10.1 ms | 16.0 | 0 | 47,216 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,419.2 | 74.4 | 348 ms | 13.4 ms | 33.6 | 0 | 72,576 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 184.3 | 185.1 | 92 ms | 5.4 ms | 1.0 | 1 | 5,529 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 212.5 | 100.8 | 141 ms | 9.9 ms | 21.2 | 0 | 6,374 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 580.2 | 145.2 | 139 ms | 6.9 ms | 4.0 | 4 | 17,404 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 773.1 | 96.7 | 185 ms | 10.3 ms | 11.5 | 0 | 23,193 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,448.2 | 88.5 | 328 ms | 11.3 ms | 16.8 | 0 | 43,440 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,051.5 | 63.4 | 617 ms | 15.8 ms | 37.9 | 0 | 61,536 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 178.6 | 179.4 | 118 ms | 5.6 ms | 1.0 | 1 | 5,357 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 126.1 | 62.1 | 162 ms | 16.1 ms | 31.9 | 0 | 3,782 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 491.3 | 123.8 | 216 ms | 8.1 ms | 4.7 | 4 | 14,737 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 663.1 | 83.2 | 366 ms | 12.0 ms | 12.0 | 0 | 19,888 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,120.4 | 68.5 | 686 ms | 14.6 ms | 20.7 | 0 | 33,600 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 168.9 | 170.1 | 175 ms | 5.9 ms | 1.0 | 1 | 5,066 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 158.8 | 77.3 | 243 ms | 12.9 ms | 14.3 | 0 | 4,762 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 445.3 | 108.0 | 390 ms | 9.3 ms | 4.6 | 0 | 13,360 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 725.9 | 90.9 | 801 ms | 11.0 ms | 8.0 | 0 | 21,776 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 152.2 | 153.8 | 292 ms | 6.5 ms | 1.0 | 1 | 4,565 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 221.3 | 106.3 | 425 ms | 9.4 ms | 4.4 | 0 | 6,638 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 414.1 | 100.8 | 860 ms | 9.9 ms | 4.0 | 0 | 12,420 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 127.5 | 129.9 | 498 ms | 7.7 ms | 1.0 | 1 | 3,824 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 206.8 | 99.2 | 862 ms | 10.1 ms | 2.2 | 0 | 6,204 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.