Speed test matrix
LFM2.5-1.2B-Instruct on H200 NVL 141GB
sglang · 0.5.6.post3.dev9418+gc26ae0439 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9418+gc26ae0439
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/tugot17/sglang
- Build
- cuda-13.2
- Commit
c26ae043924f
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-liquidai-sglang-pr-31041/bin/sglang" serve --model-path LiquidAI/LFM2.5-1.2B-Instruct --served-model-name lfm --trust-remote-code --tp 1 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.867260 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --disable-radix-cache --speculative-algorithm DSPARK --speculative-draft-model-path LiquidAI/LFM2.5-1.2B-Instruct-DSpark --speculative-draft-attention-backend flashinfer
Prefill
prompt tok/s by context length8K context79.4K103 ms TTFT
16K context107.3K151 ms TTFT
32K context112.8K287 ms TTFT
64K context97K665 ms TTFT
120K context73.4K1,646 ms TTFT
125K context74.5K1,688 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 461.082 ms TTFT | 852.982 ms TTFT | 1,550.8108 ms TTFT | 2,623.0123 ms TTFT | 4,027.8Requests queued164 ms TTFT | 5,352.3Requests queued177 ms TTFT | Not run | Not run |
| 16K | 411.9140 ms TTFT | 543.1144 ms TTFT | 1,276.9153 ms TTFT | 2,106.4189 ms TTFT | 2,556.0298 ms TTFT | 3,736.0Requests queued593 ms TTFT | Not run | Not run |
| 32K | 375.7276 ms TTFT | 422.5406 ms TTFT | 1,078.3328 ms TTFT | 1,413.6550 ms TTFT | 1,872.71,940 ms TTFT | 2,693.13,681 ms TTFT | Not run | Not run |
| 64K | 393.3657 ms TTFT | 353.7904 ms TTFT | 898.7Requests queued1,075 ms TTFT | 1,217.5Requests queued1,084 ms TTFT | 1,064.84,788 ms TTFT | 1,781.49,704 ms TTFT | Not run | Over capacity |
| 120K | 320.31,602 ms TTFT | 262.72,366 ms TTFT | 542.6Requests queued2,226 ms TTFT | 562.86,591 ms TTFT | 595.812,215 ms TTFT | 581.8Requests queued21,459 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 79.4K | — | 103 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 107.3K | — | 151 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 112.8K | — | 287 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 97K | — | 665 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 73.4K | — | 1,646 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 74.5K | — | 1,688 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 461.0 | 467.5 | 82 ms | 2.1 ms | 1.0 | 3 | 13,829 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 852.9 | 430.8 | 82 ms | 2.3 ms | 2.0 | 6 | 25,587 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,550.8 | 391.3 | 108 ms | 2.6 ms | 4.0 | 12 | 46,521 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,623.0 | 336.0 | 123 ms | 3.0 ms | 8.5 | 16 | 78,692 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 4,027.8 | 251.9 | 164 ms | 4.0 ms | 15.9 | 32 | 120,837 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 5,352.3 | 130.8 | 177 ms | 7.6 ms | 31.8 | 59 | 160,527 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 411.9 | 419.3 | 140 ms | 2.4 ms | 1.0 | 3 | 12,355 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 543.1 | 369.8 | 144 ms | 2.7 ms | 15.2 | 4 | 16,292 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,276.9 | 326.4 | 153 ms | 3.1 ms | 4.3 | 8 | 38,304 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,106.4 | 260.6 | 189 ms | 3.8 ms | 7.9 | 16 | 63,188 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,556.0 | 163.7 | 298 ms | 6.1 ms | 20.0 | 16 | 76,682 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 3,736.0 | 111.2 | 593 ms | 9.0 ms | 31.8 | 31 | 112,062 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 375.7 | 391.0 | 276 ms | 2.6 ms | 1.0 | 3 | 11,271 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 422.5 | 241.1 | 406 ms | 4.7 ms | 17.2 | 2 | 12,674 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,078.3 | 269.7 | 328 ms | 3.7 ms | 3.9 | 8 | 32,349 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,413.6 | 178.5 | 550 ms | 5.6 ms | 9.7 | 8 | 42,405 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,872.7 | 110.8 | 1,940 ms | 9.0 ms | 20.1 | 0 | 56,172 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,693.1 | 77.3 | 3,681 ms | 12.9 ms | 32.0 | 0 | 80,794 | 30.0 s |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 393.3 | 421.3 | 657 ms | 2.4 ms | 0.9 | 3 | 11,797 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 353.7 | 223.2 | 904 ms | 5.1 ms | 18.6 | 2 | 10,611 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 898.7 | 226.2 | 1,075 ms | 4.4 ms | 4.3 | 4 | 26,957 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 1,217.5 | 149.4 | 1,084 ms | 6.7 ms | 7.8 | 8 | 36,522 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,064.8 | 58.8 | 4,788 ms | 17.0 ms | 24.0 | 0 | 31,935 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,781.4 | 39.4 | 9,704 ms | 25.4 ms | 33.2 | 0 | 53,441 | 30.0 s |
| Sustained decode | 64K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 320.3 | 358.4 | 1,602 ms | 2.8 ms | 0.9 | 2 | 9,608 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 262.7 | 117.7 | 2,366 ms | 8.5 ms | 17.5 | 1 | 7,869 | 30.0 s |
| Sustained decode | 120K | 4 | Requests queued | 542.6 | 136.7 | 2,226 ms | 7.3 ms | 4.2 | 4 | 16,277 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 562.8 | 60.6 | 6,591 ms | 16.5 ms | 12.0 | 0 | 16,875 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 595.8 | 27.7 | 12,215 ms | 36.0 ms | 25.1 | 0 | 17,872 | 30.0 s |
| Sustained decode | 120K | 32 | Requests queued | 581.8 | 17.0 | 21,459 ms | 58.9 ms | 45.0 | 0 | 17,446 | 30.0 s |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.