Speed test matrix
LFM2.5-24B-A2B on Tenstorrent P150 32GB
vllm · 0.1.dev14187+g03fa3af2e · BFP46 contexts (0–120K) × 5 concurrency levels (1–16) · prefill at 2K, 4K, 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- AI accelerator
- GPU
- Tenstorrent P150
- VRAM
- 32 GB
- CPU
- AMD Ryzen 9 9950X
- RAM
- 96 GB
- OS
- Ubuntu 26.04 LTS
Engine
- Engine
- vllm
- Version
- 0.1.dev14187+g03fa3af2e
- Quantization
- BFP4
- Backend
- tt-metal
Run
- Served model
- LiquidAI/LFM2.5-24B-A2B
- Revision
- 1769516ac588c4c558089c94f27d96b0dc67b183
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
python3 launch.py --revision m12 --max-num-seqs 16 --max-model-len 131072 --kv-pool-tokens 1048576 --device-ownership-confirmed # vLLM + TT plugin, image lottolabs/lfm2.5-24b-a2b-tt-p150:m12-20261014, GPTQ-BFP4 experts / BFP8 dense, sampling preset liquid, prompt-lookup speculation off (>1 slot)
Prefill
prompt tok/s by context length2K context7,717281 ms TTFT
4K context8,725479 ms TTFT
8K context9,804838 ms TTFT
16K context10.6K1,538 ms TTFT
32K context9,6833,341 ms TTFT
64K context7,9968,076 ms TTFT
120K context6,03220,053 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 163.074 ms TTFT | 261.7134 ms TTFT | 395.7258 ms TTFT | 580.7497 ms TTFT | 805.5982 ms TTFT |
| 4K | 160.584 ms TTFT | 250.6159 ms TTFT | 381.0299 ms TTFT | 547.6579 ms TTFT | 758.51,162 ms TTFT |
| 16K | 154.0249 ms TTFT | Looped | 343.1960 ms TTFT | 449.41,907 ms TTFT | 616.92,840 ms TTFT |
| 32K | 143.0280 ms TTFT | Looped | 293.81,081 ms TTFT | 380.72,145 ms TTFT | 490.84,282 ms TTFT |
| 64K | 128.1390 ms TTFT | 181.3761 ms TTFT | Looped | 286.52,985 ms TTFT | Over capacity |
| 120K | 106.1238 ms TTFT | 143.1427 ms TTFT | 181.0775 ms TTFT | 205.51,532 ms TTFT | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 2K | 1 | Valid | 7,717.0 | — | 281 ms | — | — | 1 | — | — |
| Prefill | 4K | 1 | Valid | 8,725.0 | — | 479 ms | — | — | 1 | — | — |
| Prefill | 8K | 1 | Valid | 9,804.0 | — | 838 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.6K | — | 1,538 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,683.0 | — | 3,341 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,996.0 | — | 8,076 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 6,032.0 | — | 20,053 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 163.0 | 163.1 | 74 ms | 6.1 ms | 1.0 | 0 | 4,888 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 261.7 | 131.6 | 134 ms | 7.6 ms | 2.0 | 0 | 7,850 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 395.7 | 100.3 | 258 ms | 10.0 ms | 4.0 | 0 | 11,872 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 580.7 | 73.8 | 497 ms | 13.5 ms | 8.0 | 0 | 17,416 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 805.5 | 52.0 | 982 ms | 19.2 ms | 16.0 | 0 | 24,160 | 30.0 s |
| Sustained decode | 4K | 1 | Valid | 160.5 | 160.6 | 84 ms | 6.2 ms | 1.0 | 0 | 4,813 | 30.0 s |
| Sustained decode | 4K | 2 | Valid | 250.6 | 126.1 | 159 ms | 7.9 ms | 2.0 | 0 | 7,516 | 30.0 s |
| Sustained decode | 4K | 4 | Valid | 381.0 | 96.5 | 299 ms | 10.4 ms | 4.0 | 0 | 11,428 | 30.0 s |
| Sustained decode | 4K | 8 | Valid | 547.6 | 70.3 | 579 ms | 14.2 ms | 8.0 | 0 | 16,424 | 30.0 s |
| Sustained decode | 4K | 16 | Valid | 758.5 | 49.7 | 1,162 ms | 20.1 ms | 16.0 | 0 | 22,752 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 154.0 | 154.2 | 249 ms | 6.5 ms | 1.0 | 0 | 4,621 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Valid | 343.1 | 86.9 | 960 ms | 11.5 ms | 4.0 | 0 | 10,292 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 449.4 | 57.4 | 1,907 ms | 17.4 ms | 8.0 | 0 | 13,480 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 616.9 | 38.6 | 2,840 ms | 25.9 ms | 16.0 | 0 | 18,496 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 143.0 | 143.1 | 280 ms | 7.0 ms | 1.0 | 0 | 4,288 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 293.8 | 74.3 | 1,081 ms | 13.5 ms | 4.0 | 0 | 8,812 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 380.7 | 48.4 | 2,145 ms | 20.7 ms | 8.0 | 0 | 11,416 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 490.8 | 31.6 | 4,282 ms | 31.6 ms | 16.0 | 0 | 14,720 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 128.1 | 128.1 | 390 ms | 7.8 ms | 1.0 | 0 | 3,842 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 181.3 | 91.0 | 761 ms | 11.0 ms | 2.0 | 0 | 5,438 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 286.5 | 36.4 | 2,985 ms | 27.4 ms | 8.0 | 0 | 8,592 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 106.1 | 106.4 | 238 ms | 9.4 ms | 1.0 | 0 | 3,183 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 143.1 | 71.9 | 427 ms | 13.9 ms | 2.0 | 0 | 4,292 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 181.0 | 45.7 | 775 ms | 21.9 ms | 4.0 | 0 | 5,428 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 205.5 | 26.2 | 1,532 ms | 38.2 ms | 8.0 | 0 | 6,160 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.