Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
Matching runs 699
Submitted by @keennay. Newest first. Page 24 of 28.- gemma-4-31B-itH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 617.8 tok/s
- Peak prefill
- 7,857 tok/s
- gemma-4-26B-A4B-itH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,153.3 tok/s
- Peak prefill
- 35.6K tok/s
- gemma-4-12B-itH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,247.7 tok/s
- Peak prefill
- 19.4K tok/s
- Peak decode
- 6,830.5 tok/s
- Peak prefill
- 30.8K tok/s
- Peak decode
- 6,566.0 tok/s
- Peak prefill
- 31.3K tok/s
- command-a-plus-05-2026-w4a4H200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP45 contexts × 8 concurrency levels
- Peak decode
- 2,814.9 tok/s
- Peak prefill
- 9,940 tok/s
- command-a-plus-05-2026-fp8H200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP85 contexts × 8 concurrency levels
- Peak decode
- 1,344.3 tok/s
- Peak prefill
- 19.6K tok/s
- Trinity-Nano-PreviewH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 6,792.3 tok/s
- Peak prefill
- 77.8K tok/s
- Trinity-Nano-Preview-W4A16H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT45 contexts × 8 concurrency levels
- Peak decode
- 8,973.3 tok/s
- Peak prefill
- 67.4K tok/s
- Peak decode
- 8,393.9 tok/s
- Peak prefill
- 69.1K tok/s
- Trinity-Nano-Preview-FP8-BlockH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP85 contexts × 8 concurrency levels
- Peak decode
- 6,528.7 tok/s
- Peak prefill
- 76.8K tok/s
- Trinity-Nano-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 8,293.8 tok/s
- Peak prefill
- 76.5K tok/s
- Trinity-MiniH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 5,784.5 tok/s
- Peak prefill
- 41.4K tok/s
- Trinity-Mini-W4A16H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT45 contexts × 8 concurrency levels
- Peak decode
- 7,297.7 tok/s
- Peak prefill
- 35.4K tok/s
- Peak decode
- 6,204.8 tok/s
- Peak prefill
- 34K tok/s
- Trinity-Mini-FP8-BlockH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP85 contexts × 8 concurrency levels
- Peak decode
- 6,041.8 tok/s
- Peak prefill
- 48.7K tok/s
- Trinity-Mini-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 5,521.0 tok/s
- Peak prefill
- 44.1K tok/s
- Trinity-Large-Thinking-W4A16H200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · INT46 contexts × 8 concurrency levels
- Peak decode
- 1,265.3 tok/s
- Peak prefill
- 16.3K tok/s
- Olmo-3.1-7B-RL-Zero-MathH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF164 contexts × 8 concurrency levels
- Peak decode
- 1,419.5 tok/s
- Peak prefill
- 31.5K tok/s
- Olmo-3.1-7B-RL-Zero-CodeH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF164 contexts × 8 concurrency levels
- Peak decode
- 1,423.2 tok/s
- Peak prefill
- 30.2K tok/s
- Olmo-3.1-32B-ThinkH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF164 contexts × 8 concurrency levels
- Peak decode
- 1,639.9 tok/s
- Peak prefill
- 8,490 tok/s
- Olmo-3.1-32B-InstructH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF164 contexts × 8 concurrency levels
- Peak decode
- 1,638.4 tok/s
- Peak prefill
- 8,567 tok/s
- Olmo-3.1-32B-Instruct-SFTH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF164 contexts × 8 concurrency levels
- Peak decode
- 1,641.7 tok/s
- Peak prefill
- 8,414 tok/s
- Olmo-3.1-32B-Instruct-DPOH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF164 contexts × 8 concurrency levels
- Peak decode
- 1,640.0 tok/s
- Peak prefill
- 8,762 tok/s
- Olmo-3-7B-ThinkH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF164 contexts × 8 concurrency levels
- Peak decode
- 1,418.8 tok/s
- Peak prefill
- 32K tok/s