Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
All runs 1,102
Newest first. Page 42 of 45.- gemma-4-E4B-itH200 NVL 141GB · sglang 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts × 8 concurrency levels
- Peak decode
- 5,036.5 tok/s
- Peak prefill
- 25.3K tok/s
- Peak decode
- 3,776.1 tok/s
- Peak prefill
- 28.6K tok/s
- Nanbeige4.2-3BH200 NVL 141GB · sglang 0.0.0.dev17517+g3e59d89e5 · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,195.7 tok/s
- Peak prefill
- 26.6K tok/s
- gemma-4-E2B-itH200 NVL 141GB · sglang 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts × 8 concurrency levels
- Peak decode
- 11.2K tok/s
- Peak prefill
- 33K tok/s
- gemma-4-26B-A4B-itH200 NVL 141GB · sglang 0.5.6.post3.dev9265+g05c584c44 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,659.8 tok/s
- Peak prefill
- 15.8K tok/s
- gemma-4-26B-A4B-it-FP8-dynamicH200 NVL 141GB · sglang 0.5.6.post3.dev9265+g05c584c44 · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,382.1 tok/s
- Peak prefill
- 18.1K tok/s
- Qwen3.8-27B-NVFP4H200 NVL 141GB · sglang 0.0.0.dev16762+g0c088029c · NVFP4+FP86 contexts × 8 concurrency levels
- Peak decode
- 1,559.6 tok/s
- Peak prefill
- 6,303 tok/s
- Qwen3.8-27B-FP8H200 NVL 141GB · sglang 0.0.0.dev16762+g0c088029c · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,297.6 tok/s
- Peak prefill
- 13.9K tok/s
- Qwen3.8-Flash-Next-NVFP4H200 NVL 141GB ×2 · sglang 0.5.6.post3.dev9023+g73a255206 · NVFP46 contexts × 8 concurrency levels
- Peak decode
- 2,235.0 tok/s
- Peak prefill
- 19.7K tok/s
- Qwen3.8-27B-NVFP4H200 NVL 141GB · vllm 0.27.2rc1.dev113+g5cecfc013 · NVFP4+FP86 contexts × 8 concurrency levels
- Peak decode
- 2,348.0 tok/s
- Peak prefill
- 6,938 tok/s
- Qwen3.8-27BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,147.3 tok/s
- Peak prefill
- 9,033 tok/s
- Qwen3.8-27BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,178.5 tok/s
- Peak prefill
- 9,219 tok/s
- Qwen3.8-27B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,471.5 tok/s
- Peak prefill
- 13K tok/s
- Qwen3.8-27B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,172.9 tok/s
- Peak prefill
- 13.4K tok/s
- Qwen3.6-35B-A3BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,236.4 tok/s
- Peak prefill
- 35.5K tok/s
- Qwen3.6-35B-A3BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,623.2 tok/s
- Peak prefill
- 37K tok/s
- Qwen3.6-35B-A3B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 3,602.8 tok/s
- Peak prefill
- 46.5K tok/s
- Qwen3.6-35B-A3B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 6,165.8 tok/s
- Peak prefill
- 47.7K tok/s
- Qwen3.6-27BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,692.5 tok/s
- Peak prefill
- 9,618 tok/s
- Qwen3.6-27BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,189.5 tok/s
- Peak prefill
- 9,781 tok/s
- Qwen3.6-27B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,389.1 tok/s
- Peak prefill
- 14.2K tok/s
- Qwen3.6-27B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,412.8 tok/s
- Peak prefill
- 14.5K tok/s
- Qwen3.5-9BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 5,688.7 tok/s
- Peak prefill
- 28.1K tok/s
- Qwen3.5-9BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,455.0 tok/s
- Peak prefill
- 28.9K tok/s
- Qwen3.5-9B-BaseH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 2,710.2 tok/s
- Peak prefill
- 28.4K tok/s