Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
All runs 823
Newest first. Page 27 of 33.- Nanbeige4.2-3B-BaseH200 NVL 141GB · vllm 0.1.dev20429+g62f6de733 · BF165 contexts × 8 concurrency levels
- Peak decode
- 2,511.7 tok/s
- Peak prefill
- 30.8K tok/s
- Shieldstral-1.0-3BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF163 contexts × 8 concurrency levels
- Peak decode
- 4,360.4 tok/s
- Peak prefill
- 61.6K tok/s
- Mistral-Small-4-119B-2603H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,058.0 tok/s
- Peak prefill
- 21.6K tok/s
- Mistral-Small-4-119B-2603-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP46 contexts × 8 concurrency levels
- Peak decode
- 1,290.9 tok/s
- Peak prefill
- 14.2K tok/s
- Mistral-Medium-3.5-128BH200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,215.4 tok/s
- Peak prefill
- 4,755 tok/s
- Mistral-Medium-3.5-128BH200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 647.5 tok/s
- Peak prefill
- 4,837 tok/s
- Ministral-3-8B-Reasoning-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,125.9 tok/s
- Peak prefill
- 18.8K tok/s
- Ministral-3-8B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 3,440.2 tok/s
- Peak prefill
- 26.2K tok/s
- Ministral-3-8B-Base-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,161.1 tok/s
- Peak prefill
- 30.6K tok/s
- Ministral-3-3B-Reasoning-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,953.3 tok/s
- Peak prefill
- 28.4K tok/s
- Ministral-3-3B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 4,312.9 tok/s
- Peak prefill
- 38.5K tok/s
- Ministral-3-3B-Base-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,346.0 tok/s
- Peak prefill
- 58.9K tok/s
- Ministral-3-14B-Reasoning-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,755.7 tok/s
- Peak prefill
- 13.9K tok/s
- Ministral-3-14B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,079.1 tok/s
- Peak prefill
- 20.2K tok/s
- Ministral-3-14B-Base-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,905.1 tok/s
- Peak prefill
- 20.3K tok/s
- Devstral-Small-2-24B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,750.2 tok/s
- Peak prefill
- 14K tok/s
- Devstral-2-123B-Instruct-2512H200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 648.2 tok/s
- Peak prefill
- 4,984 tok/s
- Peak decode
- 6,566.4 tok/s
- Peak prefill
- 32.4K tok/s
- Peak decode
- 7,437.9 tok/s
- Peak prefill
- 48.5K tok/s
- Fara1.5-27BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 2,070.3 tok/s
- Peak prefill
- 10.4K tok/s
- LFM2.5-VL-3BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF163 contexts × 8 concurrency levels
- Peak decode
- 14.9K tok/s
- Peak prefill
- 69.4K tok/s
- LFM2.5-350MH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 24.3K tok/s
- Peak prefill
- 171.8K tok/s
- LFM2.5-230MH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 13.3K tok/s
- Peak prefill
- 192.2K tok/s
- LFM2.5-2.6BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 12.7K tok/s
- Peak prefill
- 66.2K tok/s
- LFM2.5-1.2B-InstructH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 22.2K tok/s
- Peak prefill
- 108.8K tok/s