Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
All runs 1,102
Newest first. Page 2 of 45.- Qwen3.5-9B-AWQ-4bitCMP 170HX 64GB · vllm 0.32.0rc1.dev249+g18305948a · W4A16 g326 contexts × 8 concurrency levels
- Peak decode
- 3,715.5 tok/s
- Peak prefill
- 6,231 tok/s
- DeepSeek-V4.1-Flash-1017-lossless-CSFRTX PRO 6000 Blackwell Server Edition 96GB ×4 · vllm · MXFP4+FP87 contexts × 8 concurrency levels
- Peak decode
- 1,583.5 tok/s
- Peak prefill
- 18.3K tok/s
- Peak decode
- 632.9 tok/s
- Peak prefill
- 3,997 tok/s
- DeepSeek-V4.1-Flash-1017RTX PRO 6000 Blackwell Server Edition 96GB ×4 · vllm 0.33.0rc1.dev141 · MXFP4+FP87 contexts × 8 concurrency levels
- Peak decode
- 2,801.2 tok/s
- Peak prefill
- 19.4K tok/s
- Peak decode
- 160.0 tok/s
- Peak prefill
- 1,823 tok/s
- Qwen3-Coder-30B-A3B-Instruct-AWQ-4bitRTX 3090 24GB · vllm 0.32.0 · AWQ 4-bit4 contexts × 8 concurrency levels
- Peak decode
- 753.1 tok/s
- Peak prefill
- 8,583 tok/s
- Peak decode
- 328.0 tok/s
- Peak prefill
- 1,859 tok/s
- DeepSeek-V4.1-Flash-1017-NVFP4RTX PRO 6000 Blackwell Server Edition 96GB ×4 · vllm 0.33.0rc1.dev141 · NVFP47 contexts × 8 concurrency levels
- Peak decode
- 1,605.6 tok/s
- Peak prefill
- 22.3K tok/s
- Mistral-Medium-3.5-128B-NVFP4CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · NVFP4 mixed2 contexts × 8 concurrency levels
- Peak decode
- 6.1 tok/s
- Peak prefill
- 368.0 tok/s
- Peak decode
- 160.2 tok/s
- Peak prefill
- 1,842 tok/s
- GLM-4.7-185B-W4A16CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · GPTQ W4A164 contexts × 8 concurrency levels
- Peak decode
- 31.3 tok/s
- Peak prefill
- 594.0 tok/s
- DeepSeek-V4.1-Flash-1017-NVFP4RTX PRO 6000 Blackwell Server Edition 96GB ×4 · vllm 0.33.0rc1.dev141 · NVFP47 contexts × 8 concurrency levels
- Peak decode
- 3,170.7 tok/s
- Peak prefill
- 22.4K tok/s
- Nemotron-Cascade-2-30B-A3B-AWQ-4bitRTX 3090 24GB · vllm 0.32.0 · AWQ 4-bit6 contexts × 8 concurrency levels
- Peak decode
- 1,020.5 tok/s
- Peak prefill
- 9,394 tok/s
- Peak decode
- 326.6 tok/s
- Peak prefill
- 1,849 tok/s
- Peak decode
- 439.4 tok/s
- Peak prefill
- 5,181 tok/s
- NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · NVFP4 mixed5 contexts × 8 concurrency levels
- Peak decode
- 778.0 tok/s
- Peak prefill
- 1,038 tok/s
- Peak decode
- 1,165.5 tok/s
- Peak prefill
- 5,802 tok/s
- internlm_internlm3-8b-instruct-autoround-int8-gs128-symRTX 3090 24GB · vllm 0.32.0 · AutoRound INT8 W8A16 gs1283 contexts × 8 concurrency levels
- Peak decode
- 1,107.2 tok/s
- Peak prefill
- 4,541 tok/s
- Qwen3-235B-A22B-Instruct-2507-REAP-nvfp4CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · NVFP46 contexts × 8 concurrency levels
- Peak decode
- 366.1 tok/s
- Peak prefill
- 1,048 tok/s
- Peak decode
- 1,763.9 tok/s
- Peak prefill
- 36.4K tok/s
- Peak decode
- 1,630.6 tok/s
- Peak prefill
- 20.1K tok/s
- Peak decode
- 1,567.1 tok/s
- Peak prefill
- 11.3K tok/s
- Llama-3.2-3B-Instruct-FP8-dynamicRTX 3090 24GB · vllm 0.32.0 · FP8 dynamic5 contexts × 8 concurrency levels
- Peak decode
- 1,494.4 tok/s
- Peak prefill
- 13.1K tok/s
- Peak decode
- 3,175.4 tok/s
- Peak prefill
- 10.8K tok/s
- Seed-OSS-36B-Instruct-FP8-StaticCMP 170HX 64GB · vllm 0.32.0rc1.dev249+g18305948a · FP8 static7 contexts × 8 concurrency levels
- Peak decode
- 559.6 tok/s
- Peak prefill
- 573.0 tok/s