Speed test matrix
Mistral-Medium-3.5-128B-NVFP4 on CMP 170HX 64GB ×2
vllm · 0.32.0rc1.dev249+g18305948a · NVFP4 mixed6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- GPU count
- 2
- VRAM
- 64 GB (2 × 32 GB)
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- NVFP4 mixed
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head. Layout: PP2 (stock vLLM flag) # 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve nvidia/Mistral-Medium-3.5-128B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context370.022,522 ms TTFT
16K context294.056,078 ms TTFT
32K context204.0161,324 ms TTFT
64K context124.0528,594 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 3.2Requests queued83,493 ms TTFT | 23.6Requests queued63,066 ms TTFT | 35.3236 ms TTFT | 69.3290 ms TTFT | 127.0345 ms TTFT | 234.7430 ms TTFT | Over capacity | Over capacity |
| 16K | 10.3Requests queued379 ms TTFT | 15.1499 ms TTFT | 27.9755 ms TTFT | 45.4928 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 8.7Requests queued593 ms TTFT | 13.2774 ms TTFT | 23.21,172 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 6.5Requests queued1,052 ms TTFT | 10.61,433 ms TTFT | 16.82,707 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 370.0 | — | 22,522 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 294.0 | — | 56,078 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 204.0 | — | 161,324 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 124.0 | — | 528,594 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Requests queued | 3.2 | 12.8 | 83,493 ms | 78.1 ms | 1.0 | 0 | 97 | 30.0 s |
| Sustained decode | 0 | 2 | Requests queued | 23.6 | 12.6 | 63,066 ms | 79.1 ms | 1.9 | 0 | 708 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 35.3 | 8.8 | 236 ms | 113.2 ms | 4.0 | 0 | 1,058 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 69.3 | 8.7 | 290 ms | 115.4 ms | 8.0 | 0 | 2,078 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 127.0 | 8.0 | 345 ms | 125.8 ms | 16.0 | 0 | 3,808 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 234.7 | 7.3 | 430 ms | 136.3 ms | 32.0 | 0 | 7,038 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Requests queued | 10.3 | 10.3 | 379 ms | 96.7 ms | 1.0 | 0 | 310 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 15.1 | 7.6 | 499 ms | 132.4 ms | 2.0 | 0 | 453 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 27.9 | 7.0 | 755 ms | 143.1 ms | 4.0 | 0 | 836 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 45.4 | 5.7 | 928 ms | 176.8 ms | 8.0 | 0 | 1,360 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Requests queued | 8.7 | 8.7 | 593 ms | 115.4 ms | 1.0 | 0 | 260 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 13.2 | 6.6 | 774 ms | 152.0 ms | 2.0 | 0 | 395 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 23.2 | 5.8 | 1,172 ms | 171.7 ms | 4.0 | 0 | 696 | 29.9 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Requests queued | 6.5 | 6.6 | 1,052 ms | 152.0 ms | 1.0 | 0 | 194 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 10.6 | 5.3 | 1,433 ms | 190.2 ms | 2.0 | 0 | 316 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 16.8 | 4.2 | 2,707 ms | 236.9 ms | 4.0 | 0 | 504 | 29.9 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Over capacity | — | — | — | — | 0.7 | 0 | 0 | 30.0 s |
| Sustained decode | 128K | 2 | Over capacity | — | — | — | — | 0.4 | 0 | 0 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Over capacity | — | — | — | — | 0.7 | 0 | 0 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.