Speed test matrix
NVIDIA-Nemotron-3-Nano-4B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context10.4K806 ms TTFT
16K context9,9191,665 ms TTFT
32K context9,5703,434 ms TTFT
64K context8,5227,696 ms TTFT
128K context6,84919,125 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 169.383 ms TTFT | Looped | 646.2640 ms TTFT | 1,175.2210 ms TTFT | 1,909.6304 ms TTFT | 2,747.5450 ms TTFT | 3,363.0586 ms TTFT | 3,664.8939 ms TTFT |
| 16K | 171.3235 ms TTFT | 347.4257 ms TTFT | 614.9403 ms TTFT | 1,026.1620 ms TTFT | 1,530.81,076 ms TTFT | 2,028.42,008 ms TTFT | 2,379.73,832 ms TTFT | 2,539.58,259 ms TTFT |
| 32K | 165.1377 ms TTFT | 331.5516 ms TTFT | 562.6727 ms TTFT | 901.01,162 ms TTFT | 1,268.31,925 ms TTFT | 1,600.63,722 ms TTFT | 1,839.27,150 ms TTFT | 1,944.014,891 ms TTFT |
| 64K | 168.5529 ms TTFT | 303.5671 ms TTFT | 490.8984 ms TTFT | 723.71,329 ms TTFT | 943.82,106 ms TTFT | 1,136.13,698 ms TTFT | 1,274.28,714 ms TTFT | Over capacity |
| 128K | 157.8783 ms TTFT | 264.41,049 ms TTFT | 385.81,525 ms TTFT | 518.92,509 ms TTFT | 619.04,279 ms TTFT | 712.57,811 ms TTFT | Over capacity | Over capacity |
| 248K | 135.01,991 ms TTFT | 208.92,479 ms TTFT | 273.33,443 ms TTFT | 334.95,445 ms TTFT | 369.89,484 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.4K | — | 806 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,919.0 | — | 1,665 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,570.0 | — | 3,434 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,522.0 | — | 7,696 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,849.0 | — | 19,125 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 169.3 | 169.7 | 83 ms | 5.9 ms | 1.0 | 0 | 5,077 | 30.0 s |
| Sustained decode | 0 | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 0 | 4 | Valid | 646.2 | 161.3 | 640 ms | 6.2 ms | 4.0 | 0 | 19,400 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,175.2 | 147.1 | 210 ms | 6.8 ms | 8.0 | 0 | 35,256 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,909.6 | 120.1 | 304 ms | 8.3 ms | 16.0 | 0 | 57,296 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,747.5 | 86.4 | 450 ms | 11.6 ms | 32.0 | 0 | 82,426 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 3,363.0 | 52.6 | 586 ms | 19.0 ms | 64.0 | 0 | 100,897 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,664.8 | 28.4 | 939 ms | 35.2 ms | 128.0 | 0 | 109,952 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 171.3 | 171.3 | 235 ms | 5.8 ms | 1.0 | 0 | 5,138 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 347.4 | 173.3 | 257 ms | 5.8 ms | 2.0 | 0 | 10,428 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 614.9 | 153.6 | 403 ms | 6.5 ms | 4.0 | 0 | 18,448 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,026.1 | 127.7 | 620 ms | 7.8 ms | 8.0 | 0 | 30,784 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,530.8 | 94.3 | 1,076 ms | 10.6 ms | 16.0 | 0 | 45,920 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,028.4 | 61.4 | 2,008 ms | 16.3 ms | 32.0 | 0 | 60,864 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,379.7 | 34.9 | 3,832 ms | 28.6 ms | 64.0 | 0 | 71,424 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,539.5 | 18.0 | 8,259 ms | 55.5 ms | 128.0 | 0 | 76,160 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 165.1 | 165.1 | 377 ms | 6.1 ms | 1.0 | 0 | 4,951 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 331.5 | 165.4 | 516 ms | 6.0 ms | 2.0 | 0 | 9,942 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 562.6 | 139.8 | 727 ms | 7.2 ms | 4.0 | 0 | 16,877 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 901.0 | 110.9 | 1,162 ms | 9.0 ms | 8.0 | 0 | 27,024 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,268.3 | 76.6 | 1,925 ms | 13.1 ms | 16.0 | 0 | 38,048 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,600.6 | 46.6 | 3,722 ms | 21.4 ms | 32.0 | 0 | 48,032 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,839.2 | 25.2 | 7,150 ms | 39.7 ms | 64.0 | 0 | 55,168 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 1,944.0 | 12.4 | 14,891 ms | 80.6 ms | 128.0 | 0 | 58,496 | 30.1 s |
| Sustained decode | 64K | 1 | Valid | 168.5 | 168.7 | 529 ms | 5.9 ms | 1.0 | 0 | 5,054 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 303.5 | 151.4 | 671 ms | 6.6 ms | 2.0 | 0 | 9,108 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 490.8 | 122.3 | 984 ms | 8.2 ms | 4.0 | 0 | 14,738 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 723.7 | 89.3 | 1,329 ms | 11.2 ms | 8.0 | 0 | 21,712 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 943.8 | 57.1 | 2,106 ms | 17.5 ms | 16.0 | 0 | 28,304 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,136.1 | 33.3 | 3,698 ms | 30.0 ms | 32.0 | 0 | 34,080 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,274.2 | 18.0 | 8,714 ms | 55.4 ms | 64.0 | 0 | 38,208 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 157.8 | 158.6 | 783 ms | 6.3 ms | 1.0 | 0 | 4,734 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 264.4 | 132.9 | 1,049 ms | 7.5 ms | 2.0 | 0 | 7,934 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 385.8 | 97.2 | 1,525 ms | 10.3 ms | 4.0 | 0 | 11,584 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 518.9 | 65.2 | 2,509 ms | 15.3 ms | 8.0 | 0 | 15,568 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 619.0 | 38.8 | 4,279 ms | 25.8 ms | 16.0 | 0 | 18,576 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 712.5 | 22.6 | 7,811 ms | 44.2 ms | 32.0 | 0 | 21,376 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 135.0 | 136.0 | 1,991 ms | 7.4 ms | 1.0 | 0 | 4,048 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 208.9 | 104.8 | 2,479 ms | 9.5 ms | 2.0 | 0 | 6,266 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 273.3 | 67.6 | 3,443 ms | 14.8 ms | 4.0 | 0 | 8,200 | 30.0 s |
| Sustained decode | 248K | 8 | Valid | 334.9 | 40.3 | 5,445 ms | 24.8 ms | 8.0 | 0 | 10,048 | 30.0 s |
| Sustained decode | 248K | 16 | Valid | 369.8 | 21.7 | 9,484 ms | 46.1 ms | 16.0 | 0 | 11,088 | 30.0 s |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.