Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · NVFP48 contexts (0–1016K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --max-model-len 1048576 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context8,465987 ms TTFT
16K context8,0212,059 ms TTFT
32K context7,7654,233 ms TTFT
64K context6,9799,397 ms TTFT
128K context5,69523,001 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 210.8101 ms TTFT | 338.2723 ms TTFT | 536.6183 ms TTFT | 825.6284 ms TTFT | 1,173.2321 ms TTFT | 1,678.9524 ms TTFT | 2,328.3778 ms TTFT | 2,954.91,122 ms TTFT |
| 16K | 208.0592 ms TTFT | 316.3848 ms TTFT | Looped | 770.22,302 ms TTFT | 1,096.44,258 ms TTFT | 1,556.08,178 ms TTFT | 2,073.216,059 ms TTFT | 2,082.5Requests queued32,528 ms TTFT |
| 32K | 206.4706 ms TTFT | Looped | Looped | 746.12,508 ms TTFT | 1,039.64,582 ms TTFT | 1,451.28,648 ms TTFT | 1,884.217,013 ms TTFT | 1,623.8Requests queued34,229 ms TTFT |
| 64K | 201.8839 ms TTFT | 301.21,152 ms TTFT | 468.81,682 ms TTFT | 702.42,752 ms TTFT | 939.44,872 ms TTFT | 1,268.19,227 ms TTFT | 1,589.718,087 ms TTFT | 1,854.136,445 ms TTFT |
| 128K | 195.01,013 ms TTFT | 288.61,314 ms TTFT | 430.01,759 ms TTFT | 621.12,641 ms TTFT | 799.54,341 ms TTFT | 1,025.37,705 ms TTFT | 1,220.814,910 ms TTFT | Over capacity |
| 256K | 179.52,334 ms TTFT | 260.43,136 ms TTFT | 370.24,248 ms TTFT | 507.96,608 ms TTFT | 610.311,086 ms TTFT | 733.520,602 ms TTFT | Over capacity | Over capacity |
| 512K | 142.9Requests queued3,670 ms TTFT | Looped | 289.06,551 ms TTFT | 368.110,355 ms TTFT | 411.017,624 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1016K | Over capacity | Over capacity | 198.216,678 ms TTFT | 236.023,181 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,465.0 | — | 987 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,021.0 | — | 2,059 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,765.0 | — | 4,233 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,979.0 | — | 9,397 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,695.0 | — | 23,001 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 210.8 | 210.0 | 101 ms | 4.8 ms | 1.0 | 0 | 6,323 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 338.2 | 169.1 | 723 ms | 5.9 ms | 2.0 | 0 | 10,145 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 536.6 | 134.7 | 183 ms | 7.4 ms | 4.0 | 0 | 16,096 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 825.6 | 104.4 | 284 ms | 9.6 ms | 8.0 | 0 | 24,784 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,173.2 | 75.0 | 321 ms | 13.3 ms | 16.0 | 0 | 35,200 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,678.9 | 53.8 | 524 ms | 18.6 ms | 32.0 | 0 | 50,368 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,328.3 | 37.1 | 778 ms | 27.0 ms | 64.0 | 0 | 69,824 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,954.9 | 23.2 | 1,122 ms | 43.1 ms | 128.0 | 0 | 88,704 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 208.0 | 207.4 | 592 ms | 4.8 ms | 1.0 | 0 | 6,239 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 316.3 | 157.5 | 848 ms | 6.3 ms | 2.0 | 0 | 9,490 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 770.2 | 92.3 | 2,302 ms | 10.8 ms | 8.0 | 0 | 23,104 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,096.4 | 62.7 | 4,258 ms | 15.9 ms | 16.0 | 0 | 32,896 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,556.0 | 40.6 | 8,178 ms | 24.6 ms | 32.0 | 0 | 46,688 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,073.2 | 23.4 | 16,059 ms | 42.8 ms | 64.0 | 0 | 62,208 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 2,082.5 | 10.5 | 32,528 ms | 95.6 ms | 126.6 | 0 | 62,426 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 206.4 | 206.7 | 706 ms | 4.8 ms | 1.0 | 0 | 6,188 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 746.1 | 89.3 | 2,508 ms | 11.2 ms | 8.0 | 0 | 22,400 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,039.6 | 59.1 | 4,582 ms | 16.9 ms | 16.0 | 0 | 31,184 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,451.2 | 37.6 | 8,648 ms | 26.6 ms | 32.0 | 0 | 43,520 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,884.2 | 20.9 | 17,013 ms | 47.8 ms | 64.0 | 0 | 56,551 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,623.8 | 8.6 | 34,229 ms | 116.6 ms | 124.7 | 0 | 48,682 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 201.8 | 201.9 | 839 ms | 5.0 ms | 1.0 | 0 | 6,055 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 301.2 | 150.1 | 1,152 ms | 6.7 ms | 2.0 | 0 | 9,036 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 468.8 | 115.2 | 1,682 ms | 8.7 ms | 4.0 | 0 | 14,064 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 702.4 | 84.1 | 2,752 ms | 11.9 ms | 8.0 | 0 | 21,088 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 939.4 | 53.4 | 4,872 ms | 18.7 ms | 16.0 | 0 | 28,176 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,268.1 | 32.6 | 9,227 ms | 30.7 ms | 32.0 | 0 | 38,048 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,589.7 | 17.6 | 18,087 ms | 57.0 ms | 64.0 | 0 | 47,680 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 1,854.1 | 8.5 | 36,445 ms | 117.4 ms | 128.0 | 0 | 55,552 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 195.0 | 195.5 | 1,013 ms | 5.1 ms | 1.0 | 0 | 5,849 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 288.6 | 144.2 | 1,314 ms | 6.9 ms | 2.0 | 0 | 8,658 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 430.0 | 106.4 | 1,759 ms | 9.4 ms | 4.0 | 0 | 12,904 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 621.1 | 75.4 | 2,641 ms | 13.3 ms | 8.0 | 0 | 18,632 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 799.5 | 46.7 | 4,341 ms | 21.4 ms | 16.0 | 0 | 23,985 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,025.3 | 27.9 | 7,705 ms | 35.9 ms | 32.0 | 0 | 30,752 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,220.8 | 14.7 | 14,910 ms | 67.9 ms | 64.0 | 0 | 36,608 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 179.5 | 181.2 | 2,334 ms | 5.5 ms | 1.0 | 0 | 5,385 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 260.4 | 129.4 | 3,136 ms | 7.7 ms | 2.0 | 0 | 7,811 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 370.2 | 89.2 | 4,248 ms | 11.2 ms | 4.0 | 0 | 11,104 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 507.9 | 57.7 | 6,608 ms | 17.3 ms | 8.0 | 0 | 15,232 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 610.3 | 31.1 | 11,086 ms | 32.1 ms | 16.0 | 0 | 18,304 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 733.5 | 15.7 | 20,602 ms | 63.8 ms | 32.0 | 0 | 21,984 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Requests queued | 142.9 | 159.7 | 3,670 ms | 6.3 ms | 0.9 | 0 | 4,287 | 30.0 s |
| Sustained decode | 512K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 4 | Valid | 289.0 | 70.8 | 6,551 ms | 14.1 ms | 4.0 | 0 | 8,668 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 368.1 | 43.3 | 10,355 ms | 23.1 ms | 8.0 | 0 | 11,040 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 411.0 | 22.9 | 17,624 ms | 43.6 ms | 16.0 | 0 | 12,320 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 1 | Over capacity | — | — | — | — | 0.8 | 0 | 0 | 30.0 s |
| Sustained decode | 1016K | 2 | Over capacity | — | — | — | — | 1.4 | 0 | 0 | 30.0 s |
| Sustained decode | 1016K | 4 | Valid | 198.2 | 44.9 | 16,678 ms | 22.3 ms | 4.0 | 0 | 5,944 | 30.0 s |
| Sustained decode | 1016K | 8 | Valid | 236.0 | 23.2 | 23,181 ms | 43.1 ms | 8.0 | 0 | 7,072 | 30.0 s |
| Sustained decode | 1016K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.