Speed test matrix
NVIDIA-Nemotron-3-Nano-4B-FP8 on RTX 3090 24GB
vllm · 0.32.0 · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context10.8K781 ms TTFT
16K context10.5K1,586 ms TTFT
32K context9,7793,383 ms TTFT
64K context8,6567,623 ms TTFT
128K context7,03318,743 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 156.522 ms TTFT | 302.0180 ms TTFT | 561.350 ms TTFT | 989.684 ms TTFT | 1,610.2123 ms TTFT | 2,318.4226 ms TTFT | 2,918.5456 ms TTFT | 3,175.4687 ms TTFT |
| 16K | 152.5125 ms TTFT | 294.1179 ms TTFT | 537.6292 ms TTFT | 896.1513 ms TTFT | 1,369.6970 ms TTFT | 1,837.41,932 ms TTFT | 2,234.44,026 ms TTFT | Over capacity |
| 32K | 148.9249 ms TTFT | 286.1362 ms TTFT | 512.2585 ms TTFT | 813.51,023 ms TTFT | 1,178.41,933 ms TTFT | 1,509.43,812 ms TTFT | Over capacity | Over capacity |
| 64K | 142.3315 ms TTFT | 271.2446 ms TTFT | 466.6686 ms TTFT | 681.91,184 ms TTFT | 922.22,200 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 130.6363 ms TTFT | 244.7481 ms TTFT | 393.5689 ms TTFT | 512.01,123 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 113.3846 ms TTFT | 206.41,040 ms TTFT | 301.11,501 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.8K | — | 781 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.5K | — | 1,586 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,779.0 | — | 3,383 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,656.0 | — | 7,623 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,033.0 | — | 18,743 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 156.5 | 156.6 | 22 ms | 6.4 ms | 1.0 | 0 | 4,694 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 302.0 | 151.2 | 180 ms | 6.6 ms | 2.0 | 0 | 9,060 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 561.3 | 140.6 | 50 ms | 7.1 ms | 4.0 | 0 | 16,840 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 989.6 | 124.1 | 84 ms | 8.1 ms | 8.0 | 0 | 29,688 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,610.2 | 101.0 | 123 ms | 9.9 ms | 16.0 | 0 | 48,305 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,318.4 | 72.7 | 226 ms | 13.8 ms | 32.0 | 0 | 69,536 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,918.5 | 45.6 | 456 ms | 21.9 ms | 64.0 | 0 | 87,552 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,175.4 | 24.5 | 687 ms | 40.7 ms | 128.0 | 0 | 95,232 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 152.5 | 152.6 | 125 ms | 6.6 ms | 1.0 | 0 | 4,575 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 294.1 | 147.0 | 179 ms | 6.8 ms | 2.0 | 0 | 8,822 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 537.6 | 134.0 | 292 ms | 7.5 ms | 4.0 | 0 | 16,127 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 896.1 | 111.1 | 513 ms | 9.0 ms | 8.0 | 0 | 26,880 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,369.6 | 84.0 | 970 ms | 11.9 ms | 16.0 | 0 | 41,089 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,837.4 | 55.2 | 1,932 ms | 18.1 ms | 32.0 | 0 | 55,104 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,234.4 | 32.4 | 4,026 ms | 30.9 ms | 64.0 | 0 | 67,008 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 148.9 | 149.1 | 249 ms | 6.7 ms | 1.0 | 0 | 4,468 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 286.1 | 142.7 | 362 ms | 7.0 ms | 2.0 | 0 | 8,582 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 512.2 | 127.0 | 585 ms | 7.9 ms | 4.0 | 0 | 15,364 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 813.5 | 99.7 | 1,023 ms | 10.0 ms | 8.0 | 0 | 24,400 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,178.4 | 70.6 | 1,933 ms | 14.2 ms | 16.0 | 0 | 35,344 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,509.4 | 43.3 | 3,812 ms | 23.1 ms | 32.0 | 0 | 45,280 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 142.3 | 142.5 | 315 ms | 7.0 ms | 1.0 | 0 | 4,270 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 271.2 | 135.3 | 446 ms | 7.4 ms | 2.0 | 0 | 8,136 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 466.6 | 115.6 | 686 ms | 8.7 ms | 4.0 | 0 | 13,996 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 681.9 | 83.4 | 1,184 ms | 12.0 ms | 8.0 | 0 | 20,456 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 922.2 | 54.9 | 2,200 ms | 18.2 ms | 16.0 | 0 | 27,664 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 130.6 | 130.8 | 363 ms | 7.6 ms | 1.0 | 0 | 3,919 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 244.7 | 122.2 | 481 ms | 8.2 ms | 2.0 | 0 | 7,340 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 393.5 | 97.7 | 689 ms | 10.2 ms | 4.0 | 0 | 11,804 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 512.0 | 62.9 | 1,123 ms | 15.9 ms | 8.0 | 0 | 15,360 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 113.3 | 113.5 | 846 ms | 8.8 ms | 1.0 | 0 | 3,399 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 206.4 | 102.7 | 1,040 ms | 9.7 ms | 2.0 | 0 | 6,190 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 301.1 | 74.1 | 1,501 ms | 13.5 ms | 4.0 | 0 | 9,032 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.