Speed test matrix
Qwen3.5-4B-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context8,430971 ms TTFT
16K context6,8532,367 ms TTFT
32K context4,9496,525 ms TTFT
64K context3,16520,353 ms TTFT
128K context1,85769,302 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 169.225 ms TTFT | 313.742 ms TTFT | 587.354 ms TTFT | 1,031.9102 ms TTFT | 1,712.3148 ms TTFT | 2,487.6246 ms TTFT | 3,276.2478 ms TTFT | 3,858.9773 ms TTFT |
| 16K | 138.0394 ms TTFT | 249.7586 ms TTFT | 400.9964 ms TTFT | 630.81,726 ms TTFT | 894.83,251 ms TTFT | 1,093.96,373 ms TTFT | 1,244.012,894 ms TTFT | Over capacity |
| 32K | 116.4531 ms TTFT | 208.6781 ms TTFT | 300.21,290 ms TTFT | 444.82,281 ms TTFT | 593.04,294 ms TTFT | 689.58,414 ms TTFT | Over capacity | Over capacity |
| 64K | 88.7602 ms TTFT | 155.6854 ms TTFT | 202.31,369 ms TTFT | 281.02,418 ms TTFT | 354.94,532 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 60.01,859 ms TTFT | 102.52,698 ms TTFT | 121.94,406 ms TTFT | 161.07,904 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 34.4Requests queued2,964 ms TTFT | 62.34,245 ms TTFT | 70.06,882 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,430.0 | — | 971 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,853.0 | — | 2,367 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,949.0 | — | 6,525 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,165.0 | — | 20,353 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,857.0 | — | 69,302 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 169.2 | 170.5 | 25 ms | 5.9 ms | 1.0 | 0 | 5,075 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 313.7 | 158.2 | 42 ms | 6.3 ms | 2.0 | 0 | 9,408 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 587.3 | 148.6 | 54 ms | 6.7 ms | 4.0 | 0 | 17,616 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,031.9 | 131.1 | 102 ms | 7.6 ms | 8.0 | 0 | 30,952 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,712.3 | 109.5 | 148 ms | 9.1 ms | 16.0 | 0 | 51,360 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,487.6 | 79.6 | 246 ms | 12.6 ms | 32.0 | 0 | 74,624 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 3,276.2 | 52.0 | 478 ms | 19.2 ms | 64.0 | 0 | 98,240 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,858.9 | 30.2 | 773 ms | 33.1 ms | 128.0 | 0 | 115,712 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 138.0 | 138.5 | 394 ms | 7.2 ms | 1.0 | 0 | 4,141 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 249.7 | 124.7 | 586 ms | 8.0 ms | 2.0 | 0 | 7,490 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 400.9 | 99.1 | 964 ms | 10.1 ms | 4.0 | 0 | 12,025 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 630.8 | 76.3 | 1,726 ms | 13.1 ms | 8.0 | 0 | 18,920 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 894.8 | 52.0 | 3,251 ms | 19.2 ms | 16.0 | 0 | 26,832 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,093.9 | 29.6 | 6,373 ms | 33.8 ms | 32.0 | 0 | 32,800 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,244.0 | 15.1 | 12,894 ms | 66.2 ms | 64.0 | 0 | 37,312 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 116.4 | 116.7 | 531 ms | 8.6 ms | 1.0 | 0 | 3,492 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 208.6 | 103.9 | 781 ms | 9.6 ms | 2.0 | 0 | 6,258 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 300.2 | 73.7 | 1,290 ms | 13.6 ms | 4.0 | 0 | 9,004 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 444.8 | 53.2 | 2,281 ms | 18.8 ms | 8.0 | 0 | 13,341 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 593.0 | 33.7 | 4,294 ms | 29.7 ms | 16.0 | 0 | 17,790 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 689.5 | 17.9 | 8,414 ms | 55.9 ms | 32.0 | 0 | 20,672 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 88.7 | 88.8 | 602 ms | 11.3 ms | 1.0 | 0 | 2,659 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 155.6 | 77.4 | 854 ms | 12.9 ms | 2.0 | 0 | 4,668 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 202.3 | 49.6 | 1,369 ms | 20.2 ms | 4.0 | 0 | 6,068 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 281.0 | 33.6 | 2,418 ms | 29.8 ms | 8.0 | 0 | 8,424 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 354.9 | 20.1 | 4,532 ms | 49.7 ms | 16.0 | 0 | 10,640 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 60.0 | 60.1 | 1,859 ms | 16.6 ms | 1.0 | 0 | 1,801 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 102.5 | 50.1 | 2,698 ms | 20.0 ms | 2.0 | 0 | 3,074 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 121.9 | 28.4 | 4,406 ms | 35.2 ms | 4.0 | 0 | 3,656 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 161.0 | 17.2 | 7,904 ms | 58.2 ms | 8.0 | 0 | 4,824 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 34.4 | 37.4 | 2,964 ms | 26.7 ms | 0.9 | 0 | 1,031 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 62.3 | 30.1 | 4,245 ms | 33.3 ms | 2.0 | 0 | 1,868 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 70.0 | 15.8 | 6,882 ms | 63.5 ms | 4.0 | 0 | 2,100 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.