Speed test matrix
Qwen3.5-9B-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · W4A16 g326 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- W4A16 g32
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve cyankiwi/Qwen3.5-9B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context5,4891,492 ms TTFT
16K context4,7683,402 ms TTFT
32K context3,7588,592 ms TTFT
64K context2,64024,397 ms TTFT
128K context1,67177,010 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 109.931 ms TTFT | 205.447 ms TTFT | 395.276 ms TTFT | 757.4143 ms TTFT | 1,290.9224 ms TTFT | 2,009.2380 ms TTFT | 2,643.1764 ms TTFT | 3,107.61,171 ms TTFT |
| 16K | 95.2518 ms TTFT | 175.1768 ms TTFT | 298.91,268 ms TTFT | 510.12,278 ms TTFT | 753.24,337 ms TTFT | 977.88,469 ms TTFT | 1,127.417,011 ms TTFT | Over capacity |
| 32K | 84.3643 ms TTFT | 153.4943 ms TTFT | 238.11,556 ms TTFT | 380.92,777 ms TTFT | 528.05,229 ms TTFT | 643.710,200 ms TTFT | Over capacity | Over capacity |
| 64K | 68.6662 ms TTFT | 122.4949 ms TTFT | 172.01,541 ms TTFT | 254.62,720 ms TTFT | 332.35,106 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 50.11,965 ms TTFT | 87.22,885 ms TTFT | 110.14,715 ms TTFT | 152.68,477 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 30.5Requests queued3,047 ms TTFT | 56.64,381 ms TTFT | 66.17,077 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,489.0 | — | 1,492 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,768.0 | — | 3,402 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,758.0 | — | 8,592 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,640.0 | — | 24,397 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,671.0 | — | 77,010 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 109.9 | 110.3 | 31 ms | 9.1 ms | 1.0 | 0 | 3,298 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 205.4 | 103.1 | 47 ms | 9.7 ms | 2.0 | 0 | 6,162 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 395.2 | 99.3 | 76 ms | 10.1 ms | 4.0 | 0 | 11,852 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 757.4 | 95.5 | 143 ms | 10.5 ms | 8.0 | 0 | 22,721 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,290.9 | 81.6 | 224 ms | 12.2 ms | 16.0 | 0 | 38,720 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,009.2 | 63.6 | 380 ms | 15.7 ms | 32.0 | 0 | 60,256 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,643.1 | 41.6 | 764 ms | 24.0 ms | 64.0 | 0 | 79,296 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,107.6 | 24.0 | 1,171 ms | 41.6 ms | 128.0 | 0 | 93,184 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 95.2 | 95.4 | 518 ms | 10.5 ms | 1.0 | 0 | 2,856 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 175.1 | 87.1 | 768 ms | 11.5 ms | 2.0 | 0 | 5,254 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 298.9 | 73.3 | 1,268 ms | 13.6 ms | 4.0 | 0 | 8,968 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 510.1 | 60.9 | 2,278 ms | 16.4 ms | 8.0 | 0 | 15,296 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 753.2 | 42.7 | 4,337 ms | 23.4 ms | 16.0 | 0 | 22,592 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 977.8 | 25.3 | 8,469 ms | 39.5 ms | 32.0 | 0 | 29,312 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,127.4 | 12.7 | 17,011 ms | 78.8 ms | 64.0 | 0 | 33,792 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 84.3 | 84.4 | 643 ms | 11.8 ms | 1.0 | 0 | 2,528 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 153.4 | 76.2 | 943 ms | 13.1 ms | 2.0 | 0 | 4,602 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 238.1 | 58.1 | 1,556 ms | 17.2 ms | 4.0 | 0 | 7,144 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 380.9 | 45.0 | 2,777 ms | 22.2 ms | 8.0 | 0 | 11,424 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 528.0 | 29.4 | 5,229 ms | 34.0 ms | 16.0 | 0 | 15,840 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 643.7 | 16.1 | 10,200 ms | 62.3 ms | 32.0 | 0 | 19,296 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 68.6 | 68.7 | 662 ms | 14.6 ms | 1.0 | 0 | 2,057 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 122.4 | 60.8 | 949 ms | 16.4 ms | 2.0 | 0 | 3,672 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 172.0 | 42.0 | 1,541 ms | 23.8 ms | 4.0 | 0 | 5,160 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 254.6 | 30.2 | 2,720 ms | 33.1 ms | 8.0 | 0 | 7,632 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 332.3 | 18.6 | 5,106 ms | 53.7 ms | 16.0 | 0 | 9,968 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 50.1 | 50.2 | 1,965 ms | 19.9 ms | 1.0 | 0 | 1,503 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 87.2 | 42.6 | 2,885 ms | 23.5 ms | 2.0 | 0 | 2,616 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 110.1 | 25.5 | 4,715 ms | 39.2 ms | 4.0 | 0 | 3,300 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 152.6 | 16.1 | 8,477 ms | 62.0 ms | 8.0 | 0 | 4,576 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 30.5 | 33.3 | 3,047 ms | 30.0 ms | 0.9 | 0 | 915 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 56.6 | 27.3 | 4,381 ms | 36.6 ms | 2.0 | 0 | 1,698 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 66.1 | 14.8 | 7,077 ms | 67.5 ms | 4.0 | 0 | 1,980 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.