Speed test matrix
Qwen3.6-35B-A3B on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · INT46 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- INT4
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- qwen36-35b
- Revision
- 995ad96eacd98c81ed38be0c5b274b04031597b0
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36s2-i-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36s2-i-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-35B-A3B --host 0.0.0.0 --served-model-name qwen36-35b --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context12.3K665 ms TTFT
32K context11.6K2,795 ms TTFT
64K context9,8766,524 ms TTFT
128K context7,57217,000 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 98.681 ms TTFT | 187.7120 ms TTFT | 352.8146 ms TTFT | 631.6Requests queued215 ms TTFT | 833.4145 ms TTFT |
| 8K | 96.0105 ms TTFT | 180.4Requests queued158 ms TTFT | 339.3185 ms TTFT | 581.5Requests queued197 ms TTFT | 739.3249 ms TTFT |
| 32K | 89.3170 ms TTFT | 163.4241 ms TTFT | 290.8Requests queued309 ms TTFT | 477.9481 ms TTFT | 600.1524 ms TTFT |
| 64K | 82.3271 ms TTFT | 145.5Requests queued348 ms TTFT | 240.8Requests queued507 ms TTFT | 363.1Requests queued767 ms TTFT | 460.5Requests queued883 ms TTFT |
| 128K | 69.8517 ms TTFT | 116.7Requests queued682 ms TTFT | 181.2Requests queued853 ms TTFT | 254.0Requests queued1,438 ms TTFT | 325.3Requests queued1,809 ms TTFT |
| 200K | 58.8867 ms TTFT | 90.8Requests queued1,113 ms TTFT | 146.1Requests queued1,540 ms TTFT | 185.6Requests queued2,301 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.3K | — | 665 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.6K | — | 2,795 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,876.0 | — | 6,524 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,572.0 | — | 17,000 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 98.6 | 101.4 | 81 ms | 9.9 ms | 0.9 | 13 | 2,959 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 187.7 | 97.5 | 120 ms | 10.3 ms | 1.9 | 25 | 5,630 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 352.8 | 92.9 | 146 ms | 10.8 ms | 3.9 | 48 | 10,584 | 30.0 s |
| Sustained decode | 0 | 8 | Requests queued | 631.6 | 84.0 | 215 ms | 11.9 ms | 7.6 | 80 | 18,948 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 833.4 | 53.4 | 145 ms | 18.7 ms | 15.7 | 112 | 25,001 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 96.0 | 99.5 | 105 ms | 10.0 ms | 1.0 | 12 | 2,881 | 30.0 s |
| Sustained decode | 8K | 2 | Requests queued | 180.4 | 95.4 | 158 ms | 10.5 ms | 1.9 | 24 | 5,414 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 339.3 | 90.0 | 185 ms | 11.1 ms | 4.0 | 44 | 10,176 | 30.0 s |
| Sustained decode | 8K | 8 | Requests queued | 581.5 | 77.1 | 197 ms | 13.0 ms | 7.6 | 72 | 17,442 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 739.3 | 48.7 | 249 ms | 20.5 ms | 15.3 | 96 | 22,171 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 89.3 | 94.4 | 170 ms | 10.6 ms | 0.9 | 11 | 2,680 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 163.4 | 87.4 | 241 ms | 11.4 ms | 1.9 | 20 | 4,900 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 290.8 | 78.8 | 309 ms | 12.7 ms | 3.6 | 36 | 8,724 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 477.9 | 66.7 | 481 ms | 15.0 ms | 7.6 | 64 | 14,334 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 600.1 | 40.2 | 524 ms | 24.9 ms | 15.4 | 65 | 17,990 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 82.3 | 89.4 | 271 ms | 11.2 ms | 0.9 | 10 | 2,469 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 145.5 | 79.8 | 348 ms | 12.5 ms | 1.7 | 20 | 4,362 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 240.8 | 67.5 | 507 ms | 14.8 ms | 3.6 | 32 | 7,226 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 363.1 | 52.7 | 767 ms | 19.0 ms | 7.2 | 48 | 10,888 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 460.5 | 31.2 | 883 ms | 32.0 ms | 15.1 | 48 | 13,810 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 69.8 | 80.8 | 517 ms | 12.4 ms | 0.9 | 9 | 2,059 | 29.5 s |
| Sustained decode | 128K | 2 | Requests queued | 116.7 | 68.3 | 682 ms | 14.7 ms | 1.7 | 14 | 3,498 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 181.2 | 53.8 | 853 ms | 18.6 ms | 3.5 | 24 | 5,388 | 29.7 s |
| Sustained decode | 128K | 8 | Requests queued | 254.0 | 38.2 | 1,438 ms | 26.2 ms | 7.0 | 32 | 7,619 | 30.0 s |
| Sustained decode | 128K | 16 | Requests queued | 325.3 | 22.3 | 1,809 ms | 44.9 ms | 14.5 | 32 | 9,755 | 30.0 s |
| Sustained decode | 200K | 1 | Valid | 58.8 | 72.8 | 867 ms | 13.7 ms | 0.8 | 7 | 1,763 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 90.8 | 58.4 | 1,113 ms | 17.2 ms | 1.7 | 12 | 2,725 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 146.1 | 44.8 | 1,540 ms | 22.3 ms | 3.3 | 16 | 4,382 | 30.0 s |
| Sustained decode | 200K | 8 | Requests queued | 185.6 | 29.3 | 2,301 ms | 34.1 ms | 6.7 | 24 | 5,563 | 30.0 s |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.