Speed test matrix
Qwen3.8-27B on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · INT46 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- INT4
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- qwen38-27b
- Revision
- 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q38s-i-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q38s-i-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.8-27B --host 0.0.0.0 --served-model-name qwen38-27b --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context3,7482,187 ms TTFT
32K context3,4809,242 ms TTFT
64K context3,07920,827 ms TTFT
128K context2,50851,061 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 50.487 ms TTFT | 129.8139 ms TTFT | 253.4195 ms TTFT | 468.2301 ms TTFT | 458.1507 ms TTFT |
| 8K | 49.4114 ms TTFT | 125.9157 ms TTFT | 239.6265 ms TTFT | 417.1379 ms TTFT | 422.8669 ms TTFT |
| 32K | 46.2261 ms TTFT | 108.7Requests queued345 ms TTFT | 197.7Requests queued675 ms TTFT | 310.6Requests queued846 ms TTFT | 332.82,131 ms TTFT |
| 64K | 43.5378 ms TTFT | 96.3Requests queued515 ms TTFT | 172.2624 ms TTFT | 270.9Requests queued1,006 ms TTFT | 286.21,009 ms TTFT |
| 128K | 37.8695 ms TTFT | 76.0Requests queued911 ms TTFT | 121.8Requests queued1,561 ms TTFT | 181.8Requests queued2,092 ms TTFT | Over capacity |
| 200K | 33.81,003 ms TTFT | 63.3Requests queued1,283 ms TTFT | 97.9Requests queued1,905 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,748.0 | — | 2,187 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,480.0 | — | 9,242 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,079.0 | — | 20,827 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,508.0 | — | 51,061 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 50.4 | 50.9 | 87 ms | 19.6 ms | 1.0 | 6 | 1,511 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 129.8 | 66.8 | 139 ms | 15.0 ms | 1.9 | 16 | 3,892 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 253.4 | 66.1 | 195 ms | 15.1 ms | 4.0 | 32 | 7,600 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 468.2 | 62.1 | 301 ms | 16.1 ms | 7.8 | 64 | 14,022 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 458.1 | 30.1 | 507 ms | 33.3 ms | 16.0 | 48 | 13,744 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 49.4 | 50.4 | 114 ms | 19.8 ms | 1.0 | 6 | 1,483 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 125.9 | 65.1 | 157 ms | 15.4 ms | 2.0 | 16 | 3,776 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 239.6 | 63.6 | 265 ms | 15.7 ms | 3.8 | 32 | 7,168 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 417.1 | 56.6 | 379 ms | 17.7 ms | 7.7 | 56 | 12,472 | 29.9 s |
| Sustained decode | 8K | 16 | Valid | 422.8 | 28.2 | 669 ms | 35.5 ms | 15.6 | 48 | 12,682 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 46.2 | 48.5 | 261 ms | 20.6 ms | 0.9 | 6 | 1,385 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 108.7 | 58.9 | 345 ms | 17.0 ms | 1.9 | 14 | 3,258 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 197.7 | 56.9 | 675 ms | 17.6 ms | 3.7 | 24 | 5,930 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 310.6 | 45.3 | 846 ms | 22.1 ms | 7.3 | 40 | 9,312 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 332.8 | 24.1 | 2,131 ms | 41.4 ms | 15.5 | 32 | 9,984 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 43.5 | 46.3 | 378 ms | 21.6 ms | 0.9 | 5 | 1,305 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 96.3 | 53.3 | 515 ms | 18.8 ms | 1.8 | 12 | 2,888 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 172.2 | 47.8 | 624 ms | 20.9 ms | 4.0 | 20 | 5,164 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 270.9 | 39.0 | 1,006 ms | 25.7 ms | 7.5 | 32 | 8,128 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 286.2 | 18.9 | 1,009 ms | 52.8 ms | 15.3 | 32 | 8,585 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 37.8 | 42.5 | 695 ms | 23.6 ms | 0.9 | 5 | 1,132 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 76.0 | 44.8 | 911 ms | 22.4 ms | 1.9 | 10 | 2,278 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 121.8 | 38.5 | 1,561 ms | 26.1 ms | 3.5 | 16 | 3,652 | 30.0 s |
| Sustained decode | 128K | 8 | Requests queued | 181.8 | 26.9 | 2,092 ms | 37.2 ms | 7.0 | 24 | 5,352 | 29.4 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 33.8 | 38.9 | 1,003 ms | 25.7 ms | 0.9 | 4 | 1,014 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 63.3 | 38.2 | 1,283 ms | 26.2 ms | 1.8 | 8 | 1,899 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 97.9 | 29.1 | 1,905 ms | 34.3 ms | 3.5 | 12 | 2,936 | 30.0 s |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.