Speed test matrix
gemma-4-26B-A4B-it on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · BF164 contexts (0–64K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- gemma-4
- Revision
- 4d7ae4984b7db7de8f8457170b3f1a419ee76d52
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-gemb-u-tp4-ctx131k --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/gemb-u-tp4-ctx131k/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/gemma-4-26B-A4B-it --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context2,2193,696 ms TTFT
32K context679.047,628 ms TTFT
64K context341.0189,226 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 78.931 ms TTFT | 141.957 ms TTFT | 244.473 ms TTFT | 411.872 ms TTFT | 674.397 ms TTFT |
| 8K | 51.0112 ms TTFT | 91.8171 ms TTFT | 161.0200 ms TTFT | 266.4218 ms TTFT | 386.6296 ms TTFT |
| 32K | 32.7359 ms TTFT | 56.0Requests queued474 ms TTFT | 96.5721 ms TTFT | 164.3887 ms TTFT | 217.21,201 ms TTFT |
| 64K | 25.9Requests queued682 ms TTFT | 41.3893 ms TTFT | 68.41,377 ms TTFT | 112.91,834 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,219.0 | — | 3,696 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 679.0 | — | 47,628 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 341.0 | — | 189,226 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 78.9 | 79.3 | 31 ms | 12.6 ms | 1.0 | 10 | 2,366 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 141.9 | 71.7 | 57 ms | 13.9 ms | 2.0 | 18 | 4,256 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 244.4 | 61.9 | 73 ms | 16.2 ms | 4.0 | 32 | 7,328 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 411.8 | 52.0 | 72 ms | 19.2 ms | 8.0 | 56 | 12,352 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 674.3 | 42.7 | 97 ms | 23.4 ms | 16.0 | 80 | 20,229 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 51.0 | 52.0 | 112 ms | 19.2 ms | 1.0 | 6 | 1,530 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 91.8 | 47.3 | 171 ms | 21.2 ms | 2.0 | 12 | 2,754 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 161.0 | 41.5 | 200 ms | 24.1 ms | 4.0 | 20 | 4,828 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 266.4 | 34.2 | 218 ms | 29.2 ms | 7.8 | 32 | 7,987 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 386.6 | 24.9 | 296 ms | 40.1 ms | 15.6 | 48 | 11,590 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 32.7 | 34.2 | 359 ms | 29.2 ms | 0.9 | 4 | 981 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 56.0 | 29.2 | 474 ms | 34.2 ms | 1.9 | 6 | 1,680 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 96.5 | 25.9 | 721 ms | 38.7 ms | 4.0 | 12 | 2,892 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 164.3 | 21.7 | 887 ms | 46.1 ms | 7.7 | 16 | 4,928 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 217.2 | 13.9 | 1,201 ms | 71.9 ms | 15.8 | 16 | 6,511 | 30.0 s |
| Sustained decode | 64K | 1 | Requests queued | 25.9 | 27.8 | 682 ms | 36.0 ms | 0.9 | 3 | 776 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 41.3 | 21.9 | 893 ms | 45.6 ms | 1.9 | 4 | 1,238 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 68.4 | 18.8 | 1,377 ms | 53.3 ms | 4.0 | 8 | 2,052 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 112.9 | 14.9 | 1,834 ms | 67.1 ms | 7.9 | 8 | 3,384 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.