Verified 1w ago Mainly llama.cpp 200 approved runs
Submissions
1,123
200 approved public runs
Models
24
unique models
Avg output
333tok/s
Best 3,216 tok/s
Avg prefill
372tok/s
Best 3,953 tok/s
Avg total
688tok/s
Best 6,953 tok/s
Avg TTFT
5,197ms
Best 77.2 ms
Hardware
2
distinct setups
Matrix runs
36
speed test matrices
Matrix runs36+
Matrix runs 36
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Showing the newest 10.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 416.3 tok/s is the aggregate decode rate across all 8 streams, about 52.0 tok/s per stream, and the 1242 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 404.3, 416.3, 427.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 205.3 tok/s is the aggregate decode rate across all 2 streams, about 102.7 tok/s per stream, and the 380 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 206.6, 201.6, 205.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 326.8 tok/s is the aggregate decode rate across all 12 streams, about 27.2 tok/s per stream, and the 1997 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 512.9, 326.8, 326.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 111.1, 110.4, 109.2 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 16 requests sent at once (lmx --concurrency 16); 15.3 tok/s is the aggregate decode rate across all 16 streams, about 1.0 tok/s per stream, and the 16559 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 118 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47518 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF ggml-org/Step-3.5-Flash-GGUF @ c439ad64d667632f3e1507757cbe225b6a46e6b8, file Step-3.5-Flash-Q4_K.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 15.3, 17.8, 21.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 88 tok/s is the aggregate decode rate across all 8 streams, about 11.0 tok/s per stream, and the 11380 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 112.3, 88, 112.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 59.1 tok/s is the aggregate decode rate across all 2 streams, about 29.6 tok/s per stream, and the 1006 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 59.1, 59.3, 59.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 218.9 tok/s is the aggregate decode rate across all 12 streams, about 18.2 tok/s per stream, and the 12798 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 218.9, 101.5, 219.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 32.1, 31.8, 31.7 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 48 requests sent at once (batch 48, lmx --concurrency 48); 1569.6 tok/s is the aggregate decode rate across all 48 streams, about 32.7 tok/s per stream, and the 988 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (297 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 48: 1569.6, 1572, 1555.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 8.7 tok/s is the aggregate decode rate across all 12 streams, about 0.7 tok/s per stream, and the 11488 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 118 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47518 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF ggml-org/Step-3.5-Flash-GGUF @ c439ad64d667632f3e1507757cbe225b6a46e6b8, file Step-3.5-Flash-Q4_K.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 8.7, 13.5, 6.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: VRAM_RESERVE_MIB=1024. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 38.2, 40.7, 35.4 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.8. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 32.6, 40.7, 48.3 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.6. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 37.2, 44, 50.2 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: PCIE_FRAC=0.9. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 27.8, 41.3, 38.1 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: PCIE_FRAC=0.5. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 34.6, 34.4, 29.7 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 30 tok/s is the aggregate decode rate across all 4 streams, about 7.5 tok/s per stream. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: recipe defaults. Canonical reasoning-v1 prompt with a unique nonce; one discarded warm-up batch, then 3 measured batches: 29.4, 30, 30.9 tok/s; this is the median. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 37.1 tok/s is the aggregate decode rate across all 2 streams, about 18.6 tok/s per stream. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: recipe defaults. Canonical reasoning-v1 prompt with a unique nonce; one discarded warm-up batch, then 3 measured batches: 40.1, 43.4, 37.1 tok/s; this is the median. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: recipe defaults. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 34.3, 40.4, 43.2 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (batch 3, lmx --concurrency 3); 275.2 tok/s is the aggregate decode rate across all 3 streams, about 91.7 tok/s per stream, and the 143 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (297 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 3: 268.1, 292.9, 275.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (batch 24, lmx --concurrency 24); 1123.7 tok/s is the aggregate decode rate across all 24 streams, about 46.8 tok/s per stream, and the 515 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 24: 1099.4, 1123.7, 1138.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (batch 12, lmx --concurrency 12); 692.9 tok/s is the aggregate decode rate across all 12 streams, about 57.7 tok/s per stream, and the 286 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 12: 687.4, 720.1, 692.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 96 requests sent at once (batch 96, lmx --concurrency 96); 1820.2 tok/s is the aggregate decode rate across all 96 streams, about 19.0 tok/s per stream, and the 1769 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 96: 1820.2, 1811.3, 1827.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3-30b-2507-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (batch 6, lmx --concurrency 6); 450.5 tok/s is the aggregate decode rate across all 6 streams, about 75.1 tok/s per stream, and the 232 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (295 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 6: 486.8, 467.1, 450.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3-30b-2507-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 48 requests sent at once (batch 48, lmx --concurrency 48); 1442.3 tok/s is the aggregate decode rate across all 48 streams, about 30.0 tok/s per stream, and the 961 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 48: 1447.6, 1452, 1442.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3-30b-2507-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 416.3 tok/s is the aggregate decode rate across all 8 streams, about 52.0 tok/s per stream, and the 1242 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 404.3, 416.3, 427.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 205.3 tok/s is the aggregate decode rate across all 2 streams, about 102.7 tok/s per stream, and the 380 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 206.6, 201.6, 205.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 326.8 tok/s is the aggregate decode rate across all 12 streams, about 27.2 tok/s per stream, and the 1997 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 512.9, 326.8, 326.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 111.1, 110.4, 109.2 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 16 requests sent at once (lmx --concurrency 16); 15.3 tok/s is the aggregate decode rate across all 16 streams, about 1.0 tok/s per stream, and the 16559 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 118 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47518 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF ggml-org/Step-3.5-Flash-GGUF @ c439ad64d667632f3e1507757cbe225b6a46e6b8, file Step-3.5-Flash-Q4_K.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 15.3, 17.8, 21.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 88 tok/s is the aggregate decode rate across all 8 streams, about 11.0 tok/s per stream, and the 11380 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 112.3, 88, 112.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 59.1 tok/s is the aggregate decode rate across all 2 streams, about 29.6 tok/s per stream, and the 1006 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 59.1, 59.3, 59.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 218.9 tok/s is the aggregate decode rate across all 12 streams, about 18.2 tok/s per stream, and the 12798 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 218.9, 101.5, 219.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-31b-q4kxl-np16 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-31B/gemma-4-31B-it-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 65536 -np 16 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-31b-q4kxl-np16
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/gemma-4-31B-it-GGUF @ c1ac76e99d5513b141e8adde7288b85c3f9c32ec, file gemma-4-31B-it-UD-Q4_K_XL.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 32.1, 31.8, 31.7 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 48 requests sent at once (batch 48, lmx --concurrency 48); 1569.6 tok/s is the aggregate decode rate across all 48 streams, about 32.7 tok/s per stream, and the 988 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (297 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 48: 1569.6, 1572, 1555.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 8.7 tok/s is the aggregate decode rate across all 12 streams, about 0.7 tok/s per stream, and the 11488 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 118 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47518 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF ggml-org/Step-3.5-Flash-GGUF @ c439ad64d667632f3e1507757cbe225b6a46e6b8, file Step-3.5-Flash-Q4_K.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 8.7, 13.5, 6.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: VRAM_RESERVE_MIB=1024. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 38.2, 40.7, 35.4 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.8. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 32.6, 40.7, 48.3 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.6. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 37.2, 44, 50.2 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: PCIE_FRAC=0.9. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 27.8, 41.3, 38.1 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: PCIE_FRAC=0.5. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 34.6, 34.4, 29.7 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 30 tok/s is the aggregate decode rate across all 4 streams, about 7.5 tok/s per stream. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: recipe defaults. Canonical reasoning-v1 prompt with a unique nonce; one discarded warm-up batch, then 3 measured batches: 29.4, 30, 30.9 tok/s; this is the median. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 37.1 tok/s is the aggregate decode rate across all 2 streams, about 18.6 tok/s per stream. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: recipe defaults. Canonical reasoning-v1 prompt with a unique nonce; one discarded warm-up batch, then 3 measured batches: 40.1, 43.4, 37.1 tok/s; this is the median. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: recipe defaults. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 34.3, 40.4, 43.2 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (batch 3, lmx --concurrency 3); 275.2 tok/s is the aggregate decode rate across all 3 streams, about 91.7 tok/s per stream, and the 143 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (297 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 3: 268.1, 292.9, 275.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (batch 24, lmx --concurrency 24); 1123.7 tok/s is the aggregate decode rate across all 24 streams, about 46.8 tok/s per stream, and the 515 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 24: 1099.4, 1123.7, 1138.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (batch 12, lmx --concurrency 12); 692.9 tok/s is the aggregate decode rate across all 12 streams, about 57.7 tok/s per stream, and the 286 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 12: 687.4, 720.1, 692.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 96 requests sent at once (batch 96, lmx --concurrency 96); 1820.2 tok/s is the aggregate decode rate across all 96 streams, about 19.0 tok/s per stream, and the 1769 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 96: 1820.2, 1811.3, 1827.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3-30b-2507-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (batch 6, lmx --concurrency 6); 450.5 tok/s is the aggregate decode rate across all 6 streams, about 75.1 tok/s per stream, and the 232 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (295 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 6: 486.8, 467.1, 450.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3-30b-2507-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 48 requests sent at once (batch 48, lmx --concurrency 48); 1442.3 tok/s is the aggregate decode rate across all 48 streams, about 30.0 tok/s per stream, and the 961 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 48: 1447.6, 1452, 1442.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3-30b-2507-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
