Verified 1w ago Mainly llama.cpp 200 approved runs
Submissions
1,149
200 approved public runs
Models
28
unique models
Avg output
286tok/s
Best 3,216 tok/s
Avg prefill
333tok/s
Best 3,953 tok/s
Avg total
587tok/s
Best 6,953 tok/s
Avg TTFT
5,470ms
Best 77.2 ms
Hardware
2
distinct setups
Matrix runs
36
speed test matrices
Matrix runs36+
Matrix runs 36
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Showing the newest 10.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 1.9 tok/s is the aggregate decode rate across all 2 streams, about 0.9 tok/s per stream, and the 5892 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 78 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-Flash-Next-GGUF @ 766911a6b7369840a91dbcd95f9f997acaab6cd6, file Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 2.4, 1.9, 2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (batch 6, lmx --concurrency 6); 86.4 tok/s is the aggregate decode rate across all 6 streams, about 14.4 tok/s per stream, and the 2503 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (317 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 6: 86.4, 86.2, 83.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (batch 3, lmx --concurrency 3); 40.5 tok/s is the aggregate decode rate across all 3 streams, about 13.5 tok/s per stream, and the 1278 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (319 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 3: 38.7, 42, 40.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (batch 24, lmx --concurrency 24); 161.8 tok/s is the aggregate decode rate across all 24 streams, about 6.7 tok/s per stream, and the 6530 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (316 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 24: 168, 161.8, 159.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 5.3 tok/s is the aggregate decode rate across all 8 streams, about 0.7 tok/s per stream, and the 12889 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 71 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47142 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF @ 99144d1d7a318f045f1412a85bbb807f3c8a17a0, file NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Q4_0.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6, 5.1, 5.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (batch 12, lmx --concurrency 12); 124.6 tok/s is the aggregate decode rate across all 12 streams, about 10.4 tok/s per stream, and the 3810 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (318 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 12: 132.8, 129, 124.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 96 requests sent at once (batch 96, lmx --concurrency 96); 1900.6 tok/s is the aggregate decode rate across all 96 streams, about 19.8 tok/s per stream, and the 1823 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 96: 1894.1, 1900.6, 1915 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (batch 6, lmx --concurrency 6); 459.8 tok/s is the aggregate decode rate across all 6 streams, about 76.6 tok/s per stream, and the 196 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (295 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 6: 481.6, 459.8, 475.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 16 tok/s is the aggregate decode rate across all 4 streams, about 4.0 tok/s per stream, and the 4017 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 71 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47142 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF @ 99144d1d7a318f045f1412a85bbb807f3c8a17a0, file NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Q4_0.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 17.3, 16.3, 16 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 71 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47142 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF @ 99144d1d7a318f045f1412a85bbb807f3c8a17a0, file NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Q4_0.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 26.8, 27.3, 26.1 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. MTP speculative decoding from mtp-Qwen3.8-27B-Q4_0.gguf, up to 3 draft tokens. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-Q4_K_XL.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 37.6, 38.4, 37.9 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 47.5 tok/s is the aggregate decode rate across all 8 streams, about 5.9 tok/s per stream, and the 11560 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 47.4, 47.6, 47.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 23.7 tok/s is the aggregate decode rate across all 2 streams, about 11.8 tok/s per stream, and the 2952 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 23.7, 23.7, 23.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-IQ4_XS.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 14.7, 14.7, 14.7 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 78 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-Flash-Next-GGUF @ 766911a6b7369840a91dbcd95f9f997acaab6cd6, file Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 9.8, 7.8, 7.7 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 333 tok/s is the aggregate decode rate across all 8 streams, about 41.6 tok/s per stream, and the 790 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 333, 337.1, 335.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 273.7 tok/s is the aggregate decode rate across all 4 streams, about 68.4 tok/s per stream, and the 445 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 256.2, 273.7, 276.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 32 requests sent at once (lmx --concurrency 32); 532.8 tok/s is the aggregate decode rate across all 32 streams, about 16.6 tok/s per stream, and the 2428 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 532.8, 537.3, 526.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (lmx --concurrency 24); 355.6 tok/s is the aggregate decode rate across all 24 streams, about 14.8 tok/s per stream, and the 1650 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 359.1, 489.7, 355.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 204.4 tok/s is the aggregate decode rate across all 2 streams, about 102.2 tok/s per stream, and the 228 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 208.3, 225.7, 204.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 271.3 tok/s is the aggregate decode rate across all 12 streams, about 22.6 tok/s per stream, and the 907 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 396.3, 271.3, 272.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 134.7, 133.8, 133.2 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 32 requests sent at once (lmx --concurrency 32); 675.3 tok/s is the aggregate decode rate across all 32 streams, about 21.1 tok/s per stream, and the 4471 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 653.7, 702.5, 675.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (lmx --concurrency 24); 448.4 tok/s is the aggregate decode rate across all 24 streams, about 18.7 tok/s per stream, and the 3371 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 448.4, 696, 454.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 16 requests sent at once (lmx --concurrency 16); 374.8 tok/s is the aggregate decode rate across all 16 streams, about 23.4 tok/s per stream, and the 2402 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 388, 374.8, 372.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 1.9 tok/s is the aggregate decode rate across all 2 streams, about 0.9 tok/s per stream, and the 5892 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 78 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-Flash-Next-GGUF @ 766911a6b7369840a91dbcd95f9f997acaab6cd6, file Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 2.4, 1.9, 2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (batch 6, lmx --concurrency 6); 86.4 tok/s is the aggregate decode rate across all 6 streams, about 14.4 tok/s per stream, and the 2503 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (317 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 6: 86.4, 86.2, 83.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (batch 3, lmx --concurrency 3); 40.5 tok/s is the aggregate decode rate across all 3 streams, about 13.5 tok/s per stream, and the 1278 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (319 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 3: 38.7, 42, 40.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (batch 24, lmx --concurrency 24); 161.8 tok/s is the aggregate decode rate across all 24 streams, about 6.7 tok/s per stream, and the 6530 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (316 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 24: 168, 161.8, 159.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 5.3 tok/s is the aggregate decode rate across all 8 streams, about 0.7 tok/s per stream, and the 12889 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 71 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47142 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF @ 99144d1d7a318f045f1412a85bbb807f3c8a17a0, file NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Q4_0.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6, 5.1, 5.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (batch 12, lmx --concurrency 12); 124.6 tok/s is the aggregate decode rate across all 12 streams, about 10.4 tok/s per stream, and the 3810 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs 32, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (318 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 12: 132.8, 129, 124.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq72-awq-eager, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 96 requests sent at once (batch 96, lmx --concurrency 96); 1900.6 tok/s is the aggregate decode rate across all 96 streams, about 19.8 tok/s per stream, and the 1823 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 96: 1894.1, 1900.6, 1915 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (batch 6, lmx --concurrency 6); 459.8 tok/s is the aggregate decode rate across all 6 streams, about 76.6 tok/s per stream, and the 196 ms time to first token is under that load. Setup: KASA cuda-vm, 1 x NVIDIA RTX A6000 48 GB passed through to a VM (28 vCPU of an AMD EPYC 7343, 128 GiB), one card; vLLM 0.32.0 (vllm/vllm-openai@sha256:befccf0e7e4089aa23926bb6f887a85c0d46c2a99e6de5c5ffdf5cb7957c5b34); max_num_seqs ?, max_model_len 32768; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (295 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 6: 481.6, 459.8, 475.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config vq3c-30b-fp8, kasa-testing campaign ai/vm/cuda-vm/2026-10-26-a6000-localmaxxing. Selection declared before measuring: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional mean, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 16 tok/s is the aggregate decode rate across all 4 streams, about 4.0 tok/s per stream, and the 4017 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 71 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47142 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF @ 99144d1d7a318f045f1412a85bbb807f3c8a17a0, file NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Q4_0.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 17.3, 16.3, 16 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 71 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47142 MiB of VRAM in use after load, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF @ 99144d1d7a318f045f1412a85bbb807f3c8a17a0, file NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Q4_0.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 26.8, 27.3, 26.1 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. MTP speculative decoding from mtp-Qwen3.8-27B-Q4_0.gguf, up to 3 draft tokens. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-Q4_K_XL.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 37.6, 38.4, 37.9 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 47.5 tok/s is the aggregate decode rate across all 8 streams, about 5.9 tok/s per stream, and the 11560 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 47.4, 47.6, 47.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 23.7 tok/s is the aggregate decode rate across all 2 streams, about 11.8 tok/s per stream, and the 2952 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 23.7, 23.7, 23.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-27B-GGUF @ 4ca720788d1e01f1bff70c033e0d0028fd02e502, file Qwen3.8-27B-UD-IQ4_XS.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 14.7, 14.7, 14.7 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 78 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 16 generation threads, 28 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Qwen3.8-Flash-Next-GGUF @ 766911a6b7369840a91dbcd95f9f997acaab6cd6, file Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 9.8, 7.8, 7.7 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 333 tok/s is the aggregate decode rate across all 8 streams, about 41.6 tok/s per stream, and the 790 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 333, 337.1, 335.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 273.7 tok/s is the aggregate decode rate across all 4 streams, about 68.4 tok/s per stream, and the 445 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 256.2, 273.7, 276.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 32 requests sent at once (lmx --concurrency 32); 532.8 tok/s is the aggregate decode rate across all 32 streams, about 16.6 tok/s per stream, and the 2428 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 532.8, 537.3, 526.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (lmx --concurrency 24); 355.6 tok/s is the aggregate decode rate across all 24 streams, about 14.8 tok/s per stream, and the 1650 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 359.1, 489.7, 355.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 204.4 tok/s is the aggregate decode rate across all 2 streams, about 102.2 tok/s per stream, and the 228 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 208.3, 225.7, 204.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 271.3 tok/s is the aggregate decode rate across all 12 streams, about 22.6 tok/s per stream, and the 907 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 396.3, 271.3, 272.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-glm47f-q4kxl-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/GLM-4.7-Flash/GLM-4.7-Flash-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-glm47f-q4kxl-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/GLM-4.7-Flash-GGUF @ 0d32489ecb9db6d2a4fc93bd27ef01519f95474d, file GLM-4.7-Flash-UD-Q4_K_XL.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 134.7, 133.8, 133.2 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 32 requests sent at once (lmx --concurrency 32); 675.3 tok/s is the aggregate decode rate across all 32 streams, about 21.1 tok/s per stream, and the 4471 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 653.7, 702.5, 675.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 24 requests sent at once (lmx --concurrency 24); 448.4 tok/s is the aggregate decode rate across all 24 streams, about 18.7 tok/s per stream, and the 3371 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 448.4, 696, 454.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
docker run -d --name lsv-g4-26b-iq4xs-np32 --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v /mnt/models:/mnt/models -v /mnt/models/cache:/root/.cache -e HF_HOME=/mnt/models/hf ghcr.io/ggml-org/llama.cpp@sha256:b51c252687249a6990ae1c168156cb215c954bfd71f9c434eda9a065b1a40dbe -m /mnt/models/gguf/gemma-4-26B-A4B/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf --host 0.0.0.0 --port 8000 -fa on -c 131072 -np 32 --jinja -t 16 -tb 28 --metrics -ngl 99 -fit off
Extra flags
--d
--name lsv-g4-26b-iq4xs-np32
--gpus all
--ipc host
--p 127.0.0.1:8000:8000
--e HF_HOME=/mnt/models/hf
--jinja
--tb 28
--metrics
--fit off
Notes
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 16 requests sent at once (lmx --concurrency 16); 374.8 tok/s is the aggregate decode rate across all 16 streams, about 23.4 tok/s per stream, and the 2402 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 28 vCPU of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), all layers on the GPU (-ngl 99), 16 generation threads, 28 batch threads, flash attention on, 32 server slots of 4096 tokens each, context 131072. GGUF unsloth/gemma-4-26B-A4B-it-GGUF @ c099eb48e663fd284577b04978a94ffccb261841, file gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 388, 374.8, 372.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
