Verified 3w ago Mainly llama.cpp 200 approved runs
Submissions
2,318
200 approved public runs
Models
16
unique models
Avg output
24.0tok/s
Best 103 tok/s
Avg prefill
72.7tok/s
Best 281 tok/s
Avg total
43.4tok/s
Best 215 tok/s
Avg TTFT
10,220ms
Best 1,110 ms
Hardware
2
distinct setups
Matrix runs
36
speed test matrices
Matrix runs36+
Matrix runs 36
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Showing the newest 10.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.62 VRAM_RESERVE_MIB=1024 MAX_CONTEXT=65536 CONVERSATION_CACHE_MIB=2048. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 53.6, 57, 59.2 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.62 VRAM_RESERVE_MIB=1024 MAX_CONTEXT=65536 CONVERSATION_CACHE_MIB=4096. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 55.2, 55.1, 59.8 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.58 VRAM_RESERVE_MIB=1024 MAX_CONTEXT=65536 CONVERSATION_CACHE_MIB=4096. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 55.3, 55.8, 54.8 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 11 tok/s is the aggregate decode rate across all 12 streams, about 0.9 tok/s per stream, and the 43803 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 11.9, 11, 11 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 7.7 tok/s is the aggregate decode rate across all 8 streams, about 1.0 tok/s per stream, and the 34223 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 7.7, 8.2, 8.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (lmx --concurrency 6); 7.7 tok/s is the aggregate decode rate across all 6 streams, about 1.3 tok/s per stream, and the 29991 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 7.7, 8.8, 7.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 6.5 tok/s is the aggregate decode rate across all 4 streams, about 1.6 tok/s per stream, and the 13547 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5.7, 6.5, 8.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (lmx --concurrency 3); 5.9 tok/s is the aggregate decode rate across all 3 streams, about 2.0 tok/s per stream, and the 8303 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5.9, 6.6, 5.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 5.9 tok/s is the aggregate decode rate across all 2 streams, about 3.0 tok/s per stream, and the 7053 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5.9, 5.8, 6.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 1.2 tok/s is the aggregate decode rate across all 2 streams, about 0.6 tok/s per stream, and the 4499 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 122 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47658 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/GLM-4.7-GGUF @ 70deda3fcbe622671ff2111d7c0c7392f618e3b0, file GLM-4.7-UD-IQ2_M-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 1, 1.2, 1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 8.6 tok/s is the aggregate decode rate across all 8 streams, about 1.1 tok/s per stream, and the 20749 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 9.9, 9.8, 8.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 6.7, 7.2, 7.1 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (lmx --concurrency 6); 5.5 tok/s is the aggregate decode rate across all 6 streams, about 0.9 tok/s per stream, and the 19310 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6.1, 5.5, 6.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 122 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47658 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/GLM-4.7-GGUF @ 70deda3fcbe622671ff2111d7c0c7392f618e3b0, file GLM-4.7-UD-IQ2_M-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 3.3, 3.1, 3.2 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 9.2 tok/s is the aggregate decode rate across all 8 streams, about 1.1 tok/s per stream, and the 9826 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 8.6, 9.2, 10.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 6.8 tok/s is the aggregate decode rate across all 4 streams, about 1.7 tok/s per stream, and the 12323 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6.8, 8.6, 8.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (lmx --concurrency 3); 5.4 tok/s is the aggregate decode rate across all 3 streams, about 1.8 tok/s per stream, and the 9864 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5, 5.6, 5.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (lmx --concurrency 6); 4.1 tok/s is the aggregate decode rate across all 6 streams, about 0.7 tok/s per stream, and the 5303 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 4.3, 11.3, 4.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 6.1 tok/s is the aggregate decode rate across all 2 streams, about 3.0 tok/s per stream, and the 6596 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6.7, 6.6, 6.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 7.1, 7, 7.1 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 16 requests sent at once (lmx --concurrency 16); 12.9 tok/s is the aggregate decode rate across all 16 streams, about 0.8 tok/s per stream, and the 46524 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 13.4, 12.9, 13.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 7.1 tok/s is the aggregate decode rate across all 4 streams, about 1.8 tok/s per stream, and the 4644 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 7.7, 7.1, 6.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (lmx --concurrency 3); 7.9 tok/s is the aggregate decode rate across all 3 streams, about 2.6 tok/s per stream, and the 4077 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 3.4, 9.4, 7.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 7.2 tok/s is the aggregate decode rate across all 12 streams, about 0.6 tok/s per stream, and the 45206 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 9.7, 7.2, 7.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 17.3, 15.6, 15.8 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.62 VRAM_RESERVE_MIB=1024 MAX_CONTEXT=65536 CONVERSATION_CACHE_MIB=2048. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 53.6, 57, 59.2 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.62 VRAM_RESERVE_MIB=1024 MAX_CONTEXT=65536 CONVERSATION_CACHE_MIB=4096. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 55.2, 55.1, 59.8 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. Engine: XeStrata xe0.1.40.2.2 (MistVVK/XeStrata, the Intel fork of Strata; LGPL-3.0) in the chriscorbell/intel-arc-pro-b70-qwen38-flash-next recipe image built from commit 6aad3506 (local image local/xestrata-qwen38:xe0.1.40.2.2, sha256:6f03dde70323). Weights: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f920; the GPU holds the dense layers and a cache of hot experts, the CPU computes the rest of the experts from system RAM at the same time. MTP speculative decoding (4 tokens), int8 KV cache. Settings: SPEC_MIN_P=0.58 VRAM_RESERVE_MIB=1024 MAX_CONTEXT=65536 CONVERSATION_CACHE_MIB=4096. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 55.3, 55.8, 54.8 tok/s; this is the median (tie to the earlier). Campaign ai/vm/b65-vm/2026-10-27-b65-vm-localmaxxing. Prompt/output samples were added from the lmx run file; engineTimingsRaw is XeStrata's own per-request log (decode tokens and tok/s, expert-cache hit rate), because lmx does not capture it for this engine.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 11 tok/s is the aggregate decode rate across all 12 streams, about 0.9 tok/s per stream, and the 43803 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 11.9, 11, 11 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 7.7 tok/s is the aggregate decode rate across all 8 streams, about 1.0 tok/s per stream, and the 34223 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 7.7, 8.2, 8.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (lmx --concurrency 6); 7.7 tok/s is the aggregate decode rate across all 6 streams, about 1.3 tok/s per stream, and the 29991 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 7.7, 8.8, 7.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 6.5 tok/s is the aggregate decode rate across all 4 streams, about 1.6 tok/s per stream, and the 13547 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5.7, 6.5, 8.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (lmx --concurrency 3); 5.9 tok/s is the aggregate decode rate across all 3 streams, about 2.0 tok/s per stream, and the 8303 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5.9, 6.6, 5.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 5.9 tok/s is the aggregate decode rate across all 2 streams, about 3.0 tok/s per stream, and the 7053 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5.9, 5.8, 6.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 1.2 tok/s is the aggregate decode rate across all 2 streams, about 0.6 tok/s per stream, and the 4499 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 122 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47658 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/GLM-4.7-GGUF @ 70deda3fcbe622671ff2111d7c0c7392f618e3b0, file GLM-4.7-UD-IQ2_M-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 1, 1.2, 1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 8.6 tok/s is the aggregate decode rate across all 8 streams, about 1.1 tok/s per stream, and the 20749 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 9.9, 9.8, 8.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 12 server slots of 4096 tokens each, context 49152. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 6.7, 7.2, 7.1 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (lmx --concurrency 6); 5.5 tok/s is the aggregate decode rate across all 6 streams, about 0.9 tok/s per stream, and the 19310 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6.1, 5.5, 6.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 122 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47658 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/GLM-4.7-GGUF @ 70deda3fcbe622671ff2111d7c0c7392f618e3b0, file GLM-4.7-UD-IQ2_M-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 3.3, 3.1, 3.2 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 8 requests sent at once (lmx --concurrency 8); 9.2 tok/s is the aggregate decode rate across all 8 streams, about 1.1 tok/s per stream, and the 9826 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 8.6, 9.2, 10.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 6.8 tok/s is the aggregate decode rate across all 4 streams, about 1.7 tok/s per stream, and the 12323 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6.8, 8.6, 8.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (lmx --concurrency 3); 5.4 tok/s is the aggregate decode rate across all 3 streams, about 1.8 tok/s per stream, and the 9864 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 5, 5.6, 5.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 6 requests sent at once (lmx --concurrency 6); 4.1 tok/s is the aggregate decode rate across all 6 streams, about 0.7 tok/s per stream, and the 5303 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 4.3, 11.3, 4.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 2 requests sent at once (lmx --concurrency 2); 6.1 tok/s is the aggregate decode rate across all 2 streams, about 3.0 tok/s per stream, and the 6596 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 6.7, 6.6, 6.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 7.1, 7, 7.1 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 16 requests sent at once (lmx --concurrency 16); 12.9 tok/s is the aggregate decode rate across all 16 streams, about 0.8 tok/s per stream, and the 46524 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 13.4, 12.9, 13.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 2 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 4 requests sent at once (lmx --concurrency 4); 7.1 tok/s is the aggregate decode rate across all 4 streams, about 1.8 tok/s per stream, and the 4644 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 7.7, 7.1, 6.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 3 requests sent at once (lmx --concurrency 3); 7.9 tok/s is the aggregate decode rate across all 3 streams, about 2.6 tok/s per stream, and the 4077 ms time to first token is under that load. KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 3.4, 9.4, 7.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1.
CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 12 requests sent at once (lmx --concurrency 12); 7.2 tok/s is the aggregate decode rate across all 12 streams, about 0.6 tok/s per stream, and the 45206 ms time to first token is under that load. KASA b65-vm: 1 x Intel Arc Pro B65 32 GB (PCIe passthrough, xe driver, Ubuntu 26.04 kernel 7.0) in a VM with 16 vCPU pinned to 8 cores (16 threads) of an AMD EPYC 7343 and 128 GiB. llama.cpp llama-server b11913 (image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 96 GB model does not fit the 32 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU; the exact split is not logged at the default verbosity, 8 generation threads, 16 batch threads, flash attention on, 16 server slots of 4096 tokens each, context 65536. GGUF unsloth/DeepSeek-V4-Flash-0731-GGUF @ fbbb5b93fb787c21338159b0af3318bb3f4d9768, file DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf. Each request: LocalMaxxing's canonical reasoning-v1 prompt with a unique nonce, 256 output tokens. One discarded warm-up batch, then 3 measured batches: 9.7, 7.2, 7.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Selection rule applied to every run in this round: the median of the 1 measured requests whose lmx promptTokens is a whole number; the others report a fractional median, which LocalMaxxing refuses.
KASA cuda-vm: 1 x NVIDIA RTX A6000 48 GB (PCIe passthrough, driver 615.71.10) in a VM with 16 vCPU pinned to 8 of the 16 cores (16 threads) of an AMD EPYC 7343 (another VM holds the other 8 cores) and 128 GiB. llama.cpp llama-server b11889 (CUDA image ghcr.io/ggml-org/llama.cpp), placement by llama.cpp's default --fit (1 GiB margin): the 120 GB model does not fit the 48 GB GPU, so llama.cpp's --fit kept the layers on the GPU and placed part of the MoE expert weights in system RAM on the CPU (loaded with -lm none, so the CPU part is in ordinary RAM, not memory-mapped); the exact split is not logged at the default verbosity; 47140 MiB of VRAM in use after load, 8 generation threads, 16 batch threads, flash attention on, 8 server slots of 4096 tokens each, context 32768. GGUF unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF @ 41032e5471dd6ea5b349062d978626215d6c5bba, file Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 17.3, 15.6, 15.8 tok/s; this is the median (tie to the earlier). Prompt and output samples were added from the lmx run file, engineTimingsRaw from exact llama-server /metrics deltas around the request.
