A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests.
Strata (Niko1221/Strata v0.1.40.4) with its Tenstorrent P150 engine (LottoLottoLotto/Strata tt-p150 @ b6326de8: ttnn + custom Tensix kernels), serving ISTA-DASLab's unmodified GSQ-RCO IQ3_S GGUF, measured through the substrate gateway (OpenAI API, streaming). Dense layers, router, LM head and 8,823 of the 24,576 routed experts (GPTQ 4-bit block floats) run on the P150; the other experts run on the CPU with Strata's own IQ3_S kernels (Ryzen 9 9950X, DDR5). Traced decode with MTP drafts from the model's own MTP layer (verify windows of 4 rows), greedy. Canonical prompt reasoning-v1 (348 tokens with the chat template), 1024 output tokens, median of 3 timed runs after 1 warmup. Draft counters are per-request means over the 3 timed runs (the CLI summed all 3: 3534 drafted / 1891 accepted); engineTimingsRaw holds the last run's own counters (1131 / 646). TTFT includes the prompt read (~2.1 s at ~160 tok/s on this short prompt; 2K prompts read at ~320 tok/s, 8K at ~220 tok/s with the pipelined prompt read).
Strata (Niko1221/Strata v0.1.40.4) with its experimental Tenstorrent P150 engine (tt/, ttnn + custom Tensix kernels), serving ISTA-DASLab's unmodified GSQ-RCO IQ3_S GGUF. Dense layers, router, LM head and 8,823 of the 24,576 routed experts (min-MSE 4-bit block floats, chosen by routing counts measured on the card) run on the P150; the other experts run on the CPU with Strata's own IQ3_S kernels (Ryzen 9 9950X, dual-channel DDR5, ~46 GB/s). Traced decode, MTP drafts from the model's own MTP layer, greedy. Card in a PCIe Gen4 x16 slot. Prompt 2048 tokens + chat template = ~2090, past the model's 2050-position dense/sparse attention boundary. Draft acceptance 724/1653 (44%), 2.30 tokens per verify window on this prose prompt (chat/code prompts: 67-77% acceptance, 33-43 tok/s).
Peak custom-prompt measurement, not comparable to the official default-prompt result. Single Tenstorrent P150; public mixed BFP4/BFP8 target; dual-NoC runtime image sha256:9e2edc5d4c7a4e5e284090a3acaf61f6eb91438d666f6584ace6caa6d4266f14; native DFlash2 B8 single branch using z-lab/Qwen3.8-27B-DFlash2 revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4; greedy; two warmups and five measured 128-token requests; LocalMaxxing leading cache-bust nonce retained.
LocalMaxxing reasoning-v1 canonical prompt, plain /v1/completions SSE, temperature 0, 2 warmups then 5 measured sequential requests; median-throughput iteration submitted with its own timings and acceptance counters. 512 output tokens each. Decode=(output_tokens-1)/(last_text-first_text). Prefix caching disabled. One P150 32GB; custom TT-Metal runtime with packed BFP4/BF16 weights, fused GDN/SwiGLU/state commit, sharded norms, and MTP-1. TT-native cache package, not standalone CUDA/Transformers quant; pinned original checkpoint is also required. Experimental: sharded norms change numerics and prior long-prompt repeated greedy outputs varied. No peak VRAM or power measurement claimed. Reproduction and limitations: https://huggingface.co/Lottolabs/Qwen3.5-9B-TT-BFP4-P150/blob/main/manifest.json
Tenstorrent P150, Qwen3.5-9B bfp_bf4, vLLM concurrency 1. Fastest deployed MTP-1 path: one speculative draft token per cycle (specNumTokens=1), verified with the committed token in one M=2 target-model batch. One untimed warmup plus five timed 256-token streamed requests. Aggregate server counters across all six requests: 1,136 drafted verification tokens and 394 accepted speculative tokens (34.683% acceptance); 1,136 + 394 = 1,530 MTP-produced tokens, plus one initial prefill token per request = 1,536 total output tokens. Endpoint usage reported 538 prompt tokens per request; submitted speed metrics are CLI medians over the five timed requests.
Fastest verified SGLang setup for Qwen3.8-27B W4A16 AutoRound on 2x RTX 3090 with TP2. Uses DFlash speculative decoding with a local W8A16 DFlash2 drafter, 6 draft tokens, and a 40,960-token output-derived curated candidate vocabulary distributed across TP ranks. Candidate selection uses rank-local projection/top-k with compact cross-rank communication. Target execution uses torch.compile, CUDA graphs, selective SM86 W4A8 Marlin MLP kernels, BF16 KV/recurrent state, compiler-safe DFlash causal-convolution verification, and persisted tuned TorchInductor/Triton caches. Measurement is concurrency 1 with 2 warmups and 5 timed 512-token iterations. CUDA peer access and NVLink are unavailable on this PCIe topology, so TP collectives use NCCL's host-mediated path.
Q8_0 on pinned llama.cpp PR 26608 commit d8d862521, CUDA, 2x RTX 3090, all layers GPU-offloaded with 1:1 layer split, 131072 server context, flash attention. Prefill and decode are llama-bench pp2048/tg256 means over 5 repetitions after default warmup: 9243.21531 prompt tok/s and 232.774459 output tok/s. TTFT is median of 5 streaming endpoint requests after 2 warmups with prompt caching explicitly disabled: 47.77 ms (range 47.71-47.97 ms). The endpoint TTFT prompt was 42 actual tokens. Remote semantic decode was 224.4 tok/s median; llama-bench sustained decode is submitted as tokSOut.
Qwen3.5-9B fine-tune/distillation served from the named Jackrong GGUF repository; this is not a DeepSeek-V4-Flash architecture model. Remote OpenAI-compatible speed test: 1 warmup plus 5 measured 256-token generations.
A/B-tuned for 2x RTX 3090 with a usable 32K context: layer split, flash attention on, 16 CPU threads, batch 512, micro-batch 512, F16 K/V cache, automatic GPU-layer fitting with 256 MiB margin per GPU and 32768-token fit context. Five repetitions; 2048-token prompt processing and 256-token generation tests.
Strata (Niko1221/Strata v0.1.40.4) with its Tenstorrent P150 engine (LottoLottoLotto/Strata tt-p150 @ b6326de8: ttnn + custom Tensix kernels), serving ISTA-DASLab's unmodified GSQ-RCO IQ3_S GGUF, measured through the substrate gateway (OpenAI API, streaming). Dense layers, router, LM head and 8,823 of the 24,576 routed experts (GPTQ 4-bit block floats) run on the P150; the other experts run on the CPU with Strata's own IQ3_S kernels (Ryzen 9 9950X, DDR5). Traced decode with MTP drafts from the model's own MTP layer (verify windows of 4 rows), greedy. Canonical prompt reasoning-v1 (348 tokens with the chat template), 1024 output tokens, median of 3 timed runs after 1 warmup. Draft counters are per-request means over the 3 timed runs (the CLI summed all 3: 3534 drafted / 1891 accepted); engineTimingsRaw holds the last run's own counters (1131 / 646). TTFT includes the prompt read (~2.1 s at ~160 tok/s on this short prompt; 2K prompts read at ~320 tok/s, 8K at ~220 tok/s with the pipelined prompt read).
Strata (Niko1221/Strata v0.1.40.4) with its experimental Tenstorrent P150 engine (tt/, ttnn + custom Tensix kernels), serving ISTA-DASLab's unmodified GSQ-RCO IQ3_S GGUF. Dense layers, router, LM head and 8,823 of the 24,576 routed experts (min-MSE 4-bit block floats, chosen by routing counts measured on the card) run on the P150; the other experts run on the CPU with Strata's own IQ3_S kernels (Ryzen 9 9950X, dual-channel DDR5, ~46 GB/s). Traced decode, MTP drafts from the model's own MTP layer, greedy. Card in a PCIe Gen4 x16 slot. Prompt 2048 tokens + chat template = ~2090, past the model's 2050-position dense/sparse attention boundary. Draft acceptance 724/1653 (44%), 2.30 tokens per verify window on this prose prompt (chat/code prompts: 67-77% acceptance, 33-43 tok/s).
Peak custom-prompt measurement, not comparable to the official default-prompt result. Single Tenstorrent P150; public mixed BFP4/BFP8 target; dual-NoC runtime image sha256:9e2edc5d4c7a4e5e284090a3acaf61f6eb91438d666f6584ace6caa6d4266f14; native DFlash2 B8 single branch using z-lab/Qwen3.8-27B-DFlash2 revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4; greedy; two warmups and five measured 128-token requests; LocalMaxxing leading cache-bust nonce retained.
LocalMaxxing reasoning-v1 canonical prompt, plain /v1/completions SSE, temperature 0, 2 warmups then 5 measured sequential requests; median-throughput iteration submitted with its own timings and acceptance counters. 512 output tokens each. Decode=(output_tokens-1)/(last_text-first_text). Prefix caching disabled. One P150 32GB; custom TT-Metal runtime with packed BFP4/BF16 weights, fused GDN/SwiGLU/state commit, sharded norms, and MTP-1. TT-native cache package, not standalone CUDA/Transformers quant; pinned original checkpoint is also required. Experimental: sharded norms change numerics and prior long-prompt repeated greedy outputs varied. No peak VRAM or power measurement claimed. Reproduction and limitations: https://huggingface.co/Lottolabs/Qwen3.5-9B-TT-BFP4-P150/blob/main/manifest.json
Tenstorrent P150, Qwen3.5-9B bfp_bf4, vLLM concurrency 1. Fastest deployed MTP-1 path: one speculative draft token per cycle (specNumTokens=1), verified with the committed token in one M=2 target-model batch. One untimed warmup plus five timed 256-token streamed requests. Aggregate server counters across all six requests: 1,136 drafted verification tokens and 394 accepted speculative tokens (34.683% acceptance); 1,136 + 394 = 1,530 MTP-produced tokens, plus one initial prefill token per request = 1,536 total output tokens. Endpoint usage reported 538 prompt tokens per request; submitted speed metrics are CLI medians over the five timed requests.
Fastest verified SGLang setup for Qwen3.8-27B W4A16 AutoRound on 2x RTX 3090 with TP2. Uses DFlash speculative decoding with a local W8A16 DFlash2 drafter, 6 draft tokens, and a 40,960-token output-derived curated candidate vocabulary distributed across TP ranks. Candidate selection uses rank-local projection/top-k with compact cross-rank communication. Target execution uses torch.compile, CUDA graphs, selective SM86 W4A8 Marlin MLP kernels, BF16 KV/recurrent state, compiler-safe DFlash causal-convolution verification, and persisted tuned TorchInductor/Triton caches. Measurement is concurrency 1 with 2 warmups and 5 timed 512-token iterations. CUDA peer access and NVLink are unavailable on this PCIe topology, so TP collectives use NCCL's host-mediated path.
Q8_0 on pinned llama.cpp PR 26608 commit d8d862521, CUDA, 2x RTX 3090, all layers GPU-offloaded with 1:1 layer split, 131072 server context, flash attention. Prefill and decode are llama-bench pp2048/tg256 means over 5 repetitions after default warmup: 9243.21531 prompt tok/s and 232.774459 output tok/s. TTFT is median of 5 streaming endpoint requests after 2 warmups with prompt caching explicitly disabled: 47.77 ms (range 47.71-47.97 ms). The endpoint TTFT prompt was 42 actual tokens. Remote semantic decode was 224.4 tok/s median; llama-bench sustained decode is submitted as tokSOut.
Qwen3.5-9B fine-tune/distillation served from the named Jackrong GGUF repository; this is not a DeepSeek-V4-Flash architecture model. Remote OpenAI-compatible speed test: 1 warmup plus 5 measured 256-token generations.
A/B-tuned for 2x RTX 3090 with a usable 32K context: layer split, flash attention on, 16 CPU threads, batch 512, micro-batch 512, F16 K/V cache, automatic GPU-layer fitting with 256 MiB margin per GPU and 32768-token fit context. Five repetitions; 2048-token prompt processing and 256-token generation tests.
