Qwen3.8 27B GPTQ + MTP on 2× RTX 3090: the vLLM setup behind our Terminal-Bench run

This is the inference stack behind our Qwen3.8 27B Terminal-Bench 2.1 run: a 4-bit GPTQ checkpoint served by vLLM across two RTX 3090s, with the model's native MTP head drafting two speculative tokens at a time.

The configuration was designed for a difficult combination: fit a 27B-class model on 48 GB of consumer VRAM, retain the model's full 262,144-token context window, expose reasoning and tool calls through an OpenAI-compatible API, and keep long interactive agent sessions responsive enough to run for days.

The short version:

• 96.6 output tokens/s median in the controlled single-request MTP test
• 1,311 prompt tokens/s estimated cold prefill on a 1,044-token prompt
• 796 ms median time to first token in that uncached test
• 95.6% MTP draft-token acceptance at concurrency 1 on the long-prompt workload
• 175.7 aggregate output tokens/s at concurrency 16
• 45.28 GB peak total GPU memory across two 24 GB cards in the tuned speed test
• 62/89, or 69.66%, on the clean xhigh Terminal-Bench 2.1 pass

The speed result and the Terminal-Bench result describe the same model, engine, GPUs, GPTQ-Marlin path, and MTP method. They do not use exactly the same serving envelope: the headline latency test used an 8K, single-sequence, prefix-cache-disabled performance profile, while Terminal-Bench used the 262K long-context deployment profile described below. Keeping those profiles separate avoids presenting an optimized microbenchmark as if it were the latency of a 262K agent session.

Hardware and software

The checkpoint is approximately 19.54 GiB on disk. vLLM reported 9.49 GiB of model memory on each tensor-parallel worker after loading the target and MTP drafter. The tuned 8K benchmark peaked at 45.28 GB across both cards, leaving little spare VRAM but fitting entirely on the GPUs.

The Terminal-Bench serving command

The recorded launcher command was:

/home/lotto/vllm/bin/vllm serve \
  /home/lotto/models/Qwen3.8-27B-GPTQ-4bit \
  --served-model-name qwen3.8-27b \
  --host 127.0.0.1 \
  --port 8087 \
  --tensor-parallel-size 2 \
  --quantization gptq_marlin \
  --dtype float16 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --max-num-seqs 16 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking":false}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

One reproducibility wrinkle is visible in the retained logs: the launcher record contains --max-num-seqs 16, while vLLM's startup configuration echoed an effective max_num_seqs: 32. The scheduler warning also reasoned about the larger effective value. Treat 16 as the requested CLI setting and 32 as the value observed by the running engine; anyone recreating the setup should inspect vLLM's startup line instead of assuming the flag survived unchanged.

Terminal-Bench called the endpoint at concurrency 1. The higher sequence budget was still useful for normal multi-client serving and for the separate concurrency sweep.

Why each flag is there

Tensor parallelism across both 3090s

--tensor-parallel-size 2 shards each layer across both GPUs. That is the straightforward way to keep the model, MTP head, and a large cache resident in 48 GB.

This host does not expose an NVLink path between the cards: nvidia-smi topo -m reports PHB. vLLM also reported that GPU P2P was unavailable, disabled its custom all-reduce path, and selected PYNCCL/NCCL for tensor-parallel collectives. The setup therefore pays PCIe/host-bridge communication overhead every layer. It works, but it is not the ideal topology for TP=2.

GPTQ-Marlin weights

The checkpoint stores symmetric 4-bit GPTQ weights with group size 32 and activation-order quantization disabled. vLLM selected its Marlin linear kernel for the AutoGPTQ path. Activations remain FP16, so “Q4” describes weight storage and matrix multiplication, not an all-int4 pipeline.

The model config declares BF16, but the server explicitly requested FP16 and vLLM logged the BF16-to-FP16 cast. This is a practical Ampere choice, though it is another reason the report names the actual runtime dtype rather than only the checkpoint dtype.

FP8 KV cache

--kv-cache-dtype fp8 is what makes a 262K context practical alongside a 27B-class model on two 24 GB cards. The tradeoff is accuracy risk: vLLM explicitly warns that FP8 cache values can lose accuracy without an appropriate scaling factor. We did not run an isolated FP16-KV versus FP8-KV quality study, so the Terminal-Bench score should be understood as quality of this entire deployment, including FP8 cache.

Language-only mode

The underlying architecture is Qwen3_5ForConditionalGeneration and the checkpoint includes vision configuration, but Terminal-Bench was text-only. --language-model-only disabled all multimodal modalities. This avoids reserving runtime capacity for an encoder that the workload never calls.

Hybrid attention, Mamba alignment, and prefix caching

The text model has 64 layers with a repeating pattern of three linear-attention layers followed by one full-attention layer. vLLM selected the Qwen GDN prefill path for the linear layers and FlashInfer/FlashAttention paths where applicable.

--mamba-cache-mode align lets the hybrid cache line up state-space and attention pages. At startup, vLLM chose a 1,600-token attention block, then padded the Mamba page by 0.88% so both page sizes matched exactly. Prefix caching in this mode was marked experimental by vLLM 0.26.0, so it is a performance feature with a compatibility caveat rather than a free default.

Native MTP with two speculative tokens

The model metadata contains one MTP hidden layer and does not use dedicated MTP embeddings. vLLM detected that head and shared the target model's embedding and LM-head weights with the drafter. That is materially lighter than loading a separate draft model.

num_speculative_tokens: 2 asks the MTP layer to propose two tokens before target verification. vLLM warns that proposing more than one token repeatedly forwards through the same single MTP layer, which can reduce acceptance. In this workload, two remained effective: measured acceptance stayed between roughly 87% and 96% across the concurrency sweep.

The acceptance-length metric has a maximum of 3 here: one target token plus up to two accepted draft tokens. At concurrency 1, the measured length was 2.912, meaning the engine advanced almost three tokens per verification cycle on average.

Reasoning and tool-call parsing

The server exposed both --reasoning-parser qwen3 and --tool-call-parser qwen3_coder. The default chat-template argument disabled thinking for ordinary requests, but clients such as the OMP Terminal-Bench harness could request a reasoning effort explicitly. That separation allowed the same endpoint to serve short non-thinking speed tests and xhigh agent trajectories.

Controlled single-request result

The approved LocalMaxxing speed test used two warmups followed by five measured streaming iterations. Every iteration used the same 1,044-token prompt and generated 512 tokens at concurrency 1. Prefix caching was disabled for this profile, so the prefill figure represents actual prompt processing rather than a cache hit.

The five decode samples varied by only 0.2 tok/s end to end. This is a stable result, not a single lucky request.

A later same-model, same-hardware, same-prompt non-speculative vLLM run measured 65.3 tok/s decode. Relative to that result, the MTP profile's 96.6 tok/s is 47.9% faster. The two artifacts were not captured under an identical server envelope—the baseline had a very different TTFT and appears to have benefited from prefix caching—so 47.9% is an observed cross-run gain, not a laboratory-isolated MTP uplift. The clean conclusion is that MTP delivered a large decode gain in the deployed family of configurations; an exact causal percentage would require alternating MTP on/off against one unchanged server config.

Long-prompt concurrency sweep

The more deployment-representative sweep used 42,000 input tokens and 512 output tokens per request. Requests arrived as a burst. Output throughput below is aggregate across all active requests; TPOT is per-request time per output token.

Throughput scales strongly through concurrency 8, grows modestly through 16, then stops improving. Concurrency 24 is slightly slower than 16 while per-request latency is much worse. For this workload, 16 is the throughput knee: about 2.4× the aggregate output rate of concurrency 1, at the cost of substantially slower individual streams.

This also reconciles two numbers that otherwise look inconsistent. The concurrency-1 run delivers 72.85 aggregate output tok/s over the entire request wall clock because it includes the 42K prefill. Its median TPOT is 9.71 ms, equivalent to roughly 103 tok/s once autoregressive decoding is underway—close to the controlled 96.6 tok/s decode result.

MTP acceptance falls from 95.6% at concurrency 1 to the high-80s under batching, but it remains useful. The acceptance length stays around 2.74–2.79 at higher concurrency, so most of the two proposed tokens still survive target verification.

Context length and prefix-cache behavior

Cold-prompt TTFT grows sharply with context length:

The decline in effective prompt rate at longer contexts is expected: full-attention work grows with sequence length, cache traffic grows, and the hybrid model still pays periodic full-attention cost even though most layers use linear attention.

Prefix caching changes the interactive-agent experience. In a focused 42K workload labeled as a reused 40K prefix plus 2K of new context, four serial requests produced a median TTFT of 1.90 seconds. The same artifact retained a p99 of 33.10 seconds, exposing the initial cold fill. Reporting only the median would hide the cold-start penalty; reporting only the mean would hide how fast the warmed turns became.

That pattern matches Terminal-Bench and coding-agent traffic well. Early turns pay to ingest the task and system context. Later turns reuse most of it and append tool output or reasoning. Prefix caching does not make 42K tokens free, but it can move steady-state turn latency from tens of seconds toward a few seconds when the prefix remains reusable.

Scheduler and topology limits visible in the logs

The server worked reliably for the five-day evaluation, but startup diagnostics identify clear optimization targets.

1. The scheduler token budget was only 2,048. With speculative decoding enabled, vLLM warned that max_num_scheduled_tokens had been reduced to 2,048 and suggested increasing max_num_batched_tokens, reducing speculative depth, or reducing sequence concurrency. The concurrency curve flattening after 16 is consistent with a scheduler bottleneck, though the run did not isolate causality.
2. Tensor-parallel communication crossed the PCIe host bridge. No GPU P2P path was available and custom all-reduce was disabled. A platform with working P2P or NVLink should reduce collective overhead.
3. The cache stack was aggressive. FP8 KV cache and experimental prefix caching in Mamba align mode enabled the long context, but both deserve quality and correctness checks on a new vLLM release.
4. Two speculative tokens are not automatically optimal. vLLM's own warning notes that a one-layer MTP head is forwarded repeatedly for the second proposal. We measured high acceptance, but a controlled depth-1 versus depth-2 test with identical streaming metrics would tell whether the second draft token pays for itself.
5. Some sampling controls are incompatible. vLLM warned that min_p and logit_bias do not work with speculative decoding in this configuration. Clients depending on those controls should not silently reuse this endpoint profile.

What the setup delivered in Terminal-Bench 2.1

The endpoint powered the complete 89-task Terminal-Bench 2.1 experiment. With xhigh reasoning, the clean first pass solved 62/89 tasks (69.66%). Adaptive retries at medium, low, and thinking-off settings raised the cumulative best-of result to 70/89 (78.65%).

Across every benchmark attempt, the endpoint processed 880.38 million tokens over 10,821 model calls. That workload stress-tested more than peak decode speed: it exercised very long conversations, repeated prefix reuse, tool-call parsing, reasoning extraction, context compaction, and hours-long task sessions. The fact that the server remained usable through that run is the strongest operational evidence for the configuration.

The first-pass and adaptive quality scores are analyzed separately in the Terminal-Bench report. A trace-level audit of all 19 residual failures found four recurring behaviors: investigation displaced the required artifact; custom self-tests checked a proxy or repeated the implementation's assumptions; plausible but unsupported interpretations became expensive commitments; and candidates were submitted even when the model's own checks were still red. The eight fallback recoveries mostly came from corrected technical choices rather than simply shorter reasoning. The important setup result is that a 4-bit, MTP-enabled 27B model on two consumer GPUs was capable of both approximately 100 tok/s single-stream decode and sustained long-horizon agent work at a 262K context limit.

Recommended configuration changes

If rebuilding this stack, I would preserve the core choices and test three changes independently:

1. Keep GPTQ-Marlin, TP=2, FP8 KV, language-only mode, and MTP depth 2 as the baseline. This is the configuration with demonstrated benchmark quality and stability.
2. Set and sweep --max-num-batched-tokens. The startup warning is concrete, and the throughput plateau suggests room to improve scheduling. Test it at fixed concurrency 8, 16, and 24 while watching TTFT and MTP acceptance.
3. Run a strict MTP ablation. Alternate depth 0, 1, and 2 without changing max context, cache state, prompt, sampling, or server process. Report inter-token decode rate, TTFT, acceptance, and total request wall time together.
4. Separate cold and warm prefix tests. Use one cold fill, then report the next N cached turns independently. Mixed aggregates obscure both user experiences.
5. Verify FP8-cache quality. Re-run a small deterministic long-context correctness set with FP16 and FP8 KV cache before attributing all quality variation to the model or reasoning setting.
6. Prefer a working P2P topology when possible. The current PHB path is functional, but two-card tensor parallelism benefits directly from lower-latency collectives.

Reproducing the LocalMaxxing speed test

With the server running, the exact saved workload and client settings can be replayed from the LocalMaxxing run record:

lmx speed-test runs rerun \
  runs/btbtyler09-Qwen3.8-27B-GPTQ-4bit/20260817T062502Z.json \
  --out qwen38-vllm-mtp-speed-rerun.json

The saved record preserves the 1,044-token prompt, 512-token output limit, two warmups, five measured iterations, streaming mode, model identity, quantization, and hardware metadata. Replaying it against a different server configuration measures that new server; it does not prove the launch flags are identical.

Use an uncached or varied prompt when measuring prefill. Repeating the exact prompt against a prefix-caching server can produce an impressive but misleading “prefill throughput” number that is actually a cache-hit measurement.

Conclusion

This setup works because each memory-saving choice is paired with a throughput feature: 4-bit GPTQ-Marlin weights, FP8 KV cache, text-only execution, TP=2 sharding, prefix reuse for long conversations, and native MTP to recover decode speed.

The measured outcome is strong for consumer Ampere hardware: 96.6 tok/s controlled single-stream decode, nearly 176 aggregate output tok/s at concurrency 16, high MTP acceptance, and enough context capacity to complete a five-day Terminal-Bench run.

The main caveat is that there is no single “speed” for the system. A short uncached request, a warmed 42K agent turn, a cold 128K prefill, and 16 simultaneous clients exercise different bottlenecks. The useful report is therefore the whole curve: decode rate, TTFT, prefix behavior, concurrency scaling, acceptance rate, memory use, and benchmark quality together.