FreeToken vs vLLM on one RTX 3090: BF16 MoE RAM offload compared

Summary

I served the same full Qwen3.6-35B-A3B BF16 checkpoint on one NVIDIA GeForce RTX 3090 with two different RAM-offload implementations: FreeToken 0.1.2 expert offload and vLLM 0.27.1 CPU offload. Both configurations used approximately 22.3 GiB of the card's 24 GiB VRAM, a 16,384-token server allocation, at most four concurrent requests, streamed OpenAI-compatible chat completions, temperature 0, and the same prompts.

FreeToken was decisively faster once generation was established:

• 34.04 tok/s complete-request throughput versus 6.93 tok/s for vLLM
• 42.78 tok/s established decode versus 7.01 tok/s
• 47.69 tok/s four-request aggregate throughput versus 12.88 tok/s
• Approximately 1,058 prompt tok/s effective prefill versus 608 tok/s

vLLM had the better latency to the first token: 0.806 seconds versus 3.046 seconds. The sustained decode result nevertheless favored FreeToken by 6.10×. The reason is architectural: generic vLLM CPU offload moved about 45 GB of parameters across PCIe during inference, whereas FreeToken's MoE-specific backend retained a GPU LRU cache and transferred only selected expert weights on cache misses.

All figures in this report are local measurements. No result is an estimate.

Test system

The second RTX 3090 remained unused during both measurements.

Runtime configurations

FreeToken

cuda13 /home/lotto/freetoken-runtime/.venv/bin/ft serve \
  --model /home/lotto/models/Qwen3.6-35B-A3B \
  --served-model-name qwen3.6-35b-a3b \
  --moe-backend offload \
  --moe-cache-auto \
  --num-tokens 16384 \
  --max-output-tokens 1024 \
  --decode-log-interval 128

FreeToken automatically created a 2,486-slot GPU expert cache with an LRU replacement policy. The complete routed-expert bank remained available in system memory.

vLLM

I tested CPU-offload settings of 50, 48, 46, 45, and 44 GB. The fastest configuration that retained safe VRAM headroom was:

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve \
  /home/lotto/models/Qwen3.6-35B-A3B \
  --served-model-name qwen3.6-35b-a3b \
  --dtype bfloat16 \
  --cpu-offload-gb 44 \
  --kv-cache-memory 536870912 \
  --max-model-len 16384 \
  --max-num-seqs 4 \
  --enforce-eager \
  --trust-remote-code

The runtime reported 44.94 GB of CPU-offloaded parameters and a 20,695-token GPU KV cache. Eager mode avoided CUDA-graph memory so more weights could remain resident. A 512 MiB explicit KV allocation was sufficient for the matched 16,384-token server capacity while preserving weight residency.

The vLLM runtime came from the locally used patched qwen38-27b-3090:latest image, based on vLLM 0.27.1. Speculative decoding was disabled for this comparison.

Methodology

Warm-up

Each runtime first generated 256 tokens with EOS ignored or a fixed output limit. The four-request path received a separate warm-up of four simultaneous 128-token requests.

Sequential decode

The same three prompts were submitted sequentially. Each requested 512 output tokens at temperature 0 using streaming responses. FreeToken emitted 511 measured completion tokens per request; vLLM emitted 512. Throughput uses each runtime's reported completion-token count, so the one-token termination difference does not bias the rate materially.

TTFT was measured at the first streamed content or reasoning token. Established decode throughput was calculated as:

completion tokens / (total wall time - TTFT)

Prefill

The same repeated-text prompt produced 4,106 prompt tokens in both runtimes and requested one output token. Effective prompt throughput includes API and response overhead.

Concurrency

Four simultaneous requests each requested 512 output tokens after the batch warm-up. FreeToken returned 511 tokens per request; vLLM returned 512.

Headline comparison

Sequential results

FreeToken

Mean established decode was 42.78 tok/s.

vLLM

vLLM delivered its first token quickly and consistently. Afterward, each generation settled near 7 tok/s because the offloaded weight traffic repeated through the decode loop.

Prefill and concurrency

Batching helped both runtimes. FreeToken's aggregate output improved about 40% over its complete-request single-stream result. vLLM improved about 86%, from 6.93 to 12.88 tok/s, but remained 3.70× slower than FreeToken at four-way concurrency.

vLLM offload tuning

Reducing the offload allocation increased GPU-resident weight capacity and improved throughput. At 44 GB, the server used 22,787 MiB, or 22.25 GiB, of VRAM and left approximately 1.3 GiB free according to nvidia-smi. Going materially lower would remove the safety margin for runtime variability and longer requests.

The final vLLM process increased used host memory by approximately 52 GiB relative to the pre-launch state. FreeToken's measured increase was approximately 64 GiB. These host figures are process-level observations on a machine with a warm filesystem cache and occupied swap, not allocator-exclusive accounting.

Why the performance differs

vLLM uses general parameter offload

vLLM makes the model fit by placing a configured portion of parameters in CPU memory and copying those parameters to the GPU when their modules execute. This mechanism works with many architectures, but autoregressive decode repeatedly traverses the same offloaded model path. For this 66.97 GiB BF16 checkpoint, PCIe movement dominates after the first token.

FreeToken exploits MoE sparsity

Qwen3.6-35B-A3B has 256 routed experts and selects eight per token. FreeToken keeps the expert banks in host memory, reserves most available VRAM for a shared LRU expert cache, and fetches selected experts on demand. Frequently routed experts remain on the GPU instead of being retransferred every step.

This changes the offload unit from generic model modules to sparse expert traffic. The result is a 6.10× established-decode advantage at nearly identical device-memory use.

Why vLLM still wins TTFT

The short prompts require little prefill work, and vLLM can begin the first decode step before the sustained PCIe penalty accumulates across hundreds of tokens. FreeToken performs more cache and expert-path setup before its first streamed token. For short responses where TTFT dominates, vLLM's lower initial latency may matter. For sustained generation, FreeToken's much higher decode rate dominates total wall time.

Can MTP or DFlash be added to FreeToken?

Not with the tested FreeToken 0.1.2 server.

The Qwen3.6 checkpoint does contain a one-layer native MTP head: its configuration declares mtp_num_hidden_layers: 1, and its final safetensors shards contain the mtp.* tensors. FreeToken's Qwen3.5/Qwen3.6 MoE weight loader explicitly skips all mtp.* weights, however, and ft serve --help exposes no speculative-decoding, draft-model, MTP, or DFlash option. A direct launch attempt with --speculative-algorithm MTP was rejected as an unrecognized argument before model loading.

DFlash is also not a generic switch. It requires a draft checkpoint trained for the exact target model plus runtime support for drafting, target verification, acceptance sampling, and the model's hybrid attention/recurrent state. The existing Qwen3.8-27B DFlash2 draft is not compatible with this Qwen3.6-35B-A3B target, and FreeToken 0.1.2 does not expose a DFlash scheduler.

Therefore the measured FreeToken result is non-speculative. Adding MTP or DFlash would require implementation work in FreeToken rather than a launch-flag change. vLLM can serve supported models with speculative decoding, but combining its generic CPU-offload path with Qwen3.6 MTP is a separate experiment and is not represented by the measurements above.

Practical conclusion

For the full BF16 Qwen3.6-35B-A3B checkpoint on one RTX 3090:

• Choose FreeToken for long generations and sustained throughput. It delivered 34.04 tok/s end to end, 42.78 tok/s established decode, and 47.69 tok/s aggregate output at four-way concurrency.
• Choose vLLM CPU offload only when its broader runtime ecosystem or lower TTFT matters more than generation speed. Its best safe configuration delivered 6.93 tok/s end to end, 7.01 tok/s established decode, and 12.88 tok/s aggregate output at four-way concurrency.
• Do not assume native MTP weights automatically accelerate FreeToken. Version 0.1.2 intentionally drops this checkpoint's MTP head and has no speculative serving path.

The experiment shows why architecture-aware offload matters. Both runtimes fit the same 66.97 GiB checkpoint into essentially the same 22.3 GiB device footprint, but FreeToken's expert-level cache converted PCIe bandwidth into useful reuse while vLLM's generic offload repeatedly paid the transfer cost during decode.