Speed test matrix

Step-3.7-Flash-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4

sglang · 0.0.0.dev1+gf92199b56 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Oct 12, 2026, 18:13 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 4
VRAM: 384 GB (4 × 96 GB)
OS: linux

Engine

Engine: sglang
Version: 0.0.0.dev1+gf92199b56
Quantization: FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: f92199b569d2

Run

Served model: step3p7
Revision: main
Benchmark tool: llm-inference-bench v0.7.9
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_rtxpro6k-stepfun-ai-sglang-f92199b569d2/bin/sglang" serve --model-path stepfun-ai/Step-3.7-Flash-FP8 --served-model-name step3p7 --trust-remote-code --tp 4 --reasoning-parser step3p5 --tool-call-parser step3p5 --context-length 262144 --mem-fraction-static 0.77 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --ep 4 --enable-multimodal --attention-backend fa4 --fp8-gemm-backend cutlass --moe-runner-backend triton

Prefill

prompt tok/s by context length

8K context15K547 ms TTFT

16K context15.5K1,047 ms TTFT

32K context15.3K2,110 ms TTFT

64K context14.1K4,574 ms TTFT

128K context12.7K10,118 ms TTFT

251K context10.3K24,375 ms TTFT

256K context10.1K25,453 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	118.4498 ms TTFT	175.6160 ms TTFT	310.1316 ms TTFT	512.5189 ms TTFT	798.9525 ms TTFT	1,310.0379 ms TTFT	Over capacity	Over capacity
16K	115.290 ms TTFT	172.5138 ms TTFT	306.1193 ms TTFT	505.7264 ms TTFT	798.5424 ms TTFT	Over capacity	Over capacity	Over capacity
32K	109.4116 ms TTFT	165.5218 ms TTFT	295.0383 ms TTFT	490.8501 ms TTFT	772.8809 ms TTFT	Over capacity	Over capacity	Over capacity
64K	99.2261 ms TTFT	154.5237 ms TTFT	276.8401 ms TTFT	466.4831 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	84.5258 ms TTFT	135.6432 ms TTFT	246.8844 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	65.4501 ms TTFT	109.8970 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	15K	—	547 ms	—	—	1	—	—
Prefill	16K	1	Valid	15.5K	—	1,047 ms	—	—	1	—	—
Prefill	32K	1	Valid	15.3K	—	2,110 ms	—	—	1	—	—
Prefill	64K	1	Valid	14.1K	—	4,574 ms	—	—	1	—	—
Prefill	128K	1	Valid	12.7K	—	10,118 ms	—	—	1	—	—
Prefill	251K	1	Valid	10.3K	—	24,375 ms	—	—	1	—	—
Prefill	256K	1	Valid	10.1K	—	25,453 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	118.4	118.5	498 ms	8.4 ms	1.0	0	3,553	30.0 s
Sustained decode	8K	2	Valid	175.6	87.7	160 ms	11.4 ms	2.0	0	5,268	30.0 s
Sustained decode	8K	4	Valid	310.1	77.7	316 ms	12.9 ms	4.0	0	9,304	30.0 s
Sustained decode	8K	8	Valid	512.5	64.3	189 ms	15.6 ms	8.0	0	15,374	30.0 s
Sustained decode	8K	16	Valid	798.9	50.3	525 ms	19.9 ms	16.0	0	23,952	30.0 s
Sustained decode	8K	32	Valid	1,310.0	41.5	379 ms	24.1 ms	32.0	0	39,296	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	115.2	115.2	90 ms	8.7 ms	1.0	0	3,455	30.0 s
Sustained decode	16K	2	Valid	172.5	86.3	138 ms	11.6 ms	2.0	0	5,176	30.0 s
Sustained decode	16K	4	Valid	306.1	76.5	193 ms	13.1 ms	4.0	0	9,180	30.0 s
Sustained decode	16K	8	Valid	505.7	63.4	264 ms	15.8 ms	8.0	0	15,168	30.0 s
Sustained decode	16K	16	Valid	798.5	50.3	424 ms	19.9 ms	16.0	0	23,952	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	109.4	109.4	116 ms	9.1 ms	1.0	0	3,283	30.0 s
Sustained decode	32K	2	Valid	165.5	82.8	218 ms	12.1 ms	2.0	0	4,964	30.0 s
Sustained decode	32K	4	Valid	295.0	73.8	383 ms	13.6 ms	4.0	0	8,848	30.0 s
Sustained decode	32K	8	Valid	490.8	61.4	501 ms	16.3 ms	8.0	0	14,720	30.0 s
Sustained decode	32K	16	Valid	772.8	48.6	809 ms	20.6 ms	16.0	0	23,185	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	99.2	99.2	261 ms	10.1 ms	1.0	0	2,975	30.0 s
Sustained decode	64K	2	Valid	154.5	77.2	237 ms	12.9 ms	2.0	0	4,634	30.0 s
Sustained decode	64K	4	Valid	276.8	69.2	401 ms	14.4 ms	4.0	0	8,304	30.0 s
Sustained decode	64K	8	Valid	466.4	58.4	831 ms	17.1 ms	8.0	0	13,984	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	84.5	84.5	258 ms	11.8 ms	1.0	0	2,534	30.0 s
Sustained decode	128K	2	Valid	135.6	68.0	432 ms	14.7 ms	2.0	0	4,066	30.0 s
Sustained decode	128K	4	Valid	246.8	62.1	844 ms	16.1 ms	4.0	0	7,405	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	65.4	65.5	501 ms	15.3 ms	1.0	0	1,962	30.0 s
Sustained decode	251K	2	Valid	109.8	55.2	970 ms	18.1 ms	2.0	0	3,294	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.