Speed test matrix

Inkling-Small on H200 SXM 141GB ×8

vllm · 0.29.0 · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 17, 2026, 20:47 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 8
VRAM: 1128 GB (8 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: thinkingmachines/Inkling-Small
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve thinkingmachines/Inkling-Small --served-model-name thinkingmachines/Inkling-Small --trust-remote-code --tensor-parallel-size 8 --reasoning-parser inkling --enable-auto-tool-choice --tool-call-parser inkling --max-model-len 1048576 --gpu-memory-utilization 0.87 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode inkling --kernel-config.enable_flashinfer_autotune=False

Prefill

prompt tok/s by context length

8K context16.1K508 ms TTFT

16K context32K508 ms TTFT

32K context31.7K1,019 ms TTFT

64K context42.2K1,530 ms TTFT

128K context42.1K3,059 ms TTFT

256K context38.7K6,648 ms TTFT

512K context28.7K17,954 ms TTFT

1019K context19.3K53,020 ms TTFT

1024K context19.1K53,910 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	135.550 ms TTFT	238.060 ms TTFT	439.494 ms TTFT	770.4124 ms TTFT	1,424.3183 ms TTFT	2,466.0306 ms TTFT	4,222.9530 ms TTFT	7,302.01,044 ms TTFT
16K	104.173 ms TTFT	194.390 ms TTFT	364.0132 ms TTFT	653.0180 ms TTFT	1,221.9326 ms TTFT	2,139.7528 ms TTFT	3,748.5895 ms TTFT	Over capacity
32K	75.0123 ms TTFT	141.4156 ms TTFT	268.9212 ms TTFT	496.5322 ms TTFT	944.6511 ms TTFT	1,693.8946 ms TTFT	Over capacity	Over capacity
64K	47.8236 ms TTFT	91.0309 ms TTFT	176.6433 ms TTFT	334.0635 ms TTFT	643.81,012 ms TTFT	Over capacity	Over capacity	Over capacity
128K	27.6472 ms TTFT	52.9630 ms TTFT	104.2889 ms TTFT	201.41,356 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
256K	14.9973 ms TTFT	28.81,298 ms TTFT	57.21,801 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
512K	7.82,150 ms TTFT	15.12,910 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	4.04,881 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	16.1K	—	508 ms	—	—	1	—	—
Prefill	16K	1	Valid	32K	—	508 ms	—	—	1	—	—
Prefill	32K	1	Valid	31.7K	—	1,019 ms	—	—	1	—	—
Prefill	64K	1	Valid	42.2K	—	1,530 ms	—	—	1	—	—
Prefill	128K	1	Valid	42.1K	—	3,059 ms	—	—	1	—	—
Prefill	256K	1	Valid	38.7K	—	6,648 ms	—	—	1	—	—
Prefill	512K	1	Valid	28.7K	—	17,954 ms	—	—	1	—	—
Prefill	1019K	1	Valid	19.3K	—	53,020 ms	—	—	1	—	—
Prefill	1024K	1	Valid	19.1K	—	53,910 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	135.5	140.1	50 ms	7.1 ms	1.0	1	4,064	30.0 s
Sustained decode	8K	2	Valid	238.0	124.4	60 ms	8.0 ms	2.0	2	7,138	30.0 s
Sustained decode	8K	4	Valid	439.4	110.7	94 ms	9.0 ms	4.0	0	13,180	30.0 s
Sustained decode	8K	8	Valid	770.4	97.0	124 ms	10.3 ms	8.0	0	23,112	30.0 s
Sustained decode	8K	16	Valid	1,424.3	90.0	183 ms	11.1 ms	16.0	0	42,719	30.0 s
Sustained decode	8K	32	Valid	2,466.0	77.8	306 ms	12.8 ms	32.0	0	73,984	30.0 s
Sustained decode	8K	64	Valid	4,222.9	66.6	530 ms	15.0 ms	64.0	0	126,719	30.0 s
Sustained decode	8K	128	Valid	7,302.0	57.5	1,044 ms	17.4 ms	128.0	0	219,905	30.1 s
Sustained decode	16K	1	Valid	104.1	104.7	73 ms	9.6 ms	1.0	0	3,124	30.0 s
Sustained decode	16K	2	Valid	194.3	97.7	90 ms	10.2 ms	2.0	0	5,828	30.0 s
Sustained decode	16K	4	Valid	364.0	91.5	132 ms	10.9 ms	4.0	0	10,919	30.0 s
Sustained decode	16K	8	Valid	653.0	82.0	180 ms	12.2 ms	8.0	0	19,584	30.0 s
Sustained decode	16K	16	Valid	1,221.9	76.9	326 ms	13.0 ms	16.0	0	36,656	30.0 s
Sustained decode	16K	32	Valid	2,139.7	67.4	528 ms	14.8 ms	32.0	0	64,193	30.0 s
Sustained decode	16K	64	Valid	3,748.5	59.0	895 ms	16.9 ms	64.0	0	112,447	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	75.0	75.2	123 ms	13.3 ms	1.0	0	2,250	30.0 s
Sustained decode	32K	2	Valid	141.4	71.0	156 ms	14.1 ms	2.0	0	4,242	30.0 s
Sustained decode	32K	4	Valid	268.9	67.4	212 ms	14.8 ms	4.0	0	8,064	30.0 s
Sustained decode	32K	8	Valid	496.5	62.2	322 ms	16.1 ms	8.0	0	14,896	30.0 s
Sustained decode	32K	16	Valid	944.6	59.2	511 ms	16.9 ms	16.0	0	28,336	30.0 s
Sustained decode	32K	32	Valid	1,693.8	53.1	946 ms	18.8 ms	32.0	0	50,816	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	47.8	47.9	236 ms	20.9 ms	1.0	0	1,434	30.0 s
Sustained decode	64K	2	Valid	91.0	45.6	309 ms	21.9 ms	2.0	0	2,730	30.0 s
Sustained decode	64K	4	Valid	176.6	44.2	433 ms	22.6 ms	4.0	0	5,296	30.0 s
Sustained decode	64K	8	Valid	334.0	41.8	635 ms	23.9 ms	8.0	0	10,016	30.0 s
Sustained decode	64K	16	Valid	643.8	40.2	1,012 ms	24.9 ms	16.0	0	19,315	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	27.6	27.6	472 ms	36.2 ms	1.0	0	826	30.0 s
Sustained decode	128K	2	Valid	52.9	26.5	630 ms	37.7 ms	2.0	0	1,588	30.0 s
Sustained decode	128K	4	Valid	104.2	26.1	889 ms	38.4 ms	4.0	0	3,124	30.0 s
Sustained decode	128K	8	Valid	201.4	25.2	1,356 ms	39.7 ms	8.0	0	6,037	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	14.9	15.0	973 ms	66.8 ms	1.0	0	447	30.0 s
Sustained decode	256K	2	Valid	28.8	14.5	1,298 ms	69.1 ms	2.0	0	864	30.0 s
Sustained decode	256K	4	Valid	57.2	14.3	1,801 ms	69.8 ms	4.0	0	1,712	29.9 s
Sustained decode	256K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	7.8	7.8	2,150 ms	127.6 ms	1.0	0	233	29.9 s
Sustained decode	512K	2	Valid	15.1	7.6	2,910 ms	131.7 ms	2.0	0	452	29.9 s
Sustained decode	512K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	4.0	4.1	4,881 ms	246.9 ms	1.0	0	119	29.9 s
Sustained decode	1019K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.