GSM8K
Official question setGSM8K deterministic disjoint eval shards from openai/gsm8k/main:test. Running all 13 shards covers the full 1,319-question dataset; Postgres stores only shard metadata and pass/fail results.
Category: ReasoningType: Question setQuestions: 1,319Question batches: 13Runs: 343
Dry-run first:
lmx eval shard gsm8k --base-url http://localhost:8000 --questions 298 --dry-runThen submit with a real model and hardware profile:
lmx eval shard gsm8k --base-url http://localhost:8000 --questions 298 --model <hfId> --hardware hardware.json --submitEach question counts once per model, using its latest answer. The leaderboard is ranked by the lower bound of a 95% confidence interval, so a model needs enough answered questions to rank highly.
Leaderboard
pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 · 1,246 of 1,319 questions · 13 runs
97.8%
95% CI 96.9–98.5%
1,219/1,246 correct
2
nvidia/Gemma-4-31B-IT-NVFP4 · 1,319 of 1,319 questions · 20 runs
97.6%
95% CI 96.7–98.3%
1,288/1,319 correct
3
orcarouter/Qwen3.8-27B-Uncensored-FP8 · 1,319 of 1,319 questions · 13 runs
97.3%
95% CI 96.3–98.1%
1,284/1,319 correct
4
dbirks/Qwen3.8-27B-W4A16-AutoRound · 1,319 of 1,319 questions · 13 runs
97.3%
95% CI 96.2–98.0%
1,283/1,319 correct
5
huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated · 1,318 of 1,319 questions · 13 runs
97.3%
95% CI 96.2–98.0%
1,282/1,318 correct
6
local-inference-lab/Qwen3.8-Flash-Next-NVFP4 · 98 of 1,319 questions · 1 run · Limited data
100.0%
95% CI 96.2–100.0%
98/98 correct
7
zai-org/GLM-5.3-Flash · 303 of 1,319 questions · 3 runs
98.3%
95% CI 96.2–99.3%
298/303 correct
8
unsloth/Qwen3.8-27B-GGUF · 1,116 of 1,319 questions · 11 runs
97.2%
95% CI 96.1–98.0%
1,085/1,116 correct
9
Qwen/Qwen3.6-35B-A3B · 1,277 of 1,319 questions · 13 runs
97.0%
95% CI 95.9–97.8%
1,239/1,277 correct
10
unsloth/Qwen3.6-35B-A3B-GGUF · 1,319 of 1,319 questions · 14 runs
97.0%
95% CI 95.9–97.8%
1,279/1,319 correct
11
unsloth/Qwen3.6-35B-A3B-GGUF · 1,319 of 1,319 questions · 13 runs
96.8%
95% CI 95.7–97.6%
1,277/1,319 correct
12
unsloth/Qwen3.8-27B-GGUF · 912 of 1,319 questions · 9 runs
96.9%
95% CI 95.6–97.9%
884/912 correct
13
openai/gpt-oss-120b · 1,319 of 1,319 questions · 13 runs
96.7%
95% CI 95.6–97.5%
1,275/1,319 correct
14
Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF · 1,319 of 1,319 questions · 13 runs
96.6%
95% CI 95.5–97.4%
1,274/1,319 correct
15
HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive · 1,319 of 1,319 questions · 15 runs
96.5%
95% CI 95.4–97.4%
1,273/1,319 correct
16
HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive · 1,319 of 1,319 questions · 13 runs
96.4%
95% CI 95.2–97.2%
1,271/1,319 correct
17
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 · 913 of 1,319 questions · 9 runs
96.5%
95% CI 95.1–97.5%
881/913 correct
18
nvidia/Qwen3.8-Flash-Next-NVFP4 · 200 of 1,319 questions · 2 runs · Limited data
98.0%
95% CI 95.0–99.2%
196/200 correct
19
nvidia/Qwen3.6-27B-NVFP4 · 1,319 of 1,319 questions · 14 runs
96.1%
95% CI 95.0–97.0%
1,268/1,319 correct
20
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 · 1,319 of 1,319 questions · 13 runs
95.9%
95% CI 94.7–96.8%
1,265/1,319 correct
21
unsloth/gemma-4-31B-it-GGUF · 102 of 1,319 questions · 1 run · Limited data
99.0%
95% CI 94.7–99.8%
101/102 correct
22
Qwen/Qwen3.6-35B-A3B · 1,307 of 1,319 questions · 13 runs
95.9%
95% CI 94.6–96.8%
1,253/1,307 correct
23
Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF · 101 of 1,319 questions · 1 run · Limited data
99.0%
95% CI 94.6–99.8%
100/101 correct
24
HauhauCS/Qwen3.5-27B-Uncensored-HauhauCS-Aggressive · 101 of 1,319 questions · 1 run · Limited data
99.0%
95% CI 94.6–99.8%
100/101 correct
25
Qwen/Qwen3.6-35B-A3B · 1,307 of 1,319 questions · 13 runs
95.3%
95% CI 94.0–96.3%
1,245/1,307 correct
26
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 · 58 of 1,319 questions · 1 run · Limited data
100.0%
95% CI 93.8–100.0%
58/58 correct
27
brandonmusic/GLM-5.3-Flash-tr3-4bpw · 102 of 1,319 questions · 1 run · Limited data
98.0%
95% CI 93.1–99.5%
100/102 correct
28
unsloth/Qwen3.6-27B-GGUF · 102 of 1,319 questions · 1 run · Limited data
98.0%
95% CI 93.1–99.5%
100/102 correct
29
Lottolabs/Qwen3.8-27B-TT-Mixed-BFP4-BFP8-P150 · 101 of 1,319 questions · 1 run · Limited data
98.0%
95% CI 93.1–99.5%
99/101 correct
30
Qwen/Qwen3.6-35B-A3B · 100 of 1,319 questions · 1 run · Limited data
98.0%
95% CI 93.0–99.4%
98/100 correct
31
unsloth/Qwen3.8-27B-GGUF · 100 of 1,319 questions · 1 run · Limited data
98.0%
95% CI 93.0–99.4%
98/100 correct
32
unsloth/gemma-4-26B-A4B-it-qat-GGUF · 98 of 1,319 questions · 1 run · Limited data
98.0%
95% CI 92.9–99.4%
96/98 correct
33
rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP · 303 of 1,319 questions · 4 runs
95.7%
95% CI 92.8–97.5%
290/303 correct
34
palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 · 710 of 1,319 questions · 9 runs
94.6%
95% CI 92.7–96.1%
672/710 correct
35
zai-org/GLM-5.3-Flash · 203 of 1,319 questions · 2 runs · Limited data
96.1%
95% CI 92.4–98.0%
195/203 correct
36
sakamakismile/Qwen3.8-27B-MTP-NVFP4 · 102 of 1,319 questions · 1 run · Limited data
97.1%
95% CI 91.7–99.0%
99/102 correct
37
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 · 102 of 1,319 questions · 1 run · Limited data
97.1%
95% CI 91.7–99.0%
99/102 correct
38
unsloth/Qwen3.5-27B-GGUF · 102 of 1,319 questions · 1 run · Limited data
97.1%
95% CI 91.7–99.0%
99/102 correct
39
cyankiwi/Ornith-1.0-35B-AWQ-INT4 · 101 of 1,319 questions · 1 run · Limited data
97.0%
95% CI 91.6–99.0%
98/101 correct
40
ggml-org/gpt-oss-20b-GGUF · 912 of 1,319 questions · 9 runs
93.4%
95% CI 91.6–94.9%
852/912 correct
41
unsloth/Qwen3.8-27B-GGUF · 100 of 1,319 questions · 1 run · Limited data
97.0%
95% CI 91.5–99.0%
97/100 correct
42
unsloth/gemma-4-26B-A4B-it-qat-GGUF · 100 of 1,319 questions · 1 run · Limited data
97.0%
95% CI 91.5–99.0%
97/100 correct
43
peculiar-ragdoll/Unsloth-Ornith-1.5-35B-A3B · 402 of 1,319 questions · 8 runs
93.8%
95% CI 91.0–95.8%
377/402 correct
44
ornith-ai/Ornith-1.5-35B-A3B-NVFP4 · 102 of 1,319 questions · 1 run · Limited data
96.1%
95% CI 90.3–98.5%
98/102 correct
45
deepseek-ai/DeepSeek-V4-Flash-0731 · 102 of 1,319 questions · 1 run · Limited data
96.1%
95% CI 90.3–98.5%
98/102 correct
46
philbert440/Qwen3.8-27B-W4A16-AWQ · 102 of 1,319 questions · 1 run · Limited data
96.1%
95% CI 90.3–98.5%
98/102 correct
47
unsloth/Muse-Glimmer-30B-GGUF · 102 of 1,319 questions · 1 run · Limited data
96.1%
95% CI 90.3–98.5%
98/102 correct
48
unsloth/gemma-4-26B-A4B-it-GGUF · 101 of 1,319 questions · 1 run · Limited data
96.0%
95% CI 90.3–98.4%
97/101 correct
49
deepseek-ai/DeepSeek-V4-Flash-0731 · 101 of 1,319 questions · 1 run · Limited data
96.0%
95% CI 90.3–98.4%
97/101 correct
50
poolside/Laguna-XS-2.1-INT4 · 101 of 1,319 questions · 1 run · Limited data
96.0%
95% CI 90.3–98.4%
97/101 correct
51
Ar4ikov/KAT-Coder-V2.5-Dev-AWQ-W4A16-ASYM · 99 of 1,319 questions · 1 run · Limited data
96.0%
95% CI 90.1–98.4%
95/99 correct
52
poolside/Laguna-XS-2.1-GGUF · 97 of 1,319 questions · 1 run · Limited data
95.9%
95% CI 89.9–98.4%
93/97 correct
53
unsloth/Muse-Glimmer-30B-GGUF · 101 of 1,319 questions · 1 run · Limited data
95.0%
95% CI 88.9–97.9%
96/101 correct
54
unsloth/gemma-4-12b-it-GGUF · 100 of 1,319 questions · 1 run · Limited data
95.0%
95% CI 88.8–97.8%
95/100 correct
55
prism-ml/Bonsai-27B-gguf · 100 of 1,319 questions · 1 run · Limited data
95.0%
95% CI 88.8–97.8%
95/100 correct
56
bartowski/Ling-3.0-tiny-GGUF · 609 of 1,319 questions · 6 runs
91.3%
95% CI 88.8–93.3%
556/609 correct
57
RedHatAI/Muse-Glimmer-30B-NVFP4 · 102 of 1,319 questions · 1 run · Limited data
94.1%
95% CI 87.8–97.3%
96/102 correct
58
unsloth/Qwen3.8-27B-GGUF · 812 of 1,319 questions · 8 runs
89.4%
95% CI 87.1–91.3%
726/812 correct
59
poolside/Laguna-XS-2.1-NVFP4 · 102 of 1,319 questions · 1 run · Limited data
93.1%
95% CI 86.5–96.6%
95/102 correct
60
Qwen/Qwen3-Coder-30B-A3B-Instruct · 101 of 1,319 questions · 1 run · Limited data
92.1%
95% CI 85.1–95.9%
93/101 correct
61
google/gemma-4-E4B-it · 50 of 1,319 questions · 4 runs · Limited data
94.0%
95% CI 83.8–97.9%
47/50 correct
62
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP-XS · 100 of 1,319 questions · 1 run · Limited data
91.0%
95% CI 83.8–95.2%
91/100 correct
63
google/gemma-4-26B-A4B-it-qat-q4_0-gguf · 102 of 1,319 questions · 1 run · Limited data
89.2%
95% CI 81.7–93.9%
91/102 correct
64
ornith-ai/Ornith-1.0-9B-GGUF · 51 of 1,319 questions · 2 runs · Limited data
88.2%
95% CI 76.6–94.5%
45/51 correct
65
Qwen/Qwen3.6-35B-A3B · 100 of 1,319 questions · 1 run · Limited data
72.0%
95% CI 62.5–79.9%
72/100 correct
66
tencent/Hy-MT2-7B · 50 of 1,319 questions · 1 run · Limited data
68.0%
95% CI 54.2–79.2%
34/50 correct
67
openbmb/MiniCPM5-1B-GGUF · 10 of 1,319 questions · 1 run · Limited data
80.0%
95% CI 49.0–94.3%
8/10 correct
68
Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF · 2 of 1,319 questions · 1 run · Limited data
100.0%
95% CI 34.2–100.0%
2/2 correct
69
ornith-ai/Ornith-1.0-9B · 4 of 1,319 questions · 1 run · Limited data
75.0%
95% CI 30.1–95.4%
3/4 correct
Stability— historical rerun transparency
Leaderboard rank uses the latest approved answer to each question. These numbers also include earlier submissions, so reruns and changed answers are visible but do not affect rank.
13 historical runs · 1,246 unique questions
Ranked
97.8%
Row avg
97.8%
Run avg
98.0%
Repeated
0
Changed
0
20 historical runs · 1,319 unique questions
Ranked
97.6%
Row avg
97.4%
Run avg
97.4%
Repeated
711
Changed
4
13 historical runs · 1,319 unique questions
Ranked
97.3%
Row avg
97.3%
Run avg
97.3%
Repeated
0
Changed
0
13 historical runs · 1,319 unique questions
Ranked
97.3%
Row avg
97.3%
Run avg
97.3%
Repeated
0
Changed
0
13 historical runs · 1,318 unique questions
Ranked
97.3%
Row avg
97.3%
Run avg
97.3%
Repeated
0
Changed
0
1 historical run · 98 unique questions
Ranked
100.0%
Row avg
100.0%
Run avg
100.0%
Repeated
0
Changed
0
3 historical runs · 303 unique questions
Ranked
98.3%
Row avg
98.3%
Run avg
98.4%
Repeated
0
Changed
0
11 historical runs · 1,116 unique questions
Ranked
97.2%
Row avg
97.2%
Run avg
97.2%
Repeated
0
Changed
0
13 historical runs · 1,277 unique questions
Ranked
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
0
Changed
0
14 historical runs · 1,319 unique questions
Ranked
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
101
Changed
0
13 historical runs · 1,319 unique questions
Ranked
96.8%
Row avg
96.8%
Run avg
96.8%
Repeated
0
Changed
0
9 historical runs · 912 unique questions
Ranked
96.9%
Row avg
96.9%
Run avg
96.9%
Repeated
0
Changed
0
13 historical runs · 1,319 unique questions
Ranked
96.7%
Row avg
96.7%
Run avg
96.7%
Repeated
0
Changed
0
13 historical runs · 1,319 unique questions
Ranked
96.6%
Row avg
96.6%
Run avg
96.6%
Repeated
0
Changed
0
15 historical runs · 1,319 unique questions
Ranked
96.5%
Row avg
96.7%
Run avg
96.7%
Repeated
202
Changed
0
13 historical runs · 1,319 unique questions
Ranked
96.4%
Row avg
96.4%
Run avg
96.4%
Repeated
0
Changed
0
9 historical runs · 913 unique questions
Ranked
96.5%
Row avg
96.5%
Run avg
96.5%
Repeated
0
Changed
0
2 historical runs · 200 unique questions
Ranked
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
14 historical runs · 1,319 unique questions
Ranked
96.1%
Row avg
95.4%
Run avg
95.4%
Repeated
102
Changed
10
13 historical runs · 1,319 unique questions
Ranked
95.9%
Row avg
95.9%
Run avg
95.9%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
99.0%
Row avg
99.0%
Run avg
99.0%
Repeated
0
Changed
0
13 historical runs · 1,307 unique questions
Ranked
95.9%
Row avg
95.9%
Run avg
95.9%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
99.0%
Row avg
99.0%
Run avg
99.0%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
99.0%
Row avg
99.0%
Run avg
99.0%
Repeated
0
Changed
0
13 historical runs · 1,307 unique questions
Ranked
95.3%
Row avg
95.3%
Run avg
95.3%
Repeated
0
Changed
0
1 historical run · 58 unique questions
Ranked
100.0%
Row avg
100.0%
Run avg
100.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Ranked
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Ranked
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 98 unique questions
Ranked
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
4 historical runs · 303 unique questions
Ranked
95.7%
Row avg
96.0%
Run avg
96.0%
Repeated
98
Changed
1
9 historical runs · 710 unique questions
Ranked
94.6%
Row avg
94.7%
Run avg
94.7%
Repeated
203
Changed
7
2 historical runs · 203 unique questions
Ranked
96.1%
Row avg
96.1%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
97.1%
Row avg
97.1%
Run avg
97.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
97.1%
Row avg
97.1%
Run avg
97.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
97.1%
Row avg
97.1%
Run avg
97.1%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
0
Changed
0
9 historical runs · 912 unique questions
Ranked
93.4%
Row avg
93.4%
Run avg
93.4%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Ranked
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
0
Changed
0
0 historical runs · 100 unique questions
Ranked
97.0%
Row avg
—
Run avg
—
Repeated
0
Changed
0
8 historical runs · 402 unique questions
Ranked
93.8%
Row avg
93.8%
Run avg
82.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 99 unique questions
Ranked
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 97 unique questions
Ranked
95.9%
Row avg
95.9%
Run avg
95.9%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
95.0%
Row avg
95.0%
Run avg
95.0%
Repeated
0
Changed
0
0 historical runs · 100 unique questions
Ranked
95.0%
Row avg
—
Run avg
—
Repeated
0
Changed
0
1 historical run · 100 unique questions
Ranked
95.0%
Row avg
95.0%
Run avg
95.0%
Repeated
0
Changed
0
6 historical runs · 609 unique questions
Ranked
91.3%
Row avg
91.3%
Run avg
91.3%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
94.1%
Row avg
94.1%
Run avg
94.1%
Repeated
0
Changed
0
8 historical runs · 812 unique questions
Ranked
89.4%
Row avg
89.4%
Run avg
89.4%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
93.1%
Row avg
93.1%
Run avg
93.1%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Ranked
92.1%
Row avg
92.1%
Run avg
92.1%
Repeated
0
Changed
0
4 historical runs · 50 unique questions
Ranked
94.0%
Row avg
94.0%
Run avg
92.2%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Ranked
91.0%
Row avg
91.0%
Run avg
91.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Ranked
89.2%
Row avg
89.2%
Run avg
89.2%
Repeated
0
Changed
0
2 historical runs · 51 unique questions
Ranked
88.2%
Row avg
88.2%
Run avg
82.2%
Repeated
0
Changed
0
1 historical run · 10 unique questions
Ranked
80.0%
Row avg
80.0%
Run avg
80.0%
Repeated
0
Changed
0
1 historical run · 2 unique questions
Ranked
100.0%
Row avg
100.0%
Run avg
100.0%
Repeated
0
Changed
0
1 historical run · 4 unique questions
Ranked
75.0%
Row avg
75.0%
Run avg
75.0%
Repeated
0
Changed
0
