Qwen3.8 27B on Terminal-Bench 2.1: 69.7% xhigh, 78.7% adaptive

Terminal-Bench is a useful counterweight to short-form code benchmarks. A task may require compiling a toolchain, repairing a database, reverse-engineering a binary, editing a large corpus, or keeping several intermediate artifacts coherent across a long shell session. The score is binary and externally verified: the model either leaves the task container in the required state or it does not.

We ran all 89 Terminal-Bench 2.1 tasks locally with btbtyler09/Qwen3.8-27B-GPTQ-4bit, served as qwen3.8-27b, on two RTX 3090 GPUs. The initial setting was xhigh reasoning. That clean xhigh pass solved 62 of 89 tasks, or 69.66%.

We then used a deliberately adaptive fallback policy on only the failures: medium reasoning, then low reasoning, then thinking disabled. Those retries recovered eight additional tasks and produced a 70/89, or 78.65%, cumulative best-of result.

That second number is useful operationally, but it is not a conventional single-attempt benchmark score. The headline quality result is 62/89 at xhigh. The 70/89 result answers a different question: how many tasks can this local model solve when an operator is willing to change reasoning effort after a failure?

Experimental setup

The run used Terminal-Bench's prebuilt task images. LocalMaxxing created a fresh task container, handed that container to the external OMP adapter, preserved the container after the agent exited, and invoked the task's external verifier. Each task result, verifier output, timing record, token usage record, and agent trace was stored under a durable run directory.

The model endpoint exposed a 262,144-token context window. We allowed effectively unbounded whole-experiment wall time and did not use a small turn budget. “Unlimited,” however, cannot mean physically infinite: every model call still had to fit the endpoint context, and pathological trajectories were eventually stopped by the operator when they were no longer making useful progress. During bring-up, completion limits were tested at 32K and 64K; the final adapter retained a finite per-response ceiling so prompt plus completion could remain within the endpoint's context.

Running it with LocalMaxxing CLI

The workflow began with a dry-run. This validates task selection, model identity, quantization metadata, hardware metadata, image resolution, and the output checkpoint before spending days on inference.

lmx eval terminal run \
  --task-dir .tb21-bundles \
  --base-url http://127.0.0.1:8087 \
  --container-base-url http://172.17.0.1:8088 \
  --model btbtyler09/Qwen3.8-27B-GPTQ-4bit \
  --quantization GPTQ-4bit \
  --quant-format GPTQ \
  --hardware hardware.json \
  --agent-cmd ./examples/agents/omp-container-shell.sh \
  --agent-name omp \
  --agent-execution container \
  --thinking-level xhigh \
  --run-dir .terminal-runs/tb21-qwen38-xhigh \
  --resume auto \
  --json-status \
  --json \
  --dry-run

After inspecting the preflight, the same command ran without --dry-run. The durable checkpoint made the multi-day experiment resumable:

lmx eval terminal status .terminal-runs/tb21-qwen38-xhigh --json
lmx eval terminal logs .terminal-runs/tb21-qwen38-xhigh --follow

The fallback sweeps were separate task directories containing symlinks only to the currently failed bundles. Each sweep used its own run directory and changed only --thinking-level. This matters because it keeps every attempt auditable and prevents an adaptive result from being mistaken for a single canonical pass.

Results

The clean xhigh score includes fresh reruns of two attempts that were initially unscored because of harness infrastructure problems. There were no unresolved infrastructure failures in the final accounting.

Tasks recovered by changing reasoning effort

Medium recovered five:

• build-pov-ray
• extract-elf
• financial-document-processor
• large-scale-text-editing
• video-processing

Low recovered two:

• dna-insert
• query-optimize

Thinking disabled recovered one:

• mteb-retrieve

This is the central result of the experiment: higher reasoning effort was not monotonically better on every task. Xhigh was the strongest single full-suite setting we measured, but a cheaper and more direct policy sometimes escaped a failed xhigh trajectory.

The cost of long-horizon reasoning

Across every attempt, including the two initial unscored infrastructure attempts, the experiment consumed:

• 160 attempts, of which 158 were scored
• 117.34 hours of calendar time
• 111.84 aggregate task-hours
• 861.85 million input tokens
• 18.52 million output tokens
• 880.38 million total processed tokens
• 10,821 model calls
• approximately 558 MiB of stored artifacts

The input-token number is much larger than the output-token number because an interactive agent repeatedly resends conversation state, task context, tool results, and accumulated observations. It should not be read as 862 million unique training-style tokens. It is processed inference context across thousands of calls.

The asymmetry also explains why completion-token caps alone do not control cost. A response can be reasonably short while the next request carries a very large accumulated transcript. For long terminal tasks, context management and stopping policy are first-order performance parameters.

What xhigh did well

The successful set was not limited to easy shell manipulation. Xhigh completed tasks involving compilation, cryptanalysis, numerical computing, model tooling, databases, networking, polyglot code, and repository repair. Representative passes included:

• compile-compcert
• feal-differential-cryptanalysis
• fix-ocaml-gc
• mcmc-sampling-stan
• model-extraction-relu-logits
• pytorch-model-recovery
• schemelike-metacircular-eval
• sqlite-with-gcov
• torch-tensor-parallelism
• write-compressor

The strongest trajectories shared a practical pattern: inspect the environment, form a concrete hypothesis, create an intermediate artifact, run a relevant self-check, and stop when the observable requirement passed. The agent was effective when reasoning remained anchored to files, command output, and verifier-relevant behavior.

Failure audit: all 19 residual tasks

We reviewed the instruction, agent transcript, verifier output, and result metadata for every task that failed at xhigh, medium, low, and thinking-off. That is 76 stored residual trajectories, plus the eight xhigh-to-fallback pass contrasts. One original xhigh trajectory was unscored because of the harness problem described below; it was not treated as a model verdict.

The residual attempts were expensive: 73.20 task-hours, 554.53 million input tokens, 12.77 million output tokens, and 6,452 model calls. Xhigh alone accounted for 49.40 of those hours. Six xhigh failures consumed 68.4% of all input tokens used by failed xhigh attempts: make-doom-for-mips, winning-avg-corewars, extract-moves-from-video, regex-chess, path-tracing-reverse, and install-windows-3.11.

The audit found a concrete primary failure in each residual task:

Common pattern 1: investigation displaced delivery

The most frequent failure was not an incorrect final answer; it was failure to reach a final artifact. This happened across rendering, reverse engineering, media extraction, emulation, and code generation. The repeated trajectory was:

1. inspect the input;
2. create a helper, parser, renderer, profiler, or simulator;
3. find an ambiguity;
4. switch representations or deepen the helper;
5. reach timeout or process termination before writing the required path.

More reasoning often amplified this behavior. Xhigh spent 9.07 hours on extract-moves-from-video, 5.82 on path-tracing-reverse, and 4.83 on path-tracing, yet all three ended without the required file. An action-first policy should require a compilable or parseable baseline artifact early, then improve it against the real acceptance metric.

Common pattern 2: self-validation was circular or checked a proxy

Several attempts tested an internally convenient interpretation instead of the external contract:

• chess checked Stockfish against a self-transcribed but incorrect FEN;
• DNA and protein assembly weakened exact representation checks in custom validators;
• Windows and QEMU used custom screen/socket clients rather than the verifier-visible interaction path;
• the HTML sanitizer passed curated strings but not the browser-parsed malformed corpus;
• pipeline parallelism treated parameter-gradient agreement as encouraging even though required activation hooks disagreed.

The fix is not simply “run more tests.” The test must be independent and shaped like the acceptance path. A custom oracle that shares the implementation's assumptions can produce very high confidence in the wrong artifact.

Common pattern 3: unsupported interpretation became a commitment

Qwen often noticed ambiguity, selected a plausible interpretation, and then built increasingly sophisticated work on top of it without forcing an early falsification. Examples include adding a conventional 0x400000 base to a PIE extraction, mapping a modified protein residue to one amino acid, inventing a detector-to-Raman coordinate calibration, and blaming guest stalls on image corruption or a TCG bug without an A/B test.

This is where the lower-reasoning recoveries are most informative. Seven of the eight recovered tasks passed because the fallback chose a corrected technical or data-association path:

• authentic source provenance in build-pov-ray;
• no invented PIE base in extract-elf;
• correct filename-to-document bookkeeping in financial-document-processor;
• consecutive-frame motion rather than background drift in video-processing;
• verifier-matched primer decomposition in dna-insert;
• a lower-cost SQL plan in query-optimize;
• model-native query/passage prompts in mteb-retrieve.

Only large-scale-text-editing was primarily a simpler-stopping win: xhigh had a nearly correct form, debugged for 61 minutes, and left a structurally invalid final script; medium preserved the explicit setreg form and passed.

Across those eight pairs, the passing fallbacks used only 5% fewer model calls and 21% less wall time than the failed xhigh attempts. Lower reasoning did not win merely by being shorter. It won by making a different technical choice and avoiding commitment to an attractive but unsupported interpretation.

Common pattern 4: known-red candidates were still submitted

The strongest evidence of weak trajectory control came from tasks where the model's own measurements already contradicted completion:

• sampler tests reported six failures;
• pipeline self-tests reported backward-hook mismatches;
• MJCF timing repeatedly missed the 60% threshold;
• real pMARS battles showed multiple losing opponent thresholds.

In each case, more diagnosis followed, but the best known candidate was not preserved and the unmet metric did not become a hard blocker. The agent needs an explicit acceptance vector, a retained best candidate, and a rule that every required component must be green before completion.

Harness failures kept separate

Two initial xhigh attempts were invalid because of infrastructure. One adapter path assumed /workspace although the task used /app; another session encountered a stale process after the wrapper changed. We reran both from clean containers and used only those clean outcomes in the 62/89 score. No residual task above is labeled a model failure solely because of those wrapper faults.

The trace audit therefore points to an execution-control problem more than a raw reasoning shortage: build a baseline artifact early, test the exact observable contract, treat hypotheses as hypotheses, preserve the best measured candidate, and stop diagnostic detours when they do not move the acceptance vector.

Reasoning effort is a routing decision

The fallback recovery rates decreased as the residual set became harder:

These rates are not a fair head-to-head comparison of reasoning settings. Medium saw all xhigh failures; low saw only tasks that both xhigh and medium failed; thinking-off saw the hardest remaining residual. The experiment therefore does not establish that medium is globally better than low or off.

It does establish that deterministic escalation in only one direction is wasteful. After a high-effort failure, reducing reasoning can be a legitimate recovery strategy. The right production policy is likely a router that considers task shape and trajectory health:

1. Start with a bounded setting appropriate to the task.
2. Require a concrete artifact or decisive diagnostic early.
3. Detect repeated reads, repeated plans, stagnant files, and rapidly growing context.
4. Stop or restart an unhealthy trajectory before it consumes hours.
5. On retry, vary reasoning effort and start from a clean container.
6. Report the first-pass score separately from any adaptive best-of score.

What the two scores mean

The 69.66% xhigh result is the comparable single-setting result from this run. Every task received the same reasoning level, and the two infrastructure-corrupted attempts were replaced with clean reruns.

The 78.65% adaptive result measures an operator-assisted policy over multiple attempts. It is useful for estimating practical solvability and for designing retry routers, but it benefits from extra compute and post-failure setting selection. It should never be presented as a single-pass Terminal-Bench score.

This separation is more than labeling. Without it, retry-heavy evaluations reward compute without showing how much compute was spent or which choices were made after observing failures. LocalMaxxing's durable per-task results make both views reconstructable.

Practical recommendations

For anyone reproducing this experiment locally:

• Always preflight. Validate all 89 tasks, model identity, quantization, hardware, and prebuilt images before launching.
• Use a durable run directory. Multi-day runs need atomic checkpoints and --resume auto.
• Keep fallback sweeps separate. Never overwrite the canonical first-pass record.
• Treat context as a resource. Track cumulative input tokens, not only generated tokens.
• Use finite per-response limits. “Unlimited” wall time does not remove the endpoint context boundary.
• Add a progress watchdog. A long runtime is acceptable when files and tests are changing; repeated planning with no artifact is not.
• Verify infrastructure failures from clean containers. Do not mix harness repair with answer repair.
• Report compute with quality. Task-hours, model calls, and processed tokens are necessary context for an adaptive score.

Conclusion

A quantized 27B model on two consumer GPUs solved 62 of 89 Terminal-Bench 2.1 tasks in one xhigh pass. An adaptive sequence of lower-reasoning retries raised the practical solved set to 70 of 89.

The surprising lesson was not simply that more reasoning helps. It was that reasoning effort has failure modes. Xhigh unlocked difficult, multi-stage work, but it also enabled a small number of enormous trajectories that kept thinking after the useful work had stopped. Medium, low, and thinking-off settings recovered eight failures precisely because they changed that behavior.

For local agents, the next gains are likely to come from trajectory control as much as raw model quality: earlier tool use, explicit progress detection, bounded context growth, clean retries, and honest separation between first-pass and best-of results. LocalMaxxing CLI made those distinctions visible by preserving the task containers, verifier outcomes, traces, usage, and checkpoints needed to audit a five-day run.