Coding agents on one model

Four agent configurations, one model, 25 SWE-bench Verified instances. Resolve rates: 19, 19, 21, 19 of 25. The spread is inside measured run-to-run variance, so this run does not rank them.

Result

Qualifications:

• opencode 19/25 is a lower bound. Five runs were terminated at the 30-minute budget by the stall described below and count as failures. Two of those five resolve their instance when the stall does not occur, putting opencode at 21/25 or better. The table keeps 19 because every agent gets one attempt per instance.
• localcode cost is unavailable. Its usage events carry token counts without a cache-read split, and cached input tokens are priced about an order of magnitude below uncached ones. No dollar figure is derivable from what it emits.
• Stall counts: opencode 5, Claude Code 0, localcode 0 (both configurations).

Per-instance results

Instance ids link to the upstream pull request that fixed the issue, which is the reference each patch was scored against.

Instance failed by all four: django__django-12193

• Reported symptom: incorrect rendering in Django's SplitArrayWidget.
• Actual cause: CheckboxInput.get_context mutating the attrs dictionary passed to it.
• Reference fix: the mutation in CheckboxInput.get_context.
• All four configurations: a copy at the call site, leaving the mutation in place.
• Hidden test: exercises CheckboxInput directly, so the call-site change does not pass it.

- self.widget.get_context(name + '_%s' % i, widget_value, final_attrs)
+ self.widget.get_context(name + '_%s' % i, widget_value, final_attrs.copy())

All four configurations produced the same incorrect fix. The shared model is a possible cause, but this run does not isolate the cause.

Setup

Two phases, run separately:

• Phase A: each agent works inside that instance's official SWE-bench evaluation image, where the repository's test environment is already installed. Only the resulting git diff is kept.
• Phase B: the official SWE-bench harness applies the dataset's test_patch, overwriting any test file the agent touched, then runs FAIL_TO_PASS and PASS_TO_PASS. Editing tests to make them pass therefore gains nothing.

Conditions held equal across all four configurations:

• Model: one id, over the Anthropic API directly, no proxy.
• Reasoning effort: no effort flag was passed to any tool. Each one's outgoing API request was captured to establish what it sends by default: Claude Code: output_config.effort: high and thinking: adaptive; localcode: thinking: adaptive, no effort field; opencode: neither field. Omitting the thinking parameter was confirmed against the API to still produce adaptive thinking on this model, and an omitted effort takes the documented default of high, so the three resolve to the same setting. The API does not echo effort in responses, so that last step rests on the documented default rather than on a measurement here.
• Reply cap: not equal. Each tool sets its own max_tokens: 64,000 for Claude Code, 32,768 for localcode, 32,000 for opencode. No patch in this run was truncated by it.
• Home directory: rebuilt empty per run, so no host configuration, plugin, hook, credential file or prior memory reaches any agent (details).
• User: the same unprivileged account for all four. Claude Code refuses to bypass permissions as root.
• Prompt, budget and permissions: identical text, 30 minutes per run, and permission prompts fully disabled.
• Agent defaults: sub-agent and orchestration features off, except in the one configuration labelled Smart Agent on.

Fresh home directory per run

All three tools keep state in the home directory: credentials, plugins, hooks, custom commands, memory index, session list. The machine used here carried an opencode configuration with a local-model provider and a plugin, and a Claude Code installation signed in to a subscription. Mechanism:

AGENT_HOME=/tmp/agenthome
rm -rf "$AGENT_HOME"; mkdir -p "$AGENT_HOME/.config/opencode"
cp /bench/opencode.json "$AGENT_HOME/.config/opencode/opencode.json"
chown -R "$BENCH_USER" "$AGENT_HOME" "$REPO" "$OUT"

setpriv --reuid=... --regid=... --init-groups \
    env HOME="$AGENT_HOME" PATH=/agents/node/bin:... \
        ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
    python /bench/runagent.py --out "$OUT" -- "$@"

• Container started with --rm from the stock instance image, so no state survives a run or arrives from the previous one.
• Host home directory never mounted; the only bind mounts are the read-only agent toolchain, the read-only prompt and config directory, and the output directory.
• HOME set in the environment passed to the agent process rather than inherited, so passwd-based and $HOME-based lookups resolve to the same empty directory.
• API key supplied as an environment variable, so no credentials file is written.
• One file placed deliberately: opencode's permission rule, whose equivalent for the other two is a command-line flag.

Sources

• Tasks: SWE-bench/SWE-bench_Verified: 500 instances, each with issue text, base commit, reference patch and two test sets.
• Environments: official per-instance evaluation images, swebench/sweb.eval.x86_64.<instance>.
• Scoring: the SWE-bench harness, version 5.0.2, unmodified.
• “Verified”: 500 human-screened instances from SWE-bench's original 2,294.

Instance selection

• Deterministic, fixed in code before any agent ran; re-running the selector reproduces the same 25 instances.
• Drawn round-robin across repositories, ordered within each repository by a hash of the instance id. Sorting and taking the first N returns an alphabetical sample that omits whole projects.
• Difficulty mix: 8 under 15 minutes, 13 15 minutes to an hour, 4 one to four hours.
• Repository coverage: all 12 projects in the dataset. Django is 46% of SWE-bench Verified and 3 of these 25.
• Difficulty weighted above the dataset's own mix (39% / 52% / 8% / 1%). These rates are therefore not comparable to published leaderboard numbers.

opencode stall

Characteristics, from opencode's own event stream:

• Three to seven tool calls execute normally, reading and searching the repository.
• Event timestamps span 10 to 22 seconds, then the stream stops.
• No output for the remaining ~1,780 seconds until the budget expires.
• Final event is a step boundary following a successfully completed tool call.
• stderr empty: no exception, no rate-limit message, no reconnect attempt.
• Second form, on runs that succeed: the process prints its final answer, emits its terminating event, and does not exit. The same command exits in about three seconds outside the container.
• Serial re-run of the five stalled cells: two completed (76 s, 161 s), three stalled again. Contention is a contributing factor, not the whole cause.

Limits

• Sample: 25 instances, one run per configuration per instance.
• Repeat-run observations: among instances where configurations disagreed, repeated runs under identical conditions reversed the verdict in four of nine attempts. This selected subset does not estimate the reversal rate across all 25 instances.
• Conclusion: this run does not establish a reliable ranking. Each instance would need multiple runs to estimate uncertainty.
• Difficulty weighting: harder than the dataset mix, so absolute rates are not comparable to leaderboard results.

Smart Agent on versus off

Result: +2 instances, and 29 seconds faster per run, at the same number of tool calls. Not separable from noise at this sample size.

What the switches enable:

• Smart Agent: the session's model becomes an orchestrator over six built-in specialists (explore, librarian, oracle, plan, implement, verify), each routed to a model profile; plus profile fallback chains, a per-turn trace log, prompt-cache markers, and credential and workspace guards.
• Orchestrate: a plan of delegated stages authored by the model against a schema and executed by localcode.
• Both off by default, and off in every other row on this page. Both enabled together in the Smart Agent on configuration.

Confounds and observations:

• Not a single-switch difference: Smart Agent requires a profile roster, so that configuration carries three profiles against one. The profile the main conversation runs on is matched (model, reply cap, effort), as are binary, command, prompt, budget and permissions; the specialists run on a profile the other configuration has no equivalent of.
• Different outcomes on six instances: Smart Agent on resolved seaborn-3187, requests-2931, pytest-10356 and sphinx-8056; Smart Agent off resolved scikit-learn-25747 and sympy-13798.
• Accounting gap: with Smart Agent on, localcode's usage events report input_tokens: 2 rather than the context sent, so that configuration has no comparable input-token total.