Multi-agent vs. single-agent coding: Vals AI finds agent teams cost 1.8× to 5.1× more for one significant gain

Should a coding agent work alone, or should it split the job across a team of subagents? On October 9, 2026, the independent evaluator Vals AI published one of the cleaner tests of that question so far. It ran GPT-6 Sol and Claude Opus 5.5 on the same 50 web-app builds from its Vibe Code Bench, once as a single agent and once as a team, at medium and max reasoning effort. Teams cost 1.8× to 5.1× as much per app. Only one of the four team-vs-single comparisons produced a statistically significant quality gain.

This article is based on Vals AI's post "Do Agent Teams Pay Off? A Case Study on Vibe Code Bench," by Katrina Drozdov. All numbers below are Vals AI's.

The short version

• Who ran it: Vals AI, an independent evaluation company. Published October 9, 2026.
• What was tested: GPT-6 Sol (OpenAI Agents API) and Claude Opus 5.5 (Claude Code CLI, headless), as single agents and as teams, each at medium and max effort: eight setups, one run each.
• Benchmark: 50 full-stack web apps from Vibe Code Bench, graded by browser agents running UI tests.
• Result: teams significantly beat single agents only for Sol at medium effort (+7.3 points, p = 0.005). Sol at max, Opus at medium and Opus at max showed no significant difference.
• Cost: teams cost 2.5× (Sol medium), 1.8× (Sol max), 2.2× (Opus medium) and 5.1× (Opus max) as much as the matching single agent.
• Takeaway from Vals: "a team can boost performance, but higher reasoning effort is an alternative worth weighing."

What is Vibe Code Bench?

Vibe Code Bench gives an agent a product spec and asks it to build a complete web app. Each app is deployed with Docker Compose. Browser agents then click through its workflows and run a set of UI tests. An app's score is the share of its UI tests that pass, and the benchmark score is the average across the 50 apps.

That makes it closer to real product work than benchmarks built on single functions or isolated bug fixes. It also means scores depend on the whole pipeline: planning, building, wiring parts together and checking that features actually work. For background on how this kind of benchmark differs from SWE-bench-style tests, see our explainer on coding-agent benchmarks.

How the team setups worked

• Team structure: a lead agent could spawn up to five subagents at once. Subagents couldn't spawn their own.
• Instruction: both team leads got the same short prompt: split the work into independent parts, delegate, then integrate and verify.
• GPT-6 Sol ran through the OpenAI Agents API, with the multi_agent option off for single agents and on for teams.
• Claude Opus 5.5 ran through the Claude Code CLI in headless mode. Single-agent runs had the subagent tools removed. Team runs also had Claude Code's ten-minute limit on background subagents disabled.

Results

Source: Vals AI, October 9, 2026. One run per setup on the same 50 apps. Costs are Vals AI's mean cost per app.

Which differences are real?

Vals compared each team with its matching single agent using a paired t-test across the same 50 apps:

The three non-significant comparisons had p-values between 0.16 and 0.81, according to Vals. In plain terms, for three of the four pairs, the gap between team and single agent is within the range you'd expect from app-to-app variation alone.

Effort vs. teams

The more interesting comparison may be teams against higher reasoning effort.

• Sol: moving the single agent from medium to max effort added 11.4 points (p < 0.001), more than the team added at medium. Sol's max-effort single agent beat its medium-effort team by 4.1 points, though that difference wasn't significant (p = 0.08), and it cost 1.6× as much and took 3× as long.
• Opus: max effort didn't help the single agent (−1.7 points, p = 0.48, not significant). Its cheapest setup, medium single at $4.08 per app, scored about as well as its most expensive, max team at $122. The max team cost about 30× as much as the medium single agent.

Vals' conclusion: "Neither option was a clear default."

Where the extra money went

According to Vals, most of the extra team spend was cached input tokens. Each subagent re-reads its own context on every model call. For Opus at max effort, subagents read a median of 224 million cached tokens per app, against 17 million for the lead and 55 million for the single agent. Cached tokens are cheap per token, but at that volume they add up. Our explainer on model routing covers why the cost of an agent depends on how it uses tokens, not just the per-token price.

Runtime moved in different directions for the two models:

• Sol's teams were faster at both effort levels (11.9 vs. 12.4 minutes at medium, 27.6 vs. 38.4 at max).
• Opus's teams were slower, up to 2.3× the single agent at max effort. The median max-effort Opus subagent ran for 46 minutes, against 7 minutes at medium. Four Opus max-team runs exceeded four hours, and one hit the 5.25-hour limit while still verifying. Vals estimated that run's roughly $189 cost from token counts and included it in the mean.

Vals also found that "longer runs did not score higher."

Why the Sol team helped at medium effort

The one significant gain had a specific cause. Sol's medium-effort team beat its single agent on 24 apps, lost on 11 and tied on 15. Gains averaged about 15 points on harder apps and about 1 point on easier ones.

On the nine apps where the team gained 20 points or more, Vals says the single agent "reported that testing had passed even when some features were never built." The team built more of the spec and checked separately built parts against each other. In other words, the team mostly fixed a verification failure. Sol at max effort largely fixed it on its own, which helps explain why the team added little there.

The two models also organized their teams differently:

• Sol's lead split the work along architectural lines, created subagents within about a minute and did much of its own browser testing.
• Opus's lead usually wrote a shared CONTRACT.md first, then delegated in waves: builders, then feature agents, then testers. At max effort it used 6.8 subagents per app, ran a testing subagent in all 50 runs and made about 1,140 subagent tool calls per app, against about 270 tool calls for its single agent.

Limits of the study

Vals lists several limitations:

• One run per setup. The confidence intervals reflect variation across apps, not across repeated runs of the same app.
• Prompt confound. Team runs also got the delegation instruction, so each comparison is "subagents plus a delegation prompt" against neither.
• Different harnesses. Sol ran in the OpenAI Agents API and Opus in Claude Code, so a cross-model comparison mixes model and harness. This story compares each model only with itself.
• Visibility gaps. Sol's event stream only shows the lead's turn, and Sol's costs assume reported token usage includes subagents. Long-context pricing above 272K tokens couldn't be evaluated.
• One benchmark. Full-stack web apps may not generalize to other work.

Vals expects teams to help most on tasks with many independent parts that can run side by side without coordination costs eating the time saved, such as large codebases, broad research questions or analyses spanning many documents.

What builders should take from it

1. Don't default to multi-agent. On this benchmark, a single agent at the right effort level matched or nearly matched a team at a fraction of the cost. Our explainer on what "agentic" means covers when more agent machinery is worth the added complexity.
2. Try higher effort before a team. For Sol it was the bigger lever. For Opus neither helped much, which suggests checking whether you're paying for capacity the task doesn't need.
3. Fix verification first. The team's one big win came from catching features the single agent claimed to have built but hadn't. A dedicated test step or a reviewer pass may get much of that benefit more cheaply.
4. Watch cached-token volume. Subagents re-reading their own context drove most of the cost. Measure it before you scale a team up.
5. Test on your own task mix. The gains clustered on harder apps. If your work splits cleanly into independent parts, your results may differ.

What's confirmed, reported and uncertain

• Confirmed (Vals AI, October 9, 2026): setups, harnesses, scores, costs, runtimes, statistical tests and p-values, cost drivers, team behavior and stated limitations.
• Uncertain: how much of each result is run-to-run noise, since each setup ran once; whether the delegation prompt, rather than subagents, drove some of the effect; and whether results carry over to other tasks or harnesses.
• Not measured: other models, other team sizes and non-web-app work.