AI agents tax benchmark: Surge AI's Dayjob: Tax finds the top agent passes 10.8% of attempts on 50 professional tax assignments

How good are AI agents at real tax work? A new AI agents tax benchmark from Surge AI, published on October 27, 2026, says not very. Dayjob: Tax gives agents 50 assignments of the kind a tax firm handles, from a family's return to a partnership loss analysis, each with a folder of messy client records. The best result on Surge's leaderboard is Claude Sonnet 5.5, which passed 10.8% of its attempts. Surge's own summary: "Agents haven't yet passed the audit." The results are Surge's, graded by an AI judge against rubrics written by tax experts, and most of the tasks are private, so they can't be checked independently. For how other evaluations of real work have been built, see Vals AI's impossible-refund test and Mercor's map of what benchmarks cover.

This story is based on Surge AI's October 27 post "Introducing DAYJOB: Tax", its DAYJOB: Tax leaderboard, the dataset card on Hugging Face and the evaluation harness on GitHub, checked October 29, 2026. All scores are Surge's.

The short version

• Who ran it: Surge AI, whose site says AI labs have used its datasets and human evaluations. Published October 27, 2026; no author named.
• What it tests: 50 tax assignments across eight practice areas, with 1,789 workspace files. Agents must produce a finished deliverable, not answer a question.
• How it's scored: an attempt passes if it meets at least 95% of the task's rubric. Tax experts wrote the rubrics; Claude Opus 4.8 applies them as an LLM judge, per Surge's harness documentation.
• Results: Claude Sonnet 5.5 10.8%, Claude Fable 5.1 9.6%, Claude Opus 5.5 9.2%; no other model passed more than 6%. Three models scored 0%.
• What's missing: confidence intervals, an error breakdown, dollar costs in the text and a limitations section. Only 2 of the 50 tasks are public.

What is Surge AI's Dayjob: Tax benchmark?

It belongs to what Surge calls DAYJOB, "Surge AI's benchmark family for long-horizon professional agents," according to the dataset card. The tax edition "evaluates whether AI agents can take a tax engagement from an initial request to a completed deliverable."

Each task starts with a client request and a workspace of records. The agent has to produce a return, a workpaper, a technical review or written advice. Surge says the benchmark tests professional judgment, such as spotting issues the client didn't raise, not just tax knowledge or arithmetic.

The 50 assignments cover eight practice areas, as Surge lists them:

1. Individual and family tax
2. Small-business tax
3. Partnerships
4. Property and investments
5. Estates, gifts and trusts
6. International tax
7. Corporate transactions
8. Tax review and disputes

The workspaces hold 1,789 files: 1,487 PDFs, 117 Excel workbooks and 38 Word documents, plus emails, images, CSVs, JSON records and text files. The median assignment has 30 files and the largest has 123. Surge says they include "filed and draft tax returns, K-1s, W-2s, IRS notices, general ledgers, payroll records, contracts, and correspondence." Its examples of harder evidence include a handwritten receipt for resale merchandise, a field dispatch log showing a client's crews worked in five states when the client thought it operated only in Alabama, and a cross-border household's Polish employment records reconciled against cryptocurrency exchange exports.

Surge places Dayjob: Tax beside three earlier tax evaluations. In its framing, TaxBench asks whether a model can answer a hard tax question (114 questions), TaxCalcBench whether it can calculate an accurate return, and TaxLab whether an agent can gather missing information from a simulated taxpayer and file a federal individual return. Dayjob: Tax adds deliverables such as partnership loss analyses, estate returns and strategy letters. "The central distinction is how much responsibility remains with the agent," Surge writes.

Which AI models and agent setups did Surge test?

Surge's leaderboard lists 16 models, each with a reasoning setting. The three Claude models ran at "Adaptive/Max". Most others ran at "Max reasoning"; Grok 4.7 and Qwen 3.8 Max at "xHigh"; and Gemini 4 Argon, Gemini 3.8 Flash, Hy4 Preview and Mistral Large 4 at "High". Surge doesn't explain why the settings differ.

The agent setup comes from Surge's public harness on GitHub, not from the blog post:

• Harness: tasks use the Harbor format, with instructions, a containerized working environment and evaluation code. The README says "The harness runs models through the OpenHands SDK."
• Environment: a Linux environment "equipped for document and spreadsheet processing." Network access is limited to the model's API host; "browsing and package downloads are blocked."
• Attempts: "Leaderboard runs use 5 attempts per task."
• Errors: trials that errored "are excluded from the mean."

So every model used the same open-source agent scaffold rather than its maker's own agent product, such as Claude Code or Codex. That makes the comparison more even, but it also means the scores describe these models inside OpenHands, not how any vendor's own tool would perform.

How were the tax assignments graded, and by whom?

Tax experts built the tasks. Surge says they turned real-world situations into assignments, "developing the supporting records, expected outcomes, and grading criteria," and that each task went through "a three-layer expert review for correctness, completeness, and solvability." The post profiles three of the experts by background, including a tax attorney with more than 10 years at PwC, Deloitte and Exxon, and a Baker Tilly tax director. It doesn't say how many experts took part.

The experts didn't grade the agents' work, though. Surge's GitHub README says: "Grading uses Claude Opus 4.8 as an agentic LLM judge (OpenCode) running inside the task container." The judge checks each attempt against that task's rubric, which covers "calculations, supporting evidence, consequential issues, and the completed deliverable," according to the post.

The pass rule is strict. Surge reports "pass@1 at a 95% rubric threshold": an attempt passes only if it satisfies at least 95% of the task's criteria. An attempt that gets the numbers right but misses one consequential issue can fail.

What were the scores?

Claude Sonnet 5.5 led with 10.8%, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. Behind them, Muse Wonder 1.0 scored 6%, Grok 4.7 5%, GPT-6 Astra 4.8%, GPT-6.1 Sol 4%, Muse Spark 1.3 3.6% and Gemini 4 Argon 3.2%. Hy4 Preview scored 1.6%, DeepSeek V4.1 Flash 1017 1.2%, Gemini 3.8 Flash 0.8% and GLM 5.3 0.4%. Kimi K3, Mistral Large 4 and Qwen 3.8 Max each scored 0%.

Dayjob: Tax pass rate, top 12 of 16 models

percent of attempts passing at a 95% rubric threshold (pass@1)

Claude Sonnet 5.510.8%

Claude Fable 5.19.6%

Claude Opus 5.59.2%

Muse Wonder 1.06%

Grok 4.75%

GPT-6 Astra4.8%

GPT-6.1 Sol4%

Muse Spark 1.33.6%

Gemini 4 Argon3.2%

Hy4 Preview1.6%

DeepSeek V4.1 Flash 10171.2%

Gemini 3.8 Flash0.8%

Show the data

model	pass_rate_percent
Claude Sonnet 5.5	10.8%
Claude Fable 5.1	9.6%
Claude Opus 5.5	9.2%
Muse Wonder 1.0	6%
Grok 4.7	5%
GPT-6 Astra	4.8%
GPT-6.1 Sol	4%
Muse Spark 1.3	3.6%
Gemini 4 Argon	3.2%
Hy4 Preview	1.6%
DeepSeek V4.1 Flash 1017	1.2%
Gemini 3.8 Flash	0.8%

Two points about reading these numbers. First, the gaps at the top are small. With 50 tasks and no confidence intervals, a 1.2-point gap between Sonnet 5.5 and Fable 5.1 may not mean much. Second, Surge also charts each model's performance against its "average cost per trial" and says "Claude Sonnet 5.5 and Opus 5.5 occupy the high-performance end of the measured Pareto frontier." The chart is an image, and the post's text gives no dollar figures, so the cost side can't be quoted.

On Hacker News, where the post drew more than 200 points on October 28, one commenter said they were surprised that "Sonnet 5.5 did noticeably better than Opus and Fable." Surge doesn't offer an explanation.

What mistakes did the AI agents make?

Surge doesn't publish error categories or how often each kind of error happened. Instead, it gives two worked examples, each based on five attempts per model:

• A missed deadline. A client, Anna, moved out of her condo in August 2024 and was choosing between closing in October 2026 or January 2027. An agent correctly found that either date met the residence requirement for excluding up to $250,000 in gains, but didn't mention that she would stop qualifying around the end of August 2027. "Across five attempts, Astra never identified the deadline. Opus identified it in all five attempts," Surge writes. (The post says only "Astra"; the leaderboard's Astra entry is GPT-6 Astra.)
• An unverified assumption. A separating couple asked how to divide a Roth IRA and a 401(k). Opus 5.5 told them, "You aren't legally married," and built its advice on that. The records suggested otherwise: the couple had lived in Texas, which recognizes informal marriages under some conditions, and held a ceremony in Colorado. "Across all five attempts, Opus based its recommendations around the assumption that the couple was unmarried," Surge says. Astra flagged the need to confirm their marital status in four of five attempts.

The examples cut both ways. Opus caught the deadline Astra missed, and Astra caught the question Opus skipped. They show the kind of failure Surge is measuring: not wrong arithmetic but missing or unchecked issues that a tax professional would raise. Surge's conclusion is that "exercising professional judgment and reliably carrying out real-world tax workflows remain out of reach."

What are the limits of the Dayjob: Tax results?

Surge's post has no limitations section. The dataset card adds two caveats: the two public samples "illustrate the task format and workflow" and results on them "do not constitute a score on the full benchmark," and Surge asks that the tasks and rubrics be kept out of training data. Beyond that, the published material leaves several open questions:

• An AI judge, not a human one. Experts wrote the rubrics, but an LLM decides whether each criterion is met. Surge publishes no agreement rate between the judge and human tax professionals.
• The judge and the leaders share a maker. The judge, Claude Opus 4.8, is an Anthropic model, and the top three scores are Anthropic models. Surge doesn't address whether that matters. Nothing published shows a bias, but nothing rules one out either.
• Small and mostly private. 50 tasks, 5 attempts each, no confidence intervals, and only 2 tasks public. Outsiders can't rerun the full benchmark.
• Uneven settings. Reasoning levels differ across models: four ran at "High" while most ran at "Max."
• One harness. All models ran in OpenHands with browsing blocked. Vendors' own agent products, or agents allowed to look things up, could behave differently.
• Errors dropped. Trials that errored are excluded from the mean, and Surge doesn't say how many there were per model.
• Surge's business. Surge sells data and evaluations to AI labs. Its own site highlights that OpenAI, Anthropic and Microsoft have cited or used its datasets and human evaluations. The post carries no conflict-of-interest note.

What the benchmark does establish is narrower but useful: under Surge's rules, no tested model reliably completes a professional tax assignment end to end, and the failures it shows are judgment failures. That fits a pattern across evaluations of real work, from what coding-agent benchmarks measure and don't to Vals AI's findings on multi-agent coding.

What does this mean for using AI agents for taxes?

Surge's results don't show any tested agent completing professional tax work reliably on its own. Even the top model failed about nine attempts in ten under a strict standard, and Surge's examples show agents giving confident advice built on facts they never checked. The benchmark doesn't measure narrower uses, such as asking a model to explain a rule or to double-check a single calculation, and it doesn't compare agents with human preparers.

What's confirmed, reported and uncertain

• Confirmed (Surge AI post, October 27, 2026): 50 assignments, eight practice areas, 1,789 files and the file mix; the expert-built tasks and three-layer review; the 95% pass threshold; the top three scores; the condo and couple examples.
• Confirmed (Surge leaderboard, read October 29, 2026): all 16 models, their reasoning labels and pass rates.
• Confirmed (Surge's GitHub harness and Hugging Face card): OpenHands harness, blocked browsing, 5 attempts per task, Claude Opus 4.8 as LLM judge, errored trials excluded, 2 of 50 tasks public.
• Uncertain: how closely the LLM judge agrees with human tax experts; how many trials errored per model; the dollar cost per trial; why some models ran at "High"; whether results would hold on a larger or public task set.