Suppose two AI agents score 80% and 74% on the same benchmark. Which one is better? Now suppose the first received ten attempts, eight parallel agents, and thirty minutes per task. The second received one attempt, one agent, and five minutes. Are we still comparing models—or are we comparing compute budgets?
Those numbers are illustrative, but the measurement problem is real. A statement such as “this agent solved 80% of the benchmark” is incomplete unless the evaluator also says: with how much compute, how much time, how many retries, how many parallel agents, and what tool access?
An AI benchmark score is not simply a property of the model. It is a property of the model operating under a specific budget of compute, time, tools, and retries.
Why a benchmark score is incomplete
The UK AI Security Institute (AISI) has now made the issue explicit. Its 2026 work found that fixed budgets can systematically understate frontier capability and that newer systems can benefit disproportionately from additional inference. On the cyber and software-engineering tasks analyzed, longer tasks also tended to require more compute. AISI’s accompanying technical paper concludes that benchmark scores are protocol-dependent and recommends comparing systems across matched compute budgets rather than at one restrictive cut-off.
This fits a broader evaluation principle. NIST’s August 2026 initial public draft of the TEVV-Athlon Framework calls for clear objectives, resources, conditions, baselines, uncertainty, and context of use. The Canadian AI Safety Institute likewise asks evaluators to disclose the model or system configuration, methodology, limitations, uncertainty, and relationships that could affect interpretation. Neither framework says every evaluator needs exact FLOP accounting. Both support the more important point: conditions are part of the evidence.
What test-time compute means
Training compute builds the model. Test-time compute—also called inference-time compute—is what the system spends while answering or acting. It can include reasoning tokens, inference calls, retries, best-of-N sampling, planning and reflection loops, search depth, parallel agents, tool calls, code execution, browsing or API interactions, elapsed task time, and approximate inference cost.
A useful conceptual representation is:
P = P(M, C, T, R, A, U, E)
Here M is the model; C, compute; T, time; R, retries; A, agents or parallelism; U, tool access; and E, the evaluation environment. This is a conceptual evaluation model, not a universal mathematical law.
Agentic AI amplifies the issue
A conventional prompt may involve one inference. An agent can reason, act, observe, revise, retry, call tools, spawn another agent, and repeat. Capability is generated by a system repeatedly reasoning and acting, not just by one model response.
Model capability is what a base model achieves under a defined budget. System capability is what the full stack achieves with memory, planning, tools, retries, parallelism, and orchestration. A weaker base model may legitimately outperform a stronger one through better engineering. The requirement is transparency: a “model benchmark” is increasingly a “system benchmark.”
The Capability–Compute Curve
A single score hides how performance changes as resources increase. A Capability–Compute Curve reports measured performance across several test-time budgets. It exposes marginal gains, efficiency, plateaus, operating cost, and whether rankings change at equal budgets.
| Test-time budget | Performance |
|---|---|
| Low | 42% |
| Medium | 61% |
| High | 73% |
| Very high | 78% |
Compute-normalized evaluation
Models should be compared at equivalent or explicitly disclosed resource budgets whenever the comparison is meant to inform procurement or operations. The normalization basis depends on the decision: equal cost for a high-volume workflow, equal elapsed time for incident response, equal tokens or inference calls for research, or equal tool calls and retries where external actions dominate.
There is no universally correct denominator. The defensible one mirrors the constraint the organization will face in production.
An illustrative procurement reversal
| Condition | Agent A | Agent B |
|---|---|---|
| Published score | 82% | 76% |
| Attempts / problem | 10 | 1 |
| Time limit | 30 min | 5 min |
| Parallel agents | 8 | 0 |
| Average cost / task | $4.80 | $0.18 |
| Equal five-minute re-run | 72% | 76% |
The leaderboard says Agent A wins. The equal-latency comparison says Agent B wins. At 100,000 tasks a month, the stated average costs imply $480,000 for A and $18,000 for B before discounts, infrastructure overhead, or human review. An organization may still choose A when the extra capability is worth the price. But it should make that decision knowingly.
Procurement should ask for the budget
A benchmark that ignores resource consumption can reward brute-force inference rather than better engineering. More compute is not bad; undisclosed or incomparable compute is. Procurement teams should ask:
- System. What exact model, version, provider, prompts, scaffolding, and orchestration were tested?
- Attempts. How many attempts were allowed, was best-of-N used, and were failed or abandoned runs included?
- Budget. What were the reasoning-token, inference-call, retry, timeout, and parallel-agent limits?
- Access. Which tools, network paths, permissions, search depth, and human interventions were available?
- Economics. What were average task cost, total evaluation cost, latency, throughput, concurrency, GPU demand, energy assumptions, API pricing, and rate limits?
- Production realism. Did the evaluation reproduce production security controls, approval gates, rate limits, and failure handling?
Enterprise buyers often care less about the highest possible score than the highest defensible capability per unit of operational cost. The related question is Marginal Capability Cost: how much additional cost or compute buys the next increment of measured capability? That is a decision metric, not a universal equation. A near-maximum score at a fraction of the cost may be the stronger operating choice.
Security evaluations need high-budget stress tests
Additional compute can increase planning depth, persistence, search, recovery from failure, vulnerability discovery, and autonomous exploration. An agent that fails within five minutes may succeed with five hours. Risk evaluations should therefore disclose and stress-test the computational resources available to the agent, alongside its tools and permissions.
This connects two distinct assurance questions. Independent AI Evaluation Assurance asks whether the conditions and evidence justify the performance claim. Agentic AI Containment Assurance asks what the deployed system can reach, change, or damage before it can be stopped. Compute-aware evaluation strengthens both without collapsing one into the other.
Maple Quanta Benchmark Compute Disclosure
| Field | Disclosure |
|---|---|
| Model/version | Exact system and provider tested |
| Inference budget | Calls, tokens, or compute proxy |
| Task time limit | Maximum duration |
| Retry policy | Attempts allowed and selection method |
| Parallelism | Number of agents or processes |
| Tool budget | Available tools and call limits |
| Human assistance | Any intervention or curation |
| Average task cost | Approximate |
| Total evaluation cost | Approximate |
| Production constraints | Whether they were reproduced |
| Results | Score, variance, failures, and resource use |
This should remain practical. Exact compute measurement may be impossible with proprietary APIs, batching, caching, heterogeneous hardware, hidden reasoning, and inconsistent provider accounting. Useful proxies include elapsed time, visible tokens, inference calls, retries, tool calls, and cost. The objective is transparent resource accounting, not false precision.
The Maple Quanta perspective
Maple Quanta does not only ask, “What score did the model receive?” It asks: “Under what conditions was that score produced, and does that evidence justify the decision being made?” The evaluation record should connect the exact model and configuration, the resource budget, the operating environment, and the result—including variance, failures, and consumption.
AI capability is not measured independently of resources. As agents gain more inference, time, tools, and parallelism, benchmark performance increasingly becomes a property of the complete computational system.
If benchmark results are meant to guide procurement, governance, or risk decisions, compute conditions must become part of the evidence. The next time someone tells you an AI scored 80%, ask the question the leaderboard usually leaves out: how much compute did it get?
Sources
Source note. AISI, NIST, and the Canadian AI Safety Institute are government sources. The AISI paper and additional agent-scaling papers are technical research; their findings are task- and protocol-specific and do not establish a universal scaling law. NIST AI 200-2 is an initial public draft, not final guidance.
- UK AI Security Institute. “More compute, more capability: Why AI agent evaluations need to account for test-time compute.” 2026.
- McFadyen et al. “How Inference Compute Shapes Frontier LLM Evaluation.” arXiv:2606.17930, revised July 2026.
- National Institute of Standards and Technology. The TEVV-Athlon Framework for Evaluating AI Systems, NIST AI 200-2 ipd, August 2026.
- Canadian AI Safety Institute. “What information should AI evaluators share?” 2026.
- Zhu et al. “Scaling Test-time Compute for LLM Agents.” arXiv:2506.12928, revised March 2026.
- Li et al. “Benchmark Test-Time Scaling of General LLM Agents.” arXiv:2602.18998, 2026. This paper provides an important counterpoint: more sequential or parallel compute can plateau or degrade when context and verification fail.
Disclosure: Maple Quanta Inc. provides AI evaluation, assurance, governance, agentic containment, and technical auditing services. Readers should consider this commercial context when assessing the analysis. This Insight is for general informational purposes only and does not constitute legal, procurement, cybersecurity, investment, or regulatory advice.