A single-agent system using a 5-sample majority vote outperforms automatic multi-agent setups when inference costs are held equal. Salesforce Research tested six automatic multi-agent frameworks across four models and five benchmarks and found that these systems cost up to 10× more than chain-of-thought with self-consistency (CoT-SC) — and still scored lower.
Sakana AI took a different approach. They ran 70 research campaigns using pre-written evaluation rubrics, then systematically ablated each component to isolate what actually worked.
Datadog showed another path: by training a specialized model on a curated set of real incidents, they matched the performance of GPT-6.1 Sol High and Claude Opus 5.5 High at 3–8× lower estimated inference cost.
Together, these results point to a simple standard: if an agent system can’t demonstrate that it changes the answer — and justify the cost of doing so — it probably shouldn’t be used. We will cover the following in this article with references to oringial researches:
Salesforce cost-controlled results — 6 frameworks, 4 models, 5 benchmarks, and the 10x cost gap.
Five collapse patterns — consensus collapse, role-agnostic configs, verifier bias, workflow collapse, and test-set selection.
SMFR diagnostic design — 588 procedural finance tasks that reward parallel structure, and what the numbers show.
Expert MAS control — a hand-built system lifted GPT-5 from 57.0% to 96.5% at CoT-SC cost.
Sakana rubric discipline — 70 campaigns, 14 model-harness pairs, and ablation before claims.
Datadog golden-set discipline — AlertEval, honest agreement metrics, and cost per investigation.
Builder checklist — 8 checks that separate causal agents from inert ones.
Security implication — every inert agent adds attack surface without changing the decision
