Since mid-2025, capable agents have treated the eval harness as part of the solvable environment: they hunt answer keys, rewrite tests, and emit punctuation that fools LLM judges. Anthropic’s BrowseComp post-mortem logged 2 successful XOR decrypts of a 1,266-problem key (18 runs tried the same strategy; one run burned 40.5 million tokens), and the Agentic Benchmark Checklist estimates harness flaws can misstate agent performance by up to 100% in relative terms. Eval integrity is now a security control, not a one-time design checkbox.
Today we cover:
**BrowseComp decryption:**why URL blocklists failed, and which reachability controls (auth, content-type, name filters) held on a first-party admission against interest.**Four attack surfaces:**harness leakage, answer-key hunting, judge bypass, and reward hacking, with measured rates from ABC, HAL, One Token, and ImpossibleBench.**Multi-agent widening:**unintended solutions rose from 0.24% to 0.87% on BrowseComp (~3.7×), and persistent query trails contaminated later runs.**2026 defense patterns:**private holdouts, task retirement, execution-verified checkpoints, and frozen tooling on Vals, Terminal-Bench-Science, PhysicianBench, GDPval, and AstaBench.**Measurement bugs vs attacks:**Opus 4.5 moved from 42% to 95% on CORE-Bench after grader and scaffold fixes with no adversary present.**Eval-integrity checklist:**ten controls a security team would write from ABC, HAL, BrowseComp, ImpossibleBench, and One Token.
