Picture a coding agent that spends fifteen hours on a multi-day engineering task. It lands the core feature, keeps the regression suite green, and matches the requirements checklist. On the last visual polish pass it misses a layout edge case. Under a binary pass/fail rubric, that run is a zero. Your leaderboard says the agent failed. Your engineers know it was ninety percent done.
That gap is the evaluation problem for serious agent work. Pass/fail is still the right primary metric when you need a hard gate. It is a terrible hill-climbing signal when the work is long-horizon, multi-system, or multi-agent. You cannot improve what your score collapses to a coin flip.
This piece is an evergreen practice note, not a breaking-paper digest. The last ~20 hours were thin on brand-new primary papers. What was active—fresh discussion of Android Bench 2.0 after an October 6 agent-result refresh, an October 8 update to Era’s simulation blog, and October 10 discourse around typed decision models for RAG and eval—illustrates a triad worth locking into your eval stack. Near-window, MASBench (arXiv, published October 3) shows the same philosophy for multi-agent collaboration under partial observability.
The idea is simple: score progress continuously · grade against exact keys in simulated estates · gate and judge with typed decisions. Treat Android Bench, Era, and Jev-style decision models as design patterns, not product endorsements. This is about making evals climbable—not yesterday’s integrity-and-attack story.
Android Bench 2.0 (blog dated September 17, 2026) is the clearest public illustration of the continuous-rubric pattern. It adds thirty long-horizon tasks—multi-day engineering work—evaluated across agent harnesses such as Claude Code, Codex, and Antigravity, with five runs per task. Alongside binary pass rate, it reports a continuous completion rate in [0.0, 1.0].
The methodology (last updated 2026-09-17) is worth stealing even if you never touch Android. Completion is a weighted base across functional, regression, requirements, and visual dimensions, then multiplied by penalty terms. Pass/fail remains the headline; completion is the hill-climbing supplement. That division of labor matters. Binary is not obsolete. Binary alone is what erases ninety-percent progress.
The numbers make the design choice concrete. At launch, the blog framed the highest long-horizon pass rate as around 28%, against ~91% on the original (saturated) task set. As of an October 10 check of the live leaderboard, the long-horizon top entry is Claude Opus 5.5 on claude-code at 32.7% pass (CI 18.7–47.3) and 84.7% completion, with average latency 15.2 hours and average cost $215.4. GPT 6 Astra on Codex sits at 28.0% pass and 82.2% completion. On the 1.0 tab, Claude Opus 5 still scores 91.8%. The changelog lists an October 6, 2026 agent-result refresh—including Opus 5.5 on claude-code and GPT 6 Luna / Sol on Codex—so treat “who leads today” as perishable and the methodology as the durable takeaway.
Optional depth that transfers well: pixel-diff visual checks were noisy (methodology notes differences up to 47.5% in calibration). Android Bench replaces that with a multimodal visual judge (Gemini 3.5 Flash, scoring 0.0–1.0), which calibrated to Diff = 0.00 across 360 runs. If your agent produces UI, screenshots of dashboards, or clinical form layouts, brittle pixel equality will lie to you the same way.
Transfer. Finance ops agents that almost close a multi-system reconciliation, and clinical workflow agents that complete most of a charting sequence under constraint, need the same shape: multi-dimensional partial credit plus explicit penalty multipliers for broken invariants. Do not wait for a public Android-style leaderboard in your vertical. Build the rubric first.
Continuous scores answer how much progress you got. They do not answer whether the answer is right when the world is a mess of CRMs, tickets, and chat threads you cannot put in production evals.
