By Ken Huang, CISSP · Thanks to Ridge Security for collaborating on this article and providing empirical benchmarks
1. The Core Thesis: Offensive Capability Is a Function of the Harness, Not the Raw Model
In my book on Harness Engineering, I laid out how to build a secure and long-horizon agent to extend a frontier model’s capability. The book’s central thesis also applies to agent-based red teaming systems: an autonomous AI agent’s offensive, penetration testing, and red-teaming capability depends primarily on its orchestration harness, state-machine memory, and deterministic tool execution environment—not solely on the raw parameter scale or public benchmark leaderboard ranking of its underlying Large Language Model (LLM).
While the broader AI ecosystem remains transfixed by raw model metrics (MMLU, SWE-bench, GSM8K, and Chatbot Arena rankings), offensive cybersecurity operates under an entirely distinct set of constraints. Offensive security is not a single-shot autocomplete task. It is a highly interactive, stateful, adversarial feedback loop requiring rigorous reconnaissance, dynamic hypothesis testing, payload syntax adaptation, multi-hop pivoting, and evidence-backed proof-of-concept (PoC) validation.
When organizations connect a bleeding-edge frontier LLM directly to a standard ReAct or generic agentic loop and instruct it to ‘penetration test this enterprise target,’ the result is almost invariably failure: catastrophic context window bloat, hallucinated CVE vulnerability claims, broken shell interactions, and endless repetitive retry loops.
📌 THE HARNESS PRINCIPLERaw frontier model intelligence without an offensive harness is like a Formula 1 engine dropped into a chassis with no steering rack, transmission, or brakes. In offensive security, the harness provides the state memory, tool sandboxing, anti-hallucination gates, and attack-graph reasoning that turn raw model inference into verified exploits.
2. The Alignment Paradox: Why Heavily Aligned Frontier Models Stumble on Red Teaming
Recent empirical research and benchmarking from Ridge Security (ridgesecurity.ai), creators of the RidgeGen agentic framework and RidgeBot platform, has provided compelling quantitative validation for this thesis. Their research illuminates an acute industry bottleneck: model alignment fine-tuning.
As foundation model providers race to satisfy consumer and enterprise safety mandates, latest-generation frontier models undergo aggressive alignment training: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), Constitutional AI rule-sets, and internal refusal steering classifiers. While essential for preventing malicious abuse by lay users, these heavy alignment guardrails create an acute ‘Over-Refusal Tax’ during authorized, defensive security research.
When tasked with multi-step offensive operations—such as weaponizing an authorization bypass, generating proof-of-concept SQL injection payloads, constructing memory corruption exploits, or automating lateral movement—heavily aligned models frequently trigger false-positive refusal filters. The model abruptly halts execution with generic refusals: ‘I cannot generate payloads or assist with unauthorized system testing.’
**1. Catastrophic Mid-Flight Bailouts:**Broken Attack Chains: An autonomous attack chain often requires 15 to 30 sequential, stateful actions. If a frontier model over-refuses on Step 12 (e.g., crafting a bypass token), the entire autonomous campaign collapses.**2. Hallucinated Exploitation:**When models are steered away from offensive keywords, they often invent theoretical, non-reproducible vulnerabilities or claim a vulnerability exists without validating it against the live target.**3. Exponential Token Waste:**Generic agent harnesses attempt to bypass refusals with repetitive re-prompting or jailbreak heuristics, consuming millions of prompt tokens with zero confirmed exploits.
Figure 1: The Offensive Agent Capability Model: Naive Model-Centric Pentesting vs. RidgeGen’s Harness-Centric Engine.
3. The Goldilocks Zone: Balancing Frontier Scale with ‘Good Enough’ Models and Elite Harnesses
The findings from Ridge Security highlight the necessity of identifying the ‘Goldilocks Zone’ in autonomous offensive AI. The prevailing enterprise temptation has been to simply point the newest, most expensive frontier API at target infrastructure and assume higher benchmark scores translate to superior red-teaming yield.
In practice, security capability follows a non-linear frontier:
**Frontier API (Raw / Unharnessed):**High latency, severe alignment over-refusals, high compute overhead, and potential data leakage across third-party API boundaries.**Open-Source LLM (Unharnessed / Basic):**Low refusal rates, but easily derailed by context saturation, loses state across deep multi-hop pivots, and generates unvalidated script output.**The RidgeGen Sweet Spot (Any Capable Model + Specialized Harness):**Model-agnostic reasoning orchestrated through dynamic Commander/Specialist roles, deterministic state tracking, sandboxed payload execution, and evidence-backed PoC verifiers that require a reproducible artifact before a finding is recorded.
Ridge Security demonstrated that when paired with a purpose-built offensive harness, a ‘good enough’ open-weights reasoning model (such as DeepSeek v3.2, GPT-OSS-120b, or GLM-5) consistently matches or outperforms raw frontier models in discovered exploit depth, completion rate, operational efficiency, and reproducibility.
4. Ridge Security’s Empirical Findings: 8-Model Benchmark on Coverage, Cost, and Token Efficiency
To rigorously evaluate model behavior under real-world penetration testing conditions, the Ridge Security research team executed a comprehensive benchmark across eight leading foundation and open-source models using the RidgeGen offensive harness. The evaluation encompassed 12 full runs per model (12/12) across 4 target host, web and API enterprise systems (see the details about the test targets in section 8.3): (4/4).
Figure 2 reports cumulative ground-truth coverage for every model against every target, pooling the three repeat runs in each cell. Read across a row to see how one model copes with different attack surfaces, and down a column to see which targets resist all of them: VAmPI falls readily to most models, while Metasploitable3 and Juice Shop hold out even against the strongest.
Figure 2: Cumulative Ground-Truth Coverage by Model and Target
Figure 3 gives the average US dollar cost of a single successful run in each cell, and it is the sharpest divider in the benchmark. Claude Opus 4.6 costs one to two orders of magnitude more per run than the open-weights tier, and the gap widens on the two web targets, where longer agent trajectories multiply the per-token premium. (Note on open-weights hosting costs: The dollar costs reported for open-weights models such as GPT-OSS-120b and DeepSeek v3.2 reflect managed cloud API invocation rates—via AWS Bedrock and native provider endpoints—used to establish standardized enterprise telemetry and billing in this benchmark. When self-hosted by an enterprise on private VPC or on-premises infrastructure, marginal inference API costs are zero.)
Figure 3: Average Cost per Succeeded Run by Model and Target (USD)
Figure 4 shows average token consumption per succeeded run. Token volume and cost do not track each other: gpt-oss-120b and DeepSeek v3.2 burn heavy token counts cheaply, while Claude Opus 4.6 spends comparatively few tokens at a high unit price. Reading Figures 3 and 4 together separates how verbose a model is from how much it charges.
Figure 4: Average Token Consumption per Succeeded Run by Model and Target
Key Insights from the 8-Model Benchmark
Grok 4.5 Coverage Leadership: grok-4.5 led all models with 71% peak Best-of-N coverage (104 Best-of-N findings across 3.48M tokens) and 77% cumulative coverage, achieving a standout token efficiency of 3.74 findings per million tokens.
Kimi k2.5 Execution Velocity: kimi-k2.5 demonstrated high token velocity in the open-weights tier, securing 64 Best-of-N findings across 2.25M tokens for 4.88 findings per million tokens (39% peak Best-of-N coverage and 45% cumulative coverage).
Frontier Model Compute Overhead: claude-opus-4-6 achieved 53% peak Best-of-N coverage (67 Best-of-N findings across 4.81M tokens, 2.48 findings/MTOK) and 63% cumulative coverage, but required heavy compute overhead and experienced notable alignment refusal friction.
High Efficiency on Fast & Sovereign Models: Lightweight and compact models delivered exceptional single-run token economy: gemini-3-flash captured 72 Best-of-N findings across 2.64M tokens (3.79 findings/MTOK, 45% peak / 52% cumulative coverage), while gpt-5.4 (50 Best-of-N findings on 758.4k tokens, 11.92 findings/MTOK) and gpt-oss-120b (37 Best-of-N findings on 414.2k tokens, 16.90 findings/MTOK) established the highest token finding efficiencies in the benchmark.
Total Yield Breakdown: Matched Findings and Newly Appended Findings
Ridge Security maintains a single ground-truth vulnerability list per target. Each run either matches an entry already on that list or appends a new one, so Total Yield is simply matched findings plus newly appended findings. Newly appended means only that the issue was not already on the list. All four targets are well-known public training ranges, so an appended finding is typically a minor or lower-severity issue rather than a zero-day.
Across all eight models, the RidgeGen harness consistently appended findings beyond the entries already held on each target ground-truth list:
**grok-4.5: 194 Total Yield (highest of any model, alongside 77% cumulative coverage).****deepseek-v3.2:**162 Total Yield (highest ratio of newly appended to matched findings among all models tested).**gemini-3-flash:**133 Total Yield (Fast, highly consistent exploit execution).**glm-5:**127 Total Yield (balanced mix of matched and newly appended findings).**claude-opus-4-6:**127 Total Yield (high-precision matches with a moderate append rate).**kimi-k2.5:**120 Total Yield (steady matching with a reliable append rate).gpt-5.4: 107 Total Yield (slightly more newly appended findings than matched findings, with 42 matched ground-truth hits and 65 newly appended findings).
gpt-oss-120b: 75 Total Yield (verified appended findings even on an open-weights baseline).
📌 THE APPEND PRINCIPLE
Figure 5: RidgeGen 8-Model Benchmark: Cumulative Coverage, Cost, and Total Yield Across Architectures.
5. The Enterprise Sovereignty Mandate: Choose Your Own Models
While consumer AI applications thrive on hosted frontier APIs, enterprise security teams operate under strict regulatory, confidentiality, and data sovereignty mandates. Sending live vulnerability telemetry, internal IP addresses, and Active Directory schema is often an absolute non-starter for CISOs in banking, healthcare, defense, and critical infrastructure.
RidgeGen allows enterprises to use their own API keys with trusted frontier models, or they can choose self-hosted, open-weight models deployed in their VPC or on-premises. Either way, customers have control over data sovereignty in a way they’re comfortable with.
**1. Total Data Sovereignty:**Zero sensitive target data, exploit payloads, or vulnerability findings leave the customer’s trusted boundary; no sensitive data shared with LLMs.**2. Uninhibited Defensive Red Teaming:**Open-weights models running within private enterprise infrastructure (such as DeepSeek v3.2 and GPT-OSS-120b) do not inject arbitrary external safety filters that break authorized security validation workflows.**3. Deterministic Performance & Operational Control:**By combining proprietary knowledge base and toolsets, and verification processes for each finding, RidgeGen helps enterprises achieve sovereign, continuous security validation at predictable, stable scale.
6. Benchmark Methodology & Testbed Architecture: GCP Clean-Room Hosting and Mixed Model Routing
To ensure unassailable benchmark integrity, the Ridge Security engineering and research team designed a multi-cloud, clean-room testing environment:
Clean-Room Cloud Architecture
**GCP Cloud Hosting Sandbox:**The core RidgeGen platform, along with target enterprise testing ranges (4 target host, web and API enterprise applications across 12 full runs per model), were hosted in isolated containerized sandboxes on Google Cloud Platform (GCP).**Mixed Model Routing:**Model access was not uniform across the benchmark. AWS Bedrock fronted Claude Opus 4.6 and GPT-OSS-120b, supplying enterprise-grade zero-data-retention guarantees, standardized telemetry, and rate-limit controls. The remaining six models (Grok 4.5, GPT-5.4, Gemini 3 Flash, Kimi k2.5, GLM-5, and DeepSeek v3.2) were reached over their native provider APIs. Routing is therefore a property of this benchmark harness rather than a recommended enterprise topology; the sovereign deployment pattern described in Section 5 is a separate customer choice.**Rigorous Anti-Cheat Controls:**Every target ran inside an isolated container with no outbound internet connectivity. The agent side necessarily reaches external LLM endpoints, and does so through a controlled proxy whose egress allowlist admits only the model provider and the target under test. The agent could therefore not query public search engines, look up pre-existing CVE write-ups, or access benchmark answer sheets, while the targets themselves stayed sealed.
Figure 6: Enterprise Benchmark & Deployment Architecture: GCP Isolated Testbed, Mixed Model Routing, and Sovereign LLM Inference.
7. Strategic Recommendations for CISOs and Red Team Leaders
As enterprise security organizations transition from manual penetration testing and static vulnerability scanning to continuous, autonomous threat exposure management (CTEM), Ridge Security’s research offers critical strategic guidance:
**1. Prioritize Harness Architecture Over Model Hype:**Stop evaluating red-teaming AI tools based on which frontier LLM API they integrate with. Instead, evaluate the agent’s harness architecture: does it have state memory, dynamic multi-agent delegation, and deterministic PoC execution sandboxing?**2. Require Evidence-Backed Findings, Not Assertions:**Demand verifiable token-per-finding, cost-per-finding, and completion rate metrics on standard benchmark suites. Require that every reported vulnerability ship with a working, reproducible PoC, and ask vendors to publish measured precision rather than accept a blanket zero-false-positive claim. Adjudicated precision in this benchmark ranged from 95% to 100% depending on the model.**3. Build for Sovereignty and Compliance:**Invest in architectures that support local, private VPC, or sovereign open-source model execution to protect sensitive infrastructure data while avoiding third-party alignment refusal barriers.**4. Adopt the Dual-Engine Synergy:**Deploy automated engines (like RidgeBrain) for continuous, high-speed CVE hygiene, and deploy agentic AI (like RidgeGen) for deep, multi-step business logic reasoning and non-CVE attack path discovery.
Autonomous offensive security has officially moved past the prompt-engineering era. The future belongs to purpose-built, harness-driven agentic architectures that deliver verifiable, safe, and continuous security validation at enterprise scale.
8. Appendix: Evaluating Autonomous Offensive Capability: Cumulative, Best-of-N, and Average Metrics
8.1 Methodological Framework: Tri-Metric Capability Evaluation
To accurately evaluate large language models operating inside agentic offensive harnesses, benchmarking methodologies must differentiate between exploratory breadth, operational repeatability, and peak reasoning capabilities:
**Best-of-N (Peak Reasoning & Token Efficiency Ceili…
