Context management is how agents survive long conversations. Every LLM has a finite context window; every real task eventually exceeds it. The question is not whether you'll hit the limit, but how gracefully you handle it. A naive agent crashes with a prompt_too_long
error and loses everything. A well-engineered harness detects the approaching limit early, applies the cheapest possible remedy first, and falls back to progressively more expensive strategies only when needed — all without interrupting the user's workflow.
Claude Code treats context management as a five-strategy pipeline ordered strictly by cost. Cheap local operations run first; expensive API calls are a last resort.
The five strategies in cost order:
Snip— removes specific message ranges locally, no API callMicro-compact— edits cached prompt-cache entries instead of re-sending unchanged contentContext collapse— archives old messages and projects a collapsed view at query timeAuto-compact— summarizes old conversation history via an API call (proactive)Reactive compact— catchesprompt_too_long
API errors as an emergency fallback
Token thresholds define a soft-landing zone rather than a hard wall:
// src/services/compact/autoCompact.ts
// The effective window reserves 20K tokens for the compaction summary output.
// This guarantees there's always room to generate the summary itself.
export function getEffectiveContextWindowSize(model: string): number {
const reservedTokensForSummary = Math.min(
getMaxOutputTokensForModel(model),
MAX_OUTPUT_TOKENS_FOR_SUMMARY, // 20,000 tokens reserved
)
return getContextWindowForModel(model, getSdkBetas()) - reservedTokensForSummary
}
The threshold ladder works like this: at ~70% of the effective window, auto-compact triggers proactively. At ~90%, a warning surfaces to the user. At ~98%, new requests are blocked entirely and manual compaction is required. This layered approach means the system has three chances to recover before it ever fails.
Circuit breaker prevents runaway API spend when compaction itself is broken:
// src/services/compact/autoCompact.ts
// After 3 consecutive compaction failures, stop trying.
// Avoids burning API budget on a persistently failing operation.
if (
tracking?.consecutiveFailures !== undefined &&
tracking.consecutiveFailures >= MAX_CONSECUTIVE_AUTOCOMPACT_FAILURES // 3
) {
return { wasCompacted: false }
}
Preserved segments ensure crash recovery works after compaction. When a compact boundary is written, the system records the tailUuid
— the last message kept after compaction. If the process crashes, the resume path uses this UUID to find exactly where to restart, rather than guessing.
GC after compact boundary prevents memory leaks in long-running sessions:
// src/QueryEngine.ts
// Once a compact boundary is emitted, release all pre-compact messages
// from the in-memory array. Without this, the array grows without bound
// even though the context window is being managed correctly.
if (msg.subtype === 'compact_boundary' && msg.compactMetadata) {
const mutableBoundaryIdx = this.mutableMessages.length - 1
if (mutableBoundaryIdx > 0) {
this.mutableMessages.splice(0, mutableBoundaryIdx) // drop pre-compact messages
}
}
The design philosophy here is defense in depth: each strategy is a fallback for the one before it, and the system degrades gracefully rather than failing suddenly.
Hermes takes a simpler, more focused approach: one primary compression strategy backed by prompt caching to reduce the cost of repeated context.
ContextCompressor is a self-contained class with its own LLM client for summarization. It uses a structured summary template (Goal, Progress, Decisions, Files, Next Steps) and supports iterative updates — each subsequent compaction updates the previous summary rather than starting from scratch.
should_compress_preflight() runs a cheap token estimate before every API call, catching the need to compress before the request is even sent:
# hermes-agent/agent/context_compressor.py
# Pre-flight check using rough character-count estimate.
# Runs before every API call — no LLM needed, just arithmetic.
def should_compress_preflight(self, messages: List[Dict[str, Any]]) -> bool:
rough_estimate = estimate_messages_tokens_rough(messages)
return rough_estimate >= self.threshold_tokens
The threshold defaults to 50% of the model's context window, giving plenty of headroom for the summary to be generated and the tail to be preserved.
Prompt caching via apply_anthropic_cache_control()
is Hermes's primary cost-reduction mechanism. It places up to four cache_control
breakpoints using the system_and_3
strategy:
# hermes-agent/agent/prompt_caching.py
# Places cache_control markers on: system prompt (breakpoint 1) +
# last 3 non-system messages (breakpoints 2-4).
# Anthropic's maximum is 4 breakpoints per request.
def apply_anthropic_cache_control(
api_messages: List[Dict[str, Any]],
cache_ttl: str = "5m",
native_anthropic: bool = False,
) -> List[Dict[str, Any]]:
messages = copy.deepcopy(api_messages)
marker = {"type": "ephemeral"}
if cache_ttl == "1h":
marker["ttl"] = "1h"
breakpoints_used = 0
if messages[0].get("role") == "system":
_apply_cache_marker(messages[0], marker) # cache the stable system prompt
breakpoints_used += 1
remaining = 4 - breakpoints_used
non_sys = [i for i in range(len(messages)) if messages[i].get("role") != "system"]
for idx in non_sys[-remaining:]: # cache the last 3 non-system messages
_apply_cache_marker(messages[idx], marker)
return messages
Frozen snapshot pattern: the system prompt is built once at session start and never mutated mid-session. This is critical for cache efficiency — if the system prompt changes, Anthropic's cache prefix is invalidated and you pay full input cost again. By keeping the system prompt frozen, the first cache breakpoint hits on every turn after the first.
Token estimation uses a model-agnostic character-count heuristic:
# hermes-agent/agent/model_metadata.py
# Rough estimate: ~4 characters per token. Fast enough for pre-flight checks.
# Not accurate enough for billing, but accurate enough for threshold decisions.
def estimate_tokens_rough(text: str) -> int:
return len(text) // 4
def estimate_messages_tokens_rough(messages: List[Dict[str, Any]]) -> int:
# Serialize each message to a string and count characters
total_chars = sum(len(str(msg)) for msg in messages)
return total_chars // 4
The compression algorithm itself is a four-phase pipeline: (1) prune old tool results cheaply, (2) protect head messages (system prompt + first exchange), (3) find the tail boundary by token budget, (4) summarize the middle with a structured LLM prompt. On re-compression, the previous summary is updated iteratively rather than regenerated from scratch.
