Prompt Caching & Cost

Every agent turn re-sends a large stable prefix — the system prompt, tool definitions, knowledge-graph injection, and the frozen earlier history — on each ReAct step. Without caching, the provider bills that prefix as fresh input tokens on every call. Prompt caching lets the provider serve the unchanged prefix from its cache at a fraction of the price.

Providers expose caching in two different styles, and the engine supports both — automatically.

Two caching styles

Style	Providers	What engages the cache
Explicit	Anthropic Claude; some Alibaba Qwen models (qwen3-coder-plus, qwen3-coder-flash, qwen-plus, qwen3.7-plus, qwen3-max)	A per-block cache_control breakpoint on the stable prefix. Without it, nothing is cached.
Automatic	OpenAI, DeepSeek, Google Gemini 2.5, Grok, Moonshot, and most OpenRouter inference providers (Ionstream, Parasail, Novita, …) serving open models like qwen3-coder-next	The provider caches a repeated prefix on its own. A cache_control marker is ignored (harmless). Consistency depends on hitting the same provider endpoint on each request.

The engine gives you one lever for each style.

Explicit caching — cache_control breakpoints

For explicit-cache providers, the engine marks the stable prefix with a cache_control: {"type":"ephemeral"} breakpoint so the provider caches it. Configure it per model:

models:
  - name: claude-haiku
    provider: anthropic
    model_name: claude-haiku-4.5
    cache_control:
      enabled: true                 # default: ON for explicit-cache providers; set false to opt out
      breakpoints: [system, tools, history]  # optional; default placement
      min_prefix_tokens: 1024       # skip caching prefixes below this size

Or via the Admin Dashboard model form, or the REST API:

curl -X PATCH https://<engine>/api/v1/models/claude-haiku \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"cache_control": {"enabled": true}}'

On by default for explicit-cache providers (openai_compatible, anthropic) — a model with no cache_control config caches its stable prefix automatically. Set enabled: false to opt out; the request shape then stays byte-identical to no-cache. The marker is still gated by min_prefix_tokens, so small requests are untouched.

Who honors it: Anthropic (native) and the explicit-cache Qwen models above. Automatic-cache providers ignore the marker — leaving it on is harmless but does nothing for them. A few strict OpenAI-compatible gateways reject the in-content marker; opt out (enabled: false) on those.

Minimum prefix size. Providers don’t cache short prefixes. Anthropic’s minimum is ~1024–4096 tokens depending on the model. If usage.cached_tokens stays zero, your stable prefix may be below the threshold.

Automatic caching — provider sticky-routing

Automatic-cache providers (including the OpenRouter inference providers behind open models like qwen3-coder-next) cache repeated prefixes without any marker. The catch on OpenRouter: requests are load-balanced across providers — and across replicas within a provider — so a back-to-back follow-up call may land on a different endpoint with a cold cache.

The engine fixes this automatically: it sends a stable x-session-id header (your conversation’s session id) on every request. OpenRouter uses it for sticky routing — pinning all of a conversation’s steps and turns to the same upstream provider so its cache stays warm. No configuration required — this is on by default for OpenAI-compatible models.

What you’ll observe with sticky routing on qwen3-coder-next via OpenRouter:

step 1: cached=0       ← cold (cache write)
step 2: cached=3264    ← warm prefix served from cache
step 3+: cached>0      ← stays warm across the turn and into the next turn

Caveats for OpenRouter automatic caching

• Warm-up. The first request (and occasionally the next one or two, while the provider’s replicas warm) reads zero cached tokens. Hit-rate climbs as the conversation continues.
• Don’t pin provider.order if you want stickiness. Setting a manual provider order (via the model’s extra_body) disables OpenRouter’s sticky routing. Let sticky routing pick and pin the provider for you.
• Sticky activates only when caching is cheaper. OpenRouter engages sticky routing only for providers whose cache-read price is below their normal prompt price (most are).

OpenRouter provider routing (extra_body)

You can still pass OpenRouter provider-routing options through the model’s extra_body — it is forwarded verbatim:

models:
  - name: qwen-coder
    provider: openai_compatible
    model_name: qwen/qwen3-coder-next
    base_url: https://openrouter.ai/api/v1
    extra_body:
      provider:
        only: [ionstream, parasail]   # restrict to specific upstreams
        # order: [...]                 # ⚠ disables sticky routing — avoid for caching

Verifying cache hits

For OpenRouter models (engine 1.8.2+), cached tokens come back in the engine’s own response: the processing_stopped SSE event carries cached_prompt_tokens (next to prompt_tokens / completion_tokens), and the admin chat context bar shows a live “N cached” indicator. The engine asks OpenRouter to report this by sending usage: {"include": true} on the request — without it OpenRouter omits the cached count even on cache hits, so older engines always reported zero. (The flag is OpenRouter-specific and is sent only for OpenRouter base URLs.)

The engine also logs cached tokens in a debug log line, and cache hits are visible in your provider’s dashboard — OpenRouter, for example, shows cache-read rows at a fraction of the normal input price:

turn token usage  prompt_tokens=3267 cached_prompt_tokens=3264 completion_tokens=21

cached_prompt_tokens is the portion of the prompt served from the provider’s cache (a subset of prompt_tokens). It should climb as the conversation grows. Explicit-cache providers (Qwen/DashScope) only cache a strictly stable, growing prefix — they discard the whole cache the moment any already-sent content changes or shifts between steps, even content after the breakpoint. The engine keeps the prefix stable two ways: it builds each turn append-only (per-call reminders are appended as new messages, never rewritten or shifted, and any wall-clock time in a reminder is stamped once at the start of the turn so it can’t change mid-turn), and it places the cache_control history breakpoint on the last stable message — a moving tail that follows the conversation as it grows. Because each step’s request is a strict append-only extension of the previous one, the tail breakpoint chains to the previous request’s cache write, so the provider serves the whole prior prefix from cache and cached_prompt_tokens grows to full depth as the conversation extends — never pinned at the system prompt, and never capped at a fixed number of messages.

If cached_prompt_tokens stays frozen at roughly the system-prompt size, or stops growing partway through a long conversation while the prompt keeps climbing, upgrade to engine 1.10.2 or later. Earlier versions had two separate limits: a per-minute timestamp in the environment reminder collapsed the cache every few steps (fixed in 1.8.3), and the history breakpoint was pinned to a fixed set of early messages so a long conversation’s recent tail was re-billed at full price once it grew past them (fixed in 1.10.2, which moves the breakpoint with the conversation to full depth). If it stays zero across repeated requests with the same prefix, check: the prefix is above the provider minimum (~1024 tokens per cache block on Qwen’s explicit cache); you didn’t pin provider.order; and cache_control isn’t explicitly disabled.

Quick reference

Your model	Caching lever	Action
Anthropic Claude	Explicit	Set cache_control.enabled: true
qwen3-coder-plus / -flash, qwen-plus	Explicit	Set cache_control.enabled: true
qwen3-coder-next via OpenRouter	Automatic (sticky)	Nothing — x-session-id sticky routing is automatic
OpenAI, DeepSeek, Gemini 2.5	Automatic	Nothing — provider caches automatically