Enables Anthropic/Bedrock prompt caching for Claude routes via LiteLLM
cache_control_injection_points: system prompt + latest message everywhere,
plus tool_config on Bedrock Converse only (the sole route whose transform
consumes it; elsewhere it leaks as an unknown top-level field that native
Anthropic 400-rejects). Unmapped Bedrock Claude models run uncached instead
of crashing. Adds STRIX_PROMPT_CACHE opt-out (default on).
The two prefix breakpoints (system + tool_config) only cache the FIXED
prefix. A Strix scan's transcript is append-only, so the growing
conversation body is re-sent at full input price every turn and
cache-read decays as the transcript grows — a denominator effect, not
the prefix missing.
Add a third rolling breakpoint at index:-1 (the last message). Because
prior turns are immutable, this re-caches the whole prefix-so-far each
turn and hits on the next; LiteLLM resolves the negative index against
the live message list. Measured on a 29-turn Bedrock scan the fixed
prefix stayed pinned at ~56k tokens while per-turn input grew to ~256k
and cache-read fell 90% -> 22%; the tail point lifts modelled cache-read
to ~96% and cuts full-price input ~16x. Degrades gracefully on older
LiteLLM (unrecognised location simply not injected).
Adds an end-to-end test driving LiteLLM 1.90.1's _apply_message_injections
to confirm the breakpoint tracks the tail across a growing transcript.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cache breakpoints are gated on _is_claude_model (name contains
"claude"), but LiteLLM's AnthropicCacheControlHook only *consumes*
cache_control_injection_points for models it recognises as cache-capable
via its statically bundled model map. On a Bedrock route whose model
isn't in that map, the marker passes straight through and Bedrock's
Converse API rejects it outright:
ValidationException: cache_control_injection_points: Extra inputs are
not permitted
— which fails the whole scan at the first LLM call. This bites any
Bedrock Claude model LiteLLM hasn't mapped yet (a just-released model),
and is made worse when LiteLLM can't refresh its remote model map (e.g.
behind a TLS-intercepting corporate proxy) and falls back to a stale
local copy. Observed live on bedrock/global.anthropic.claude-sonnet-5.
Fix: withhold the marker only for a Bedrock route LiteLLM can't confirm
supports prompt caching. Scope is deliberately narrow — Anthropic-native,
Vertex, and OpenRouter Claude tolerate/ignore the marker (or LiteLLM maps
them under keys we don't resolve), so gating those on confirmed support
would DISABLE caching for capable models — the opposite of this PR's
intent. Only Bedrock hard-rejects, so only Bedrock is guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Strix scan is a long multi-turn agentic loop that re-sends a large, STABLE
prefix every turn — the system prompt plus the tool schemas — while only the
conversation tail changes. Without a caching breakpoint that whole prefix is
re-tokenised and billed at full input rate on every turn; on Bedrock Claude
it's the single biggest lever on scan cost. Measured on a real scan: cache-read
went 0% -> 57% once these injection points are set (roughly halving input cost,
and the ratio climbs on longer scans where the stable prefix dominates more
turns).
LiteLLM already implements this end to end: when `cache_control_injection_points`
is present in the call kwargs its `AnthropicCacheControlHook` fires and emits the
provider-appropriate breakpoint (Anthropic `cache_control`; Bedrock Converse
`cachePoint`), honouring Anthropic's 4-breakpoint cap. `LitellmModel` forwards
`ModelSettings.extra_args` straight into `litellm.acompletion()`, so passing the
points there is all that's needed. We mark the two big stable segments (system
prompt + tool_config = 2 of 4 breakpoints, headroom left).
Deliberately kept at the LiteLLM-config layer rather than a general ModelSettings
caching flag — that's the direction the Agents SDK maintainer prescribed when
declining a native `cache_system_prompt` field
(openai/openai-agents-python#3008 / #3009): caching is a LiteLLM/provider
behaviour, and a ModelSettings flag would let strict OpenAI-compatible paths emit
non-standard cache_control parts. Gating on Claude keeps it a strict no-op for
every other provider (no injection points -> the hook never fires); only
Claude-family routes (Anthropic native, Bedrock, Vertex, OpenRouter -> Claude)
honour the marker.
Tests: parametrised, non-vacuous — Claude routes (bedrock/native/openrouter) get
the two injection points; non-Claude (gpt-5/gemini/o3) get extra_args=None.
feat(inputs): implement logic for required tool choice based on model
test(inputs): add tests for force_required_tool_choice behavior
test(runner): update tests to include force_required_tool_choice in settings
Ahmed Allam