mirror of
https://github.com/usestrix/strix.git
synced 2026-08-17 01:29:42 +02:00
Sean Turner
d93a99a355
A Strix scan is a long multi-turn agentic loop that re-sends a large, STABLE prefix every turn — the system prompt plus the tool schemas — while only the conversation tail changes. Without a caching breakpoint that whole prefix is re-tokenised and billed at full input rate on every turn; on Bedrock Claude it's the single biggest lever on scan cost. Measured on a real scan: cache-read went 0% -> 57% once these injection points are set (roughly halving input cost, and the ratio climbs on longer scans where the stable prefix dominates more turns). LiteLLM already implements this end to end: when `cache_control_injection_points` is present in the call kwargs its `AnthropicCacheControlHook` fires and emits the provider-appropriate breakpoint (Anthropic `cache_control`; Bedrock Converse `cachePoint`), honouring Anthropic's 4-breakpoint cap. `LitellmModel` forwards `ModelSettings.extra_args` straight into `litellm.acompletion()`, so passing the points there is all that's needed. We mark the two big stable segments (system prompt + tool_config = 2 of 4 breakpoints, headroom left). Deliberately kept at the LiteLLM-config layer rather than a general ModelSettings caching flag — that's the direction the Agents SDK maintainer prescribed when declining a native `cache_system_prompt` field (openai/openai-agents-python#3008 / #3009): caching is a LiteLLM/provider behaviour, and a ModelSettings flag would let strict OpenAI-compatible paths emit non-standard cache_control parts. Gating on Claude keeps it a strict no-op for every other provider (no injection points -> the hook never fires); only Claude-family routes (Anthropic native, Bedrock, Vertex, OpenRouter -> Claude) honour the marker. Tests: parametrised, non-vacuous — Claude routes (bedrock/native/openrouter) get the two injection points; non-Claude (gpt-5/gemini/o3) get extra_args=None.