forkless/strix
1
0
Fork 0
mirror of https://github.com/usestrix/strix.git synced 2026-08-19 10:05:12 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat(llm): enable Bedrock/Anthropic prompt caching for Claude models

Browse Source 
A Strix scan is a long multi-turn agentic loop that re-sends a large, STABLE
prefix every turn — the system prompt plus the tool schemas — while only the
conversation tail changes. Without a caching breakpoint that whole prefix is
re-tokenised and billed at full input rate on every turn; on Bedrock Claude
it's the single biggest lever on scan cost. Measured on a real scan: cache-read
went 0% -> 57% once these injection points are set (roughly halving input cost,
and the ratio climbs on longer scans where the stable prefix dominates more
turns).

LiteLLM already implements this end to end: when `cache_control_injection_points`
is present in the call kwargs its `AnthropicCacheControlHook` fires and emits the
provider-appropriate breakpoint (Anthropic `cache_control`; Bedrock Converse
`cachePoint`), honouring Anthropic's 4-breakpoint cap. `LitellmModel` forwards
`ModelSettings.extra_args` straight into `litellm.acompletion()`, so passing the
points there is all that's needed. We mark the two big stable segments (system
prompt + tool_config = 2 of 4 breakpoints, headroom left).

Deliberately kept at the LiteLLM-config layer rather than a general ModelSettings
caching flag — that's the direction the Agents SDK maintainer prescribed when
declining a native `cache_system_prompt` field
(openai/openai-agents-python#3008 / #3009): caching is a LiteLLM/provider
behaviour, and a ModelSettings flag would let strict OpenAI-compatible paths emit
non-standard cache_control parts. Gating on Claude keeps it a strict no-op for
every other provider (no injection points -> the hook never fires); only
Claude-family routes (Anthropic native, Bedrock, Vertex, OpenRouter -> Claude)
honour the marker.

Tests: parametrised, non-vacuous — Claude routes (bedrock/native/openrouter) get
the two injection points; non-Claude (gpt-5/gemini/o3) get extra_args=None.
This commit is contained in:
Sean Turner
2026-07-15 12:54:20 +01:00
parent 40f4e67320
commit d93a99a355
2 changed files with 77 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_inputs.py
+29
View File
@@ -55,6 +55,35 @@ def test_child_initial_input_no_consecutive_same_role(parent_history: list[Any])
assert all(prev != nxt for prev, nxt in pairwise(roles))
def _cache_points(model_name: str) -> Any:
extra = make_model_settings(None, model_name=model_name).extra_args or {}
return extra.get("cache_control_injection_points")
@pytest.mark.parametrize(
"model_name",
[
"bedrock/global.anthropic.claude-opus-4-8",
"anthropic/claude-sonnet-4-5",
"openrouter/anthropic/claude-3.5-sonnet",
],
)
def test_make_model_settings_enables_prompt_cache_for_claude(model_name: str) -> None:
points = _cache_points(model_name)
assert points == [
{"location": "message", "role": "system"},
{"location": "tool_config"},
]
@pytest.mark.parametrize("model_name", ["gpt-5", "vertex_ai/gemini-2.5-pro", "openai/o3"])
def test_make_model_settings_no_prompt_cache_for_non_claude(model_name: str) -> None:
# No injection points for non-Claude models: the LiteLLM cache hook never
# fires, so this stays a strict no-op (won't emit cache_control to strict
# OpenAI-compatible endpoints).
assert make_model_settings(None, model_name=model_name).extra_args is None
def test_build_root_task_empty_config() -> None:
assert build_root_task({}) == ""