mirror of
https://github.com/usestrix/strix.git
synced 2026-08-18 01:39:19 +02:00
Sean TurnerandClaude Opus 4.8
efb698a4d0
The two prefix breakpoints (system + tool_config) only cache the FIXED prefix. A Strix scan's transcript is append-only, so the growing conversation body is re-sent at full input price every turn and cache-read decays as the transcript grows — a denominator effect, not the prefix missing. Add a third rolling breakpoint at index:-1 (the last message). Because prior turns are immutable, this re-caches the whole prefix-so-far each turn and hits on the next; LiteLLM resolves the negative index against the live message list. Measured on a 29-turn Bedrock scan the fixed prefix stayed pinned at ~56k tokens while per-turn input grew to ~256k and cache-read fell 90% -> 22%; the tail point lifts modelled cache-read to ~96% and cuts full-price input ~16x. Degrades gracefully on older LiteLLM (unrecognised location simply not injected). Adds an end-to-end test driving LiteLLM 1.90.1's _apply_message_injections to confirm the breakpoint tracks the tail across a growing transcript. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>