Compare commits

..
Author SHA1 Message Date
Alex Schapiro b9e59c1f7d Fall back to token-based cost estimation when observed LiteLLM cost is missing 2026-07-17 19:45:58 +00:00
devin-ai-integration[bot]andAlex Schapiro e4548cb28c fix(proxy,tooling): serialize+reconnect Caido client, actionable HTTPQL errors, sandbox tool guidance (#794)
* fix(proxy,tooling): serialize+reconnect Caido client, actionable HTTPQL errors, sandbox tool guidance

Addresses the top recurring agent tool-call failures observed in telemetry:

- proxy: the shared Caido client had no locking or reconnect, so concurrent
  agent calls raced ("Transport is already connected") and a dead transport
  poisoned the rest of the run ("Connector is closed"/"Server disconnected").
  Add an asyncio lock + bounded reconnect in caido_api.call_with_client (sandbox
  path) and a scan-wide caido_lock in the run context that host-side proxy tools
  hold around every call. Deterministic errors are not retried.
- proxy: list_requests now returns Caido's exact parser message, echoes the
  offending query, and includes a corrected-syntax hint so agents self-correct
  instead of retrying a broken HTTPQL filter.
- shell/prompt: document that write_stdin requires a process started with
  tty=true; nudge toward writing Python to a file over deeply-nested one-liners;
  note the venv pre-installs common libs.
- agent-browser: distinguish daemon/connection failures (run doctor, don't loop)
  from malformed commands; invoke directly (no sh -c wrapper).
- containers: use POSIX '.' instead of the bashism 'source' in generated rc
  files (fixes 'sh: source: not found'); add file + xxd and pre-install
  requests/httpx/beautifulsoup4/lxml/pyjwt/cryptography in the sandbox venv.
- tests: cover proxy serialization/reconnect/no-retry and HTTPQL errors.

* fix(proxy): host-side reconnect, close stale clients, don't retry mutations

Addresses Greptile review on the reconnect logic:

- Host path had no reconnect: a dead shared context client (Caido restart /
  network blip) previously disabled proxy tools for the rest of the scan. Add
  SharedCaidoClient, a serialized reconnect-safe holder stored once per scan in
  the run context and shared across agents. On a dead transport it rebuilds via
  reconnect_caido, which re-selects the SAME Caido project (preserving captured
  traffic) instead of creating a new empty one.
- Don't repeat completed mutations: call_with_client / SharedCaidoClient.call
  take idempotent=. Reads retry once on reconnect; replay + scope
  create/update/delete heal the client but re-raise instead of risking a
  double-apply.
- Don't leak replaced clients: the stale client is aclose()d (best-effort) on
  every reconnect.
- Extend tests to cover close-on-reconnect, non-idempotent re-raise, and the
  SharedCaidoClient holder.

* fix(proxy): close replacement Caido client when project.select fails

Addresses Greptile P1: in reconnect_caido (and bootstrap_caido) a successful
connect() followed by a failing project.select()/create() discarded the
connected client without closing it, so a missing/unavailable project could
leak a transport on every retry. Close the client before re-raising.

---------

Co-authored-by: Alex Schapiro <bearsyankees@gmail.com>
2026-07-17 13:31:57 -04:00
df97c86f8f fix(prompt): down-rate or skip findings on demo data / demo environments (#793)
* fix(prompt): treat demo/sample data and demo environments as low severity or skip

* Update system_prompt.jinja

* fix(prompt): use demo context as a skip signal, not a CVSS override

* fix(prompt): let demo context honestly inform CVSS impact metrics

* fix(prompt): focus on detecting demo environments to inform CVSS impact

* fix(prompt): keep demo-environment check concise

* fix(prompt): trim demo-environment check to a short addendum

---------

Co-authored-by: Alex Schapiro <bearsyankees@gmail.com>
Co-authored-by: alex s <46074070+bearsyankees@users.noreply.github.com>
2026-07-16 22:14:36 -04:00
2 changed files with 23 additions and 7 deletions
+1 -1
View File
@@ -201,7 +201,7 @@ EFFICIENCY TACTICS:
VALIDATION REQUIREMENTS:
- Full validation required - no assumptions
- Demonstrate concrete impact with evidence
- Consider business context for severity assessment
- Consider business context for severity assessment — check whether the target is a demo/sandbox environment or content meant to be public, and factor that in
- Independent verification through subagent
- Document complete attack chain
- Keep going until you find something that matters
+22 -6
View File
@@ -19,6 +19,8 @@ class LLMUsageLedger:
self._agent_usage: dict[str, Usage] = {}
self._agent_metadata: dict[str, dict[str, str]] = {}
self._total_cost = 0.0
self._observed_cost = 0.0
self._routed_estimated_cost = 0.0
def record(
self,
@@ -41,9 +43,14 @@ class LLMUsageLedger:
if model:
metadata["model"] = model
if not _is_litellm_routed(model):
estimated = _estimate_litellm_cost(usage, model)
if estimated:
estimated = _estimate_litellm_cost(usage, model)
if estimated:
if _is_litellm_routed(model):
# Fallback for routed models whose provider-reported cost never
# arrives (e.g. missing LiteLLM pricing map entry); only counted
# when no observed cost is recorded for the run.
self._routed_estimated_cost += estimated
else:
self._total_cost += estimated
return True
@@ -51,14 +58,21 @@ class LLMUsageLedger:
def record_observed_cost(self, cost: float) -> None:
if isinstance(cost, int | float) and cost > 0:
self._total_cost += float(cost)
self._observed_cost += float(cost)
def _effective_cost(self) -> float:
if self._observed_cost > 0:
return self._total_cost
return self._total_cost + self._routed_estimated_cost
@property
def total_cost(self) -> float:
return _round_cost(self._total_cost)
return _round_cost(self._effective_cost())
def to_record(self) -> dict[str, Any]:
record = serialize_usage(self._total_usage)
record["cost"] = _round_cost(self._total_cost)
effective_cost = self._effective_cost()
record["cost"] = _round_cost(effective_cost)
record["agents"] = []
agent_tokens = {aid: _resolve_total_tokens(u) for aid, u in self._agent_usage.items()}
@@ -67,7 +81,7 @@ class LLMUsageLedger:
usage = self._agent_usage[agent_id]
metadata = self._agent_metadata.get(agent_id, {})
agent_cost = (
self._total_cost * (agent_tokens[agent_id] / total_tokens) if total_tokens else 0.0
effective_cost * (agent_tokens[agent_id] / total_tokens) if total_tokens else 0.0
)
agent_record = serialize_usage(usage)
@@ -88,6 +102,8 @@ class LLMUsageLedger:
self._agent_usage.clear()
self._agent_metadata.clear()
self._total_cost = 0.0
self._observed_cost = 0.0
self._routed_estimated_cost = 0.0
if not isinstance(raw_usage, dict):
return