From 22d668d53882f853d45868cf62b6f80f484a7687 Mon Sep 17 00:00:00 2001 From: "devin-ai-integration[bot]" <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Sat, 1 Aug 2026 16:29:18 -0700 Subject: [PATCH] docs(llm-providers): explain the structured tool_calls requirement for local endpoints (#520) (#901) Co-authored-by: Ahmed Allam --- docs/llm-providers/local.mdx | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/llm-providers/local.mdx b/docs/llm-providers/local.mdx index 212dd1b7..58b37509 100644 --- a/docs/llm-providers/local.mdx +++ b/docs/llm-providers/local.mdx @@ -71,3 +71,38 @@ export LLM_EXTRA_HEADERS='{"X-Feature-Key":"value","X-Tenant":"acme"}' For endpoints behind a private CA, point Strix at your certificate bundle with the standard `SSL_CERT_FILE=/path/to/ca-bundle.pem` — never disable TLS verification against a real endpoint. + +## Tool calling must return structured `tool_calls` + +Strix is entirely tool-driven: every working turn must be a **native** function/tool call. If your inference server returns the tool call as plain assistant text instead of a structured `tool_calls` field, Strix never sees a call it can execute, so the agent makes no real progress — it re-prompts the model for a tool call and gives up once its recovery attempts are exhausted. + +This is almost always an **inference-server configuration** problem, not a model or Strix problem. Common symptoms are the model printing a call as text such as: + +```text +{"name": "exec_command", "arguments": {"cmd": "nmap ..."}} +exec_command(cmd="nmap ...", timeout=180) +{"action": "exec_command", "params": {"cmd": "nmap ..."}} +``` + +The fix belongs on the inference server: it must be configured to parse the model's tool tokens into structured `tool_calls`. A correctly configured endpoint either returns a structured call or rejects the request outright — it never leaks the call as text. + +### Fixes by server + +**llama.cpp (`llama-server`)** +- Run with `--jinja` and a correct tool-use chat template (`--chat-template` / `--chat-template-file` matching the model). Recent builds enable `--jinja` by default — **upgrade** if yours doesn't. +- For thinking models, align or disable reasoning (`--reasoning-format`, `-rea off`) so it doesn't break tool-call parsing. +- A low temperature (e.g. `--temp 0.2`) improves tool-call reliability. + +**Ollama** +- Use a recent Ollama and a model whose template wires tools. Modern Ollama refuses tools (`tools param requires --jinja flag`) if the template lacks tool support. +- For reasoning models (e.g. qwen3), disable the model's **thinking** mode — thinking left on frequently pushes the tool call into the text `content` instead of the structured `tool_calls` field. Turn it off on the Ollama side (a non-thinking model variant, or `think: false` in the model's parameters / `Modelfile`). +- Raise **`num_ctx`** to at least 16k–32k. Strix sends a large system prompt plus many tool schemas; at Ollama's small default context the tool definitions are truncated out of the prompt and the model stops emitting valid calls. A short test prompt can look fine while a real scan fails, so set this explicitly rather than inferring it from a quick check. + +**vLLM** +- Start with `--enable-auto-tool-choice`, a matching `--tool-call-parser` (`hermes`, `qwen3_xml`, or `llama3_json`), and a matching `--reasoning-parser` for reasoning models. + +A low sampling temperature (roughly 0.2–0.6, depending on the family) also measurably reduces malformed tool calls on open-weight models. Set it on the server or in your model's parameters. + + +Even correctly configured, small models (< ~30B) emit malformed or text-form tool calls far more often than frontier models. Prefer a capable model for reliable agentic behavior. +