mirror of
https://github.com/usestrix/strix.git
synced 2026-08-22 11:02:08 +02:00
107 lines
5.3 KiB
Plaintext
107 lines
5.3 KiB
Plaintext
---
|
||
title: "Local Models"
|
||
description: "Run Strix with self-hosted LLMs for privacy and air-gapped testing"
|
||
---
|
||
|
||
Running Strix with local models allows for completely offline, privacy-first security assessments. Data never leaves your machine, making this ideal for sensitive internal networks or air-gapped environments.
|
||
|
||
## Privacy vs Performance
|
||
|
||
| Feature | Local Models | Cloud Models (GPT-5/Claude 4.5) |
|
||
|---------|--------------|--------------------------------|
|
||
| **Privacy** | 🔒 Data stays local | Data sent to provider |
|
||
| **Cost** | Free (hardware only) | Pay-per-token |
|
||
| **Reasoning** | Lower (struggles with agents) | State-of-the-art |
|
||
| **Setup** | Complex (GPU required) | Instant |
|
||
|
||
<Warning>
|
||
**Compatibility Note**: Strix relies on advanced agentic capabilities (tool use, multi-step planning, self-correction). Most local models, especially those under 70B parameters, struggle with these complex tasks.
|
||
|
||
For critical assessments, use state-of-the-art cloud models such as **Claude 4.5 Sonnet** or **GPT-5**. Use local models only when privacy is the absolute priority.
|
||
</Warning>
|
||
|
||
## Ollama
|
||
|
||
[Ollama](https://ollama.ai) is the easiest way to run local models on macOS, Linux, and Windows.
|
||
|
||
### Setup
|
||
|
||
1. Install Ollama from [ollama.ai](https://ollama.ai)
|
||
2. Pull a high-performance model:
|
||
```bash
|
||
ollama pull qwen3-vl
|
||
```
|
||
3. Configure Strix:
|
||
```bash
|
||
export STRIX_LLM="ollama/qwen3-vl"
|
||
export LLM_API_BASE="http://localhost:11434"
|
||
```
|
||
|
||
### Recommended Models
|
||
|
||
We recommend these models for the best balance of reasoning and tool use:
|
||
- **Qwen3 VL** (`ollama pull qwen3-vl`)
|
||
- **DeepSeek V3.1** (`ollama pull deepseek-v3.1`)
|
||
- **Devstral 2** (`ollama pull devstral-2`)
|
||
|
||
## LM Studio / OpenAI Compatible
|
||
|
||
If you use LM Studio, vLLM, or other runners:
|
||
|
||
```bash
|
||
export STRIX_LLM="openai/local-model"
|
||
export LLM_API_BASE="http://localhost:1234/v1" # Adjust port as needed
|
||
```
|
||
|
||
### Gateways that require custom headers
|
||
|
||
Some OpenAI-compatible gateways require extra HTTP headers (for attribution or
|
||
tenant routing) alongside the bearer token. Set them with `LLM_EXTRA_HEADERS` as
|
||
a JSON object. Strix sends these headers on every request:
|
||
|
||
```bash
|
||
export STRIX_LLM="openai/your-model"
|
||
export LLM_API_BASE="https://your-gateway.example/v1"
|
||
export LLM_API_KEY="your-bearer-token" # sent as Authorization: Bearer ...
|
||
export LLM_EXTRA_HEADERS='{"X-Feature-Key":"value","X-Tenant":"acme"}'
|
||
```
|
||
|
||
For endpoints behind a private CA, point Strix at your certificate bundle with
|
||
the standard `SSL_CERT_FILE=/path/to/ca-bundle.pem`. Do not disable TLS
|
||
verification against a real endpoint.
|
||
|
||
## Tool calling must return structured `tool_calls`
|
||
|
||
Strix is entirely tool-driven. Every working turn must be a **native** function or tool call. If the inference server returns the call as plain assistant text, Strix never sees a call it can execute. The agent makes no real progress and gives up after its recovery attempts end.
|
||
|
||
This is almost always an **inference-server configuration** problem, not a model or Strix problem. Common symptoms are the model printing a call as text such as:
|
||
|
||
```text
|
||
<tool_call>{"name": "exec_command", "arguments": {"cmd": "nmap ..."}}</tool_call>
|
||
exec_command(cmd="nmap ...", timeout=180)
|
||
{"action": "exec_command", "params": {"cmd": "nmap ..."}}
|
||
```
|
||
|
||
The fix belongs on the inference server. It must be configured to parse the model's tool tokens into structured `tool_calls`. A correctly configured endpoint either returns a structured call or rejects the request outright. It never leaks the call as text.
|
||
|
||
### Fixes by server
|
||
|
||
**llama.cpp (`llama-server`)**
|
||
- Run with `--jinja` and a correct tool-use chat template (`--chat-template` or `--chat-template-file` matching the model). Recent builds enable `--jinja` by default. Upgrade if yours does not.
|
||
- For thinking models, align or disable reasoning (`--reasoning-format` or `-rea off`) so it does not break tool-call parsing.
|
||
- A low temperature, such as `--temp 0.2`, improves tool-call reliability.
|
||
|
||
**Ollama**
|
||
- Use a recent Ollama and a model whose template wires tools. Modern Ollama returns `tools param requires --jinja flag` if the template lacks tool support.
|
||
- For reasoning models such as qwen3, disable the model's **thinking** mode. Thinking left on can push the tool call into `content` instead of the structured `tool_calls` field. Turn it off on the Ollama side with a non-thinking model variant, `think: false` in the model's parameters, or `Modelfile`.
|
||
- Raise **`num_ctx`** to at least 16k to 32k. Strix sends a large system prompt plus many tool schemas. At Ollama's small default context, the tool definitions can be truncated from the prompt. The model can then stop emitting valid calls. A short test prompt can look fine while a real scan fails. Set this explicitly instead of inferring it from a quick check.
|
||
|
||
**vLLM**
|
||
- Start with `--enable-auto-tool-choice`, a matching `--tool-call-parser` (`hermes`, `qwen3_xml`, or `llama3_json`), and a matching `--reasoning-parser` for reasoning models.
|
||
|
||
A low sampling temperature (roughly 0.2–0.6, depending on the family) also measurably reduces malformed tool calls on open-weight models. Set it on the server or in your model's parameters.
|
||
|
||
<Warning>
|
||
Even correctly configured, small models (< ~30B) emit malformed or text-form tool calls far more often than frontier models. Prefer a capable model for reliable agentic behavior.
|
||
</Warning>
|