mirror of
https://github.com/usestrix/strix.git
synced 2026-08-20 18:38:57 +02:00
107 lines
4.9 KiB
Plaintext
107 lines
4.9 KiB
Plaintext
---
|
||
title: "Local Models"
|
||
description: "Run Strix with self-hosted LLMs for privacy and air-gapped testing"
|
||
---
|
||
|
||
Running Strix with local models allows for completely offline, privacy-first security assessments. Data never leaves your machine, making this ideal for sensitive internal networks or air-gapped environments.
|
||
|
||
## Privacy Versus Performance
|
||
|
||
| Feature | Local Models | Cloud Models (GPT-5/Claude 4.5) |
|
||
|---------|--------------|--------------------------------|
|
||
| **Privacy** | 🔒 Data stays local | Data sent to provider |
|
||
| **Cost** | Free (hardware only) | Pay-per-token |
|
||
| **Reasoning** | Lower (struggles with agents) | State-of-the-art |
|
||
| **Setup** | Complex (GPU required) | Instant |
|
||
|
||
<Note>
|
||
Strix requires advanced agent capabilities, including tool use, multi-step planning, and self-correction. Most local models under 70B parameters struggle with these tasks.
|
||
|
||
Critical assessments often require capable cloud models, such as Claude 4.5 Sonnet or GPT-5. Local models suit assessments where privacy has priority.
|
||
</Note>
|
||
|
||
## Ollama
|
||
|
||
[Ollama](https://ollama.ai) is the easiest way to run local models on macOS, Linux, and Windows.
|
||
|
||
### Setup
|
||
|
||
1. Install Ollama from [ollama.ai](https://ollama.ai)
|
||
2. Pull a high-performance model:
|
||
```bash
|
||
ollama pull qwen3-vl
|
||
```
|
||
3. Configure Strix:
|
||
```bash
|
||
export STRIX_LLM="ollama/qwen3-vl"
|
||
export LLM_API_BASE="http://localhost:11434"
|
||
```
|
||
|
||
### Recommended Models
|
||
|
||
We recommend these models for a good balance of reasoning and tool use:
|
||
- **Qwen3 VL** (`ollama pull qwen3-vl`)
|
||
- **DeepSeek V3.1** (`ollama pull deepseek-v3.1`)
|
||
- **Devstral 2** (`ollama pull devstral-2`)
|
||
|
||
## LM Studio / OpenAI Compatible
|
||
|
||
If you use LM Studio, vLLM, or other runners:
|
||
|
||
```bash
|
||
export STRIX_LLM="openai/local-model"
|
||
export LLM_API_BASE="http://localhost:1234/v1" # Adjust port as needed
|
||
```
|
||
|
||
### Gateways that require custom headers
|
||
|
||
Some OpenAI-compatible gateways require extra HTTP headers (for attribution or
|
||
tenant routing) alongside the bearer token. Set them with `LLM_EXTRA_HEADERS` as
|
||
a JSON object. Strix sends these headers on every request:
|
||
|
||
```bash
|
||
export STRIX_LLM="openai/your-model"
|
||
export LLM_API_BASE="https://your-gateway.example/v1"
|
||
export LLM_API_KEY="your-bearer-token" # sent as Authorization: Bearer ...
|
||
export LLM_EXTRA_HEADERS='{"X-Feature-Key":"value","X-Tenant":"acme"}'
|
||
```
|
||
|
||
For endpoints behind a private CA, point Strix at your certificate bundle with
|
||
the standard `SSL_CERT_FILE=/path/to/ca-bundle.pem`. Do not disable TLS
|
||
verification against a real endpoint.
|
||
|
||
## Tool calling must return structured `tool_calls`
|
||
|
||
Strix requires a **native** function or tool call during each working turn. If the inference server returns plain assistant text, Strix cannot execute the call. The agent then requests another tool call and stops after its recovery attempts end.
|
||
|
||
This is almost always an **inference-server configuration** problem, not a model or Strix problem. Common symptoms are the model printing a call as text such as:
|
||
|
||
```text
|
||
<tool_call>{"name": "exec_command", "arguments": {"cmd": "nmap ..."}}</tool_call>
|
||
exec_command(cmd="nmap ...", timeout=180)
|
||
{"action": "exec_command", "params": {"cmd": "nmap ..."}}
|
||
```
|
||
|
||
Configure the inference server to parse tool tokens into structured `tool_calls`. A configured endpoint returns a structured call or rejects the request. It does not return the call as text.
|
||
|
||
### Fixes by server
|
||
|
||
**llama.cpp (`llama-server`)**
|
||
- Run with `--jinja` and a tool-use chat template that matches the model. Recent builds enable `--jinja` by default. Upgrade if yours does not.
|
||
- For thinking models, align or disable reasoning with `--reasoning-format` or `-rea off`.
|
||
- Set a low temperature, such as `--temp 0.2`, to improve tool-call reliability.
|
||
|
||
**Ollama**
|
||
- Use a recent Ollama version and a model whose template supports tools. Ollama returns `tools param requires --jinja flag` when the template lacks tool support.
|
||
- For reasoning models such as qwen3, disable **thinking** mode. Thinking mode can move tool calls into `content` instead of `tool_calls`. Disable it in Ollama with a non-thinking model or `think: false` in the model parameters or `Modelfile`.
|
||
- Raise **`num_ctx`** to a value between 16k and 32k, or higher. Strix sends a large system prompt and many tool schemas. A small context can truncate the tool definitions and stop valid calls. A short test prompt can look fine while a real scan fails.
|
||
|
||
**vLLM**
|
||
- Start with `--enable-auto-tool-choice`, a matching `--tool-call-parser` (`hermes`, `qwen3_xml`, or `llama3_json`), and a matching `--reasoning-parser` for reasoning models.
|
||
|
||
A low sampling temperature (roughly 0.2–0.6, depending on the family) also measurably reduces malformed tool calls on open-weight models. Set it on the server or in your model's parameters.
|
||
|
||
<Warning>
|
||
Even correctly configured, small models (< ~30B) emit malformed or text-form tool calls far more often than frontier models. Prefer a capable model for reliable agentic behavior.
|
||
</Warning>
|