Safeguard Docs
Safeguard CLI

Use a local model

Point the Safeguard CLI at a model you host on your own machine or network, with Ollama, LM Studio, llama.cpp or vLLM.

Use a local model

The Safeguard CLI's coding agent normally sends each turn to the Safeguard platform. With a local model, the agent sends its inference to a model you host yourself, on your machine or inside your network. Any server that speaks the OpenAI chat-completions API works: Ollama, LM Studio, a llama.cpp server or vLLM.

What moves and what does not. Only inference runs on your model. Your conversation history is still saved to your Safeguard workspace as usual, so /resume, /sessions, the web view and audit all keep working. The Safeguard platform token is never sent to your model's endpoint.

Quick start

In the terminal app, type /local. The first time, it checks your machine, recommends a model that fits, and offers to download it with Ollama. From a shell, the same flow is:

safeguard model local setup

The check reads your total and free memory, CPU cores and how busy the CPU is, GPU and video memory, free disk, and whether Ollama or LM Studio is installed or running. It then shows a plain summary, for example:

Your Mac: 32 GB RAM, 21 GB free, 10 cores (70% idle), Apple GPU.
Recommended: qwen2.5-coder:14b (about 9 GB). Estimate only: about 11 tokens/s here; yours will differ.
Choices:
  1. qwen2.5-coder:14b, about 9 GB, strong quality (recommended)
  2. qwen2.5-coder:1.5b, about 1 GB, basic quality: fine for small edits and questions, weaker on big tasks
  ...

The list always shows every size, recommended first, and you can pick any of them. The CLI never refuses: a model that needs more memory than you have usable gets a one-line warning ("needs about 22 GB; you have about 5 GB usable, so it may be very slow or fail to load"), and your choice is final.

If you accept the recommended model (or type another number from the list in a shell), the CLI runs ollama pull, shows its progress, saves the URL and model, tests the connection and switches the session to the local model. If Ollama is not installed, the CLI shows where to get it and the commands to run. It never installs software for you. In Full access mode the download is accepted automatically.

Which model fits your machine

The recommendation uses the memory a model may use: the memory available to a new program, not the total, and never into the reserve the system and your other apps keep (20% of the total, at least 2 GB and at most 4 GB). Available memory counts reclaimable memory: on macOS free, inactive, purgeable and speculative pages; on Linux MemAvailable.

Usable memoryRecommended modelDownload sizeNotes
Under 1.5 GBqwen2.5-coder:1.5babout 1 GBShown with a warning: it may be slow, so close apps first.
1.5 to 3 GBqwen2.5-coder:1.5babout 1 GBBasic quality: fine for small edits and questions, weaker on big tasks.
3 to 8 GBqwen2.5-coder:3babout 2 GBBasic quality: fine for small edits and questions, weaker on big tasks.
8 to 16 GBqwen2.5-coder:7babout 5 GBGood for everyday coding.
16 to 32 GBqwen2.5-coder:14babout 9 GBA good balance for everyday coding work.
32 GB or more, or a GPU with 24 GB or moreqwen2.5-coder:32babout 20 GBStrongest of the five; needs the memory to match.

Disk. If free disk is under the model's size plus 1 GB, the CLI says so plainly ("Need about 2 GB free disk to download; you have 1 GB"), shows the commands (ollama pull <model>, then /local set) and goes on; ollama pull itself fails if it cannot finish.

Speed depends on your memory bandwidth, GPU and what else is running, so the speed the CLI shows before you download is an estimate. safeguard model local test reports the speed it measured on your machine.

Set it by hand

Use this when you already run a model server, or want a model other than the recommended one. /local set takes any model name and never depends on the machine check.

safeguard model local set --url http://localhost:11434/v1 --model qwen2.5-coder:14b

In the terminal app: /local set http://localhost:11434/v1 qwen2.5-coder:14b.

OptionWhat it does
--urlThe server's OpenAI-compatible base URL.
--modelThe model name the server knows.
--api-key-env VARThe name of an environment variable holding the server's own key, if it needs one. Give the variable's name, never the key.
--context NThe model's context window in tokens. The default is 32768.
--allow-remoteAllows an address outside your machine and private network. See Network rules.

set saves the setting, runs a test and, in the terminal app, switches the session to the model.

Ollama

  1. Install Ollama from ollama.com/download and start it.
  2. Pull a model: ollama pull qwen2.5-coder:14b
  3. Point the CLI at it: safeguard model local set --url http://localhost:11434/v1 --model qwen2.5-coder:14b

LM Studio

  1. Install LM Studio from lmstudio.ai, download a coding model and load it.
  2. Open the Developer tab and start the local server. The default address is http://localhost:1234/v1.
  3. Point the CLI at it, using the model's identifier as LM Studio shows it: safeguard model local set --url http://localhost:1234/v1 --model <identifier>

llama.cpp

  1. Start the server with a model file: llama-server -m <model.gguf> --port 8080
  2. Point the CLI at it: safeguard model local set --url http://localhost:8080/v1 --model <name>

llama.cpp serves one model, so any name works unless you started the server with an alias.

vLLM

  1. Serve a model: vllm serve <model> --port 8000. If you started it with --api-key, export the key in a variable such as VLLM_API_KEY.
  2. Point the CLI at it: safeguard model local set --url http://<host>:8000/v1 --model <model> --api-key-env VLLM_API_KEY

Test it

safeguard model local test

In the terminal app: /local test. The test reaches the server, runs a small request, reports the speed it measured and checks tool calls.

Tool calls

To read files, edit code and run commands, the agent needs the model to answer in its tool format. The test asks the model for a tool call and tells you plainly whether it followed the format. If it did not, the model can still chat, but it will not run tools reliably. Choose a model trained for tool use (the qwen2.5-coder family is a good start) or a larger size of the same model.

Switch, check and turn off

In the terminal appIn a shellWhat it does
/localSwitches this session to the local model (first time: runs the check).
/model localSame as /local. /model also lists it as "Local (this machine)".
/local testsafeguard model local testTests the connection and tool calls.
/local offsafeguard model local offGoes back to the platform's models. Your setting is kept.
/statussafeguard model local statusShows where inference runs.

While the local model is in use, the footer shows model: local <name>, and /status says: Inference: local model at <url>. Conversation history is saved to your Safeguard workspace as usual.

Local turns are recorded in /usage as billed to local and use no platform tokens.

Before each local turn on your own machine, the CLI reads your free memory once. If less is free than the model needs, it shows a one-line warning and suggests a smaller model or the platform.

Network rules

By default the CLI accepts only addresses on your machine or private network: localhost, loopback addresses, private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, IPv6 fc00::/7) and .local names. The check happens when you save the setting and again every time the CLI connects, so a hostname that later resolves to a public address is refused. Redirects are not followed.

To use a server on a public address, the URL must be https and you must pass --allow-remote. Your code and prompts then leave your network for that server, so use it only for a server you operate.

For administrators

Set the environment variable SAFEGUARD_LOCAL_MODEL_DISABLED=1 on a machine to turn the local model option off. /local, safeguard model local and the model picker then say that it is switched off by your organization, and no turn is sent to a local endpoint.

Troubleshooting

MessageWhat to do
Could not reach the serverCheck that the server is running and the URL ends in /v1. Ollama listens on port 11434 and LM Studio on 1234.
The server does not list the modelUse the exact name the server reports, for example the output of ollama list.
Not on this machine or a private networkUse a local or private address, or pass --allow-remote with an https URL.
The model did not follow the tool formatPick a model trained for tool use, or a larger size.
Low memory warningClose other apps, or switch to a smaller model with /local set, or go back to the platform with /local off.

On this page