RoachClawChat and local models

What you can do

• Chat with local models in RoachClaw, on your Mac, without an account.
• Point other tools at the same models through the model server, which speaks the Ollama, OpenAI and Anthropic APIs.
• Use your Claude or ChatGPT plan as a model through your own CLI (Plans as models), and run agents in the Org.

Which models load

RoachInference reads GGUF files of four architectures: Llama (including Mistral and SmolLM), Qwen2 and Qwen3. Other architectures, such as Gemma, Phi or qwen3.5, don't load in 1.1.4; the model server says the architecture isn't supported.

Models already in RoachNet's Ollama folder (~/RoachNet/storage/ollama, where 1.0 kept them) are read in place, so nothing downloads twice. Models a separate Ollama pulled into ~/.ollama aren't read. Model pulls resume where they stopped, try again on their own when the connection stalls for 90 seconds, and are checked by SHA-256 before they're installed.

Home, the RoachClaw pane, Dev and the sidebar show where RoachClaw's model stands: Ready once the model server answers and the model is on this Mac (a hosted or plan model as soon as the server answers), Downloading while it comes in, Not downloaded yet when it isn't on this Mac, and Not ready until the model server answers. Settings › Models marks the default model Default.

Setup suggests a starter model for your Mac's memory:

How memory is planned

Before loading a model, RoachNet works out what it needs from the model file alone: the weights, the cache for the context you asked for, and working buffers. Loaded models may use 80% of the Mac's memory together. If a model doesn't fit:

1. Idle models are unloaded, oldest first.
2. The context is halved until the model fits, but never below 2,048 tokens.
3. If it still doesn't fit, the request fails with a 507 that names the sizes, instead of the Mac swapping itself to a halt.

A model that would fit only once a model that's busy answering finishes gets a 503, and isn't squeezed in past the budget.

Keys and values are kept in half precision on the GPU, so the same memory holds twice the context. At most two models stay loaded at once. ROACHNET_MODEL_MEMORY_GB, ROACHNET_MAX_LOADED_MODELS and ROACHNET_KV_CACHE=f32 change these; see Environment variables.

Parallel requests

A loaded model answers up to four requests at once while memory allows (ROACHNET_MODEL_SLOTS; 1 runs them one at a time). Each request slot keeps its own cache:

• The next turn of a conversation goes back to the slot that holds it, so only what's new is evaluated.
• A new conversation that starts like another, such as agents sharing a long system prompt, starts from a copy of that beginning instead of evaluating it again.

Speed

On the same Mac, with the same qwen3 8B file that Ollama 0.33.2 uses (a 665-token prompt), RoachInference evaluates the prompt at about 68% of Ollama's speed and decodes at about 85% of its speed, or 93% with a short prompt. The CPU and Metal paths produce identical tokens on the reference models.

Keep using your own Ollama

Set ROACHNET_INFERENCE_ENGINE=ollama and RoachNet sends its model requests to an Ollama you run yourself, where it already listens: ROACHNET_OLLAMA_URL if you set it, otherwise Ollama's own http://127.0.0.1:11434. RoachNet never starts or stops an Ollama, and while none answers, RoachInference does.

What goes with a prompt

RoachClaw adds context to what you ask: RoachBrain's memories and wiki pages whose words match the prompt, and the lanes you allow in Settings › RoachClaw › Context (Vault files and notes, RoachArcade, captured sites, your Dev projects, and RoachNet itself).

• Local models get every lane you allow, with two short excerpts from the Vault's text files, read only from files inside the Vault.
• Cloud models (Ollama's cloud models, tagged cloud or -cloud, and your Claude or ChatGPT plan) get none of the lanes unless you turn on Cloud models too, off by default. Then they get the lanes' names and counts, never a file's contents. They never get a memory of a conversation with a model on your Mac, or its wiki page. (Since 1.1.4.)
• Dev's assist also sends the first 8,000 characters of the file open in Dev's Editor to a model on your Mac. A cloud model gets it only when you turn on Dev's open file too, off by default; without it, Dev asks a cloud model without the file and says so, and inline assist waits for a model on your Mac. (Since 1.1.4.)

Limits in 1.1.4

• Llama, Mistral, Qwen2 and Qwen3 only. Images in chat messages are refused.
• Long prompts on large models take longer than in Ollama. Speculative decoding isn't in 1.1.
• Downloading models needs a connection. Once a model is on the Mac, chat works offline.
• RoachBrain's memories of conversations with cloud models, and the notes you save, can go with a prompt to a cloud model when their words match it.

Related

• Use RoachNet with Ollama, OpenAI and Anthropic clients
• System requirements
• RoachNet 1.1.4 release notes
• RoachClaw on the web: which replies on roachnet.org came from your Mac and which from the hosted model