Local models

Why

Agents in SDODS only ever write proposals a human reviews. That makes a local model a reasonable place to start: nothing it produces reaches your suite without a review, and nothing you send it reaches anyone else. Two situations make it the only option — source that may not be sent to a hosted provider, and a team that wants the agent features without a per-token bill.

brew install ollama            # or: curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull qwen2.5-coder:7b

sdods doctor                   # the "local models (ollama)" row should be green
sdods agent review -p demo-shop --adapter ollama --model qwen2.5-coder:7b

That is the whole setup. There is no key to create, and agents.budgetUsd stops meaning anything because nothing is billed.

The setting that decides everything: context

Ollama loads a model with a 4096-token context by default, whatever its weights allow — and it truncates a longer prompt silently. An agent prompt carries the tool schemas, so it passes 4096 on the first turn. The model then answers confidently from a fraction of the question, and nothing in the output says so. It looks like a stupid model. It is a truncated prompt.

ollama ps
# NAME           ...  CONTEXT
# llama3.1:8b         4096      ← the weights allow 131072

Set it, either per project or on the server:

agents:
  provider: ollama
  models:
    default: qwen2.5-coder:14b
    reviewer: llama3.1:8b # a cheaper role can use a cheaper model
  local:
    contextTokens: 16384 # → options.num_ctx
    requestTimeoutMs: 300000
    keepAlive: 10m

OLLAMA_CONTEXT_LENGTH=16384 ollama serve    # the same thing, server-wide

SDODS refuses to start a job whose prompt cannot fit, rather than letting it be truncated:

✖ The prompt for this job is about 11,200 tokens, but qwen2.5-coder:7b is running with a
  4,096-token context. Ollama would truncate it without saying so.
  → Set agents.contextTokens: 32768 in the project yaml, or start the server with
    OLLAMA_CONTEXT_LENGTH=32768 (this model supports up to 32,768).

Which model

sdods doctor reports what is installed and how it is loaded. Two properties matter: the model must be able to call tools, and it should be a coding model.

ollama show qwen2.5-coder:7b     # capabilities: completion, tools

A model whose capabilities do not include tools cannot drive the agent loop at all — SDODS says so rather than letting it flail.

The small profile

SDODS shows a hosted model everything it has: 33–35 tools for a role, and 24 more when the role drives a browser. A 7–8B model cannot choose from a menu that long — past roughly twenty functions it stops calling anything and starts describing the call it would make. So a local model gets a different menu, chosen automatically:

The profile is picked by resolveProfile: any ollama model, any OpenAI-compatible endpoint on a private address, any model whose name says 7b/8b/14b, any context under 16k, or any model that does not advertise tool calling. Override it per run or per project:

sdods agent generate -p rwa-bank --adapter ollama --profile small   # or full

agents:
  profile: auto # auto | full | small

Three of those rows came out of measurement rather than taste, on llama3.1:8b:

• The browser is not "more tools", it is a different job. With the five generator tools the model called feature_write. Adding a single browser_* tool made it stop calling anything and describe a plan instead. A small model writes from the step vocabulary; give browser work to a larger one.
• Grounding beats brevity. The longer prompt — the one carrying 60 real step patterns — got a tool call where the short prompt got prose.
• Some models write the call instead of making it. step_find({"project":"rwa-bank",…}) as plain text, with the structured field empty. Under the small profile that is read out of the answer and executed once, and the run says so: recovered a step_find call the model wrote as text.

What to expect from the result

A real run of agent generate on a local 8B, through this profile, produced a proposal in two turns — a feature file with the right tags, sensible scenario titles, and steps that were close to the project's real patterns rather than exactly them (I navigate to the Contacts page where the pattern is I navigate to the {string} page).

That is the honest shape of the result, and it is why nothing is applied automatically. Read the diff, fix the two steps, accept:

sdods proposals show <id>
sdods proposals accept <id> --branch sdods/<id>

Other local servers## Other local servers

Anything that speaks the OpenAI protocol works through the openai-compatible adapter, which no longer demands a key for a local address:

# vLLM, LM Studio, llama.cpp …
sdods agent review -p demo-shop \
  --adapter openai-compatible \
  --base-url http://127.0.0.1:1234/v1 \
  --model qwen2.5-coder-14b

The difference is that the OpenAI protocol has no way to set the context size, so that path relies on the server being started with the right one. The ollama adapter uses the native API precisely so SDODS can set it and check it.

What a small model can and cannot do

Local 7–8B models are good at bounded, well-described work and bad at long open-ended plans. In practice:

Mix them: agents.models takes a model per role, so planning can use a hosted model while generation stays local, or the other way round.

Every job is still a proposal:

sdods proposals list
sdods proposals show <id>
sdods proposals accept <id> --branch sdods/<id>

When something looks wrong

Next steps

• Agents — the roles, the proposal flow and the budgets
• Claude Code and Codex — the other keyless option
• Tokens and keys — every credential SDODS knows about