# Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More

> How to point Oh My Pi at vLLM, llama.cpp, SGLang, and other local servers or a gateway, plus two config fixes for omp 18.2.7 and later.

- URL: https://doug.sh/posts/oh-my-pi-custom-models/
- Author: Doug Calobrisi
- Date: 2026-09-23
- Tags: llm, ai, agents, self-hosted, oh-my-pi, qwen, vllm, llama-cpp, sglang, bifrost


Custom models let you run Oh My Pi (omp) on local models, or on models from providers it doesn't support out
of the box. omp reads them from `~/.omp/agent/models.yml`. Below are working configs for the popular
inference servers and for a gateway. After those come model roles, an allowlist that keeps omp off hosted
models, thinking effort, fallbacks, and a way to log what omp sends.

There have been recent changes to omp. If you copied the config from my
[tuning post](/posts/tuning-a-local-coding-agent-oh-my-pi/), update it for
[omp 18.2.7 and later](#what-changed-in-1827):

- Rename the provider from `local` to a name omp doesn't already use, and update the `modelRoles` entries to
  match. omp now has its own `local` provider for small on-device models, so your Qwen model under that name
  stops resolving.
- Add `qwenTemplateReasoningEffort: true` to the model's `compat` block. Without it, omp stops sending Qwen
  3.8 an effort level, and the model's chat template picks `xhigh` every time.

![Cartoon: a wooden wall cabinet of small pigeonholes under a sign reading local, each holding a tiny blue robot. A big orange robot wearing a name tag that reads qwen3.8-27b is crammed into a pigeonhole far too small for it, with its antenna bent and a question mark over its head. A small blue robot is squashed below it, and the robots next to it stare.](/images/omp-custom-models/local-cubbies.png)

## Inference servers

### vLLM

For a single vLLM server, this is all you need in `models.yml`:

```yaml
providers:
  vllm:
    baseUrl: http://192.168.1.20:8000/v1
    auth: none
    compat:
      extraBody:
        thinking_token_budget: 8192  # needs server support
    modelOverrides:
      qwen3.8-27b:                   # the name vLLM serves
        maxTokens: 32768
```

omp gets the model list and context window from vLLM itself, and sends Qwen 3.8's effort level without any
extra setting.

### llama.cpp, LM Studio and Ollama

omp finds these on its own when they're running locally on their default ports (8080, 1234, and 11434). For
one on another machine, set `LLAMA_CPP_BASE_URL`, `LM_STUDIO_BASE_URL`, or `OLLAMA_HOST`, or add the URL to
`models.yml`:

```yaml
providers:
  llama.cpp:
    baseUrl: http://192.168.1.20:8080
    api: openai-responses
    auth: none
    discovery:
      type: llama.cpp
```

llama.cpp and LM Studio get Qwen 3.8's effort level automatically, like vLLM.

### SGLang, Lemonade, ninfer and the rest

These have no built-in provider, but they speak the OpenAI API, so declare one and let omp read the model
list:

```yaml
providers:
  sglang:
    baseUrl: http://192.168.1.20:30000/v1
    api: openai-completions
    auth: none
    discovery:
      type: openai-models-list
    compat:
      qwenTemplateReasoningEffort: true  # for Qwen 3.8
```

SGLang listens on port 30000, Lemonade on 13305, and ninfer on 8080, all under `/v1`. ninfer ignores
`thinking_token_budget`, so set its budget with `--default-thinking-budget` when you start
it. The [omp-ninfer](https://github.com/alphastorm/omp-ninfer) project has a tested omp setup for it.

## Gateways and hand-listed models

A gateway, or two servers of the same kind, needs a provider of your own too. Give it a name omp doesn't
already use. I name mine after the gateway or machine behind it, and list the models by hand. Here's the
27B entry from my [Bifrost](https://github.com/maximhq/bifrost) gateway:

```yaml
providers:
  bifrost:
    baseUrl: http://192.168.1.10:8080/v1
    api: openai-completions
    apiKey: MY_GATEWAY_API_KEY               # env var (else literal)
    headers:
      x-bf-passthrough-extra-params: "true"  # or extraBody is dropped
    models:
      - id: rtx3090/qwen3.8-27b              # Bifrost's provider/model
        name: qwen3.8-27b
        contextWindow: 262144
        maxTokens: 32768
        reasoning: true
        input: [text]
        cost: {input: 0, output: 0, cacheRead: 0, cacheWrite: 0}
        thinking:
          mode: effort
          efforts: [low, medium, xhigh]
          defaultLevel: medium
        compat:
          qwenTemplateReasoningEffort: true  # needed since 18.2.7
          extraBody:
            thinking_token_budget: 8192
```

Bifrost picks the backend from the part of the id before the slash. `rtx3090/` is a box with two RTX 3090s,
and `strixhalo/` is the mini PC running Qwen3.8 Flash Next.

![Cartoon: blue courier robots carry envelopes from a laptop on a desk labelled omp up a pastel rainbow bridge. At the top, a grey gatekeeper robot in a horned Viking helmet holds a golden horn and points the way. The bridge splits in two, with one path ending at a tall tower PC labelled rtx3090 and the other at a small mini PC labelled strixhalo, where couriers hand over their envelopes.](/images/omp-custom-models/bifrost-bridge.png)

If `apiKey` starts with `!`, omp runs it as a command, which works with a password manager like 1Password:
`"!op read op://dev/gateway/key"`.

## Model roles

`modelRoles` in `~/.omp/agent/config.yml` decides which model does which job. `default` is the main agent
and `task` runs subagents. A few more cover jobs like plan mode and commit messages. You can also set them from the Roles view in `/model`. A `:level` suffix sets the effort for that
role, and I run subagents at `low` and plan mode at `xhigh`:

```yaml
modelRoles:
  default: bifrost/strixhalo/qwen3.8-flash-next:medium
  plan: bifrost/strixhalo/qwen3.8-flash-next:xhigh
  task: bifrost/rtx3090/qwen3.8-27b:low
  smol: bifrost/rtx3090/qwen3.8-27b:low
```

## Keeping omp on your own models

If a role's model doesn't resolve, or `models.yml` doesn't parse, omp doesn't stop. It falls back to the
default model of a known provider it can use, and failing that, the first model it can use at all. It can
use any provider that needs no key or that it has a key for. Keys come from logins, from a few dozen
environment variables, some of which you probably set for other tools, like `HF_TOKEN` or
`AZURE_OPENAI_API_KEY`, and from `.env` files, including one in the project you're working in. With an
AWS Bedrock token in the environment, the tuning post's `local` config sent my test prompt to
Claude Opus 5.5 on Bedrock.

An allowlist in `config.yml` prevents that:

```yaml
enabledModels:
  - "bifrost/*"
```

Now omp only starts on a `bifrost` model, and stops at startup if none of them resolves. A project's `.omp/config.yml` replaces this list
rather than adding to it, so a project with its own list needs `bifrost/*` in it too.

## Thinking effort

`efforts` lists the levels the model accepts, and `defaultLevel` is the one omp uses when a role has no
suffix. Qwen 3.8 takes `low`, `medium`, and `xhigh`.

Keep the `thinking_token_budget` too. Together with the effort level, it's what fixed the
[five-minute turns in the tuning post](/posts/tuning-a-local-coding-agent-oh-my-pi/#the-agent-thought-for-five-minutes-before-doing-anything),
and I've since raised it to 8,192 tokens without seeing the cap make the model worse at real work.

## What the catalog fills in

If you list a model by hand and leave a field out, omp copies it from its built-in catalog. A bare
`qwen3.8-27b` ends up with a hosted price, image input, and a 65,536-token reply limit. The price only
changes omp's cost estimate.

omp also ignores a misspelled key and uses the catalog's value, so `maxToken: 32768` gets you the catalog's
reply limit instead of yours. For your own providers, write out every field in the Bifrost example, and run
`omp models <provider>` to verify.

## Fallbacks

`retry.fallbackChains` in `config.yml` says what to try when a model keeps failing. A key can be a role, a
model, or `provider/*`:

```yaml
retry:
  fallbackChains:
    bifrost/strixhalo/qwen3.8-flash-next:
      - bifrost/rtx3090/qwen3.8-27b:medium
    bifrost/rtx3090/qwen3.8-27b:
      - bifrost/strixhalo/qwen3.8-flash-next:low
```

If a model's server is down, omp moves to the next one in its chain and, by default, switches back on its own
later. A hosted model can go in a chain if it's also in `enabledModels`, but then omp can start on it when a role doesn't resolve.

## What changed in 18.2.7

Both changes are in the [18.2.7 release notes](https://github.com/can1357/oh-my-pi/releases/tag/v18.2.7).
The `local` provider is under Added:

> Added model-kind and grounded-search capability metadata, along with catalogs for local inference and
> search-engine models.

The effort change is under Changed:

> Improved model routing and thinking-policy handling for llama.cpp Qwen models, Bonsai lineage aliases,
> and custom provider names.

## Logging what omp sends

![Cartoon: a laptop on a desk labelled omp is cabled to a server tower labelled GPU. Halfway along the cable, a small blue robot sits beside a receipt printer and reads the long paper strip coming out of it through a magnifying glass, which shows one highlighted line: effort: medium.](/images/omp-custom-models/logging-printer.png)

When omp does something odd with a model, I print what it's sending. This simple script just prints each
request body, minus the messages and tools, and answers "hi":

```python
import json
from http.server import BaseHTTPRequestHandler, HTTPServer


def sse_chunk(delta, finish):
    choice = {"index": 0, "delta": delta, "finish_reason": finish}
    chunk = {"object": "chat.completion.chunk", "choices": [choice]}
    return f"data: {json.dumps(chunk)}\n\n".encode()


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers["Content-Length"])
        body = json.loads(self.rfile.read(length))
        body.pop("messages", None)
        body.pop("tools", None)
        print(json.dumps(body, indent=2), flush=True)
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        self.wfile.write(sse_chunk({"content": "hi"}, None))
        self.wfile.write(sse_chunk({}, "stop"))
        self.wfile.write(b"data: [DONE]\n\n")


HTTPServer(("127.0.0.1", 18080), Handler).serve_forever()
```

Copy your model into a scratch `models.yml` under a provider named `test`, with
`baseUrl: http://127.0.0.1:18080/v1`, `api: openai-completions` and `auth: none`. Add `enabledModels: ["test/*"]` to a scratch `config.yml` so nothing
can fall back to a hosted model. Put both files in one directory and point omp at it, using your model's id:

```bash
mkdir -p /tmp/omp-test  # models.yml and config.yml go here
PI_CODING_AGENT_DIR=/tmp/omp-test \
  omp -p --no-session --model test/qwen3.8-27b:medium "Say hi."
```

*I had help with this one. Anthropic's Claude helped me test omp's releases against the logging server
and draft this post. I read all the words, checked the results, and rewrote anything that sounded like a
chatbot, so the mistakes are mine.*

## Sources

- omp releases: [18.2.7](https://github.com/can1357/oh-my-pi/releases/tag/v18.2.7) and [18.3.0](https://github.com/can1357/oh-my-pi/releases/tag/v18.3.0)
- omp docs at 18.3.0: [models.md](https://github.com/can1357/oh-my-pi/blob/v18.3.0/docs/models.md), [settings.md](https://github.com/can1357/oh-my-pi/blob/v18.3.0/docs/settings.md), and [environment-variables.md](https://github.com/can1357/oh-my-pi/blob/v18.3.0/docs/environment-variables.md)
- omp's Qwen 3.8 effort rules: [`vllm` and `lm-studio`](https://github.com/can1357/oh-my-pi/blob/v18.3.0/packages/catalog/src/compat/rules/classes/qwen.kdl) and [`llama.cpp`](https://github.com/can1357/oh-my-pi/blob/v18.3.0/packages/catalog/src/compat/rules/providers/llama.cpp.kdl)
- [omp's built-in `local` provider](https://github.com/can1357/oh-my-pi/blob/v18.3.0/packages/catalog/src/compat/rules/providers/local.kdl)
- [Bifrost (`provider/model` routing)](https://github.com/maximhq/bifrost)
- Engine docs: [SGLang server arguments (port 30000)](https://docs.sglang.io/docs/advanced_features/server_arguments), [Lemonade's OpenAI-compatible API (port 13305)](https://lemonade-server.ai/docs/api/openai/), and [ninfer serving (port 8080, `--default-thinking-budget`)](https://github.com/Neroued/ninfer/blob/master/docs/serving.md)