# Agent guide — tuning the Bonsai demo

For AI agents (and humans) helping someone set up this demo. Goal: pick the right
flags for the user's hardware and use case. The behavior notes below come from real
testing; measure on the user's own hardware before promising performance (the
`timings` object in every API response has the numbers).

## Why the 27B models (what to show off)

The 27B generation is a step change over the earlier 8B/4B/1.7B demos:

- **Vision** — image input end to end (photos, screenshots) via the mmproj on llama-server.
- **Agentic / tool calling** — much stronger at tool use; both llama-server (`--jinja`)
  and `mlx_lm.server` emit native OpenAI `tool_calls`, verified with full tool round-trips.
- **Thinking** — a reasoning model; thought is streamed separately (`reasoning_content`)
  and can be budgeted (`--reasoning-budget N`) or picked per chat in the web UI.
- **Long context** — 256k+ tokens per conversation on a 48 GB Mac (hybrid GDN keeps KV cheap).
- **Tiny footprint** — the binary 27B packs to ~1.125 bits per weight: it fits
  on a modern iPhone without memory offloading. **Bonsai 2 27B (~1.75 bpw, 5.9 GB) is the
  default**: the highest-quality build, and the one to recommend unless the user asks otherwise.

## The models

| `BONSAI_FAMILY` / `BONSAI_MODEL` | Weights | Notes |
|---|---|---|
| `bonsai2` / `27B` (**default**) | GGUF 5.9-7.2 GB + mmproj 0.63 GB; MLX 2-bit 8.6 GB | The current generation and what plain `./setup.sh` installs. Two GGUF bands, **both requiring this demo's binaries**: `PTQ1_0` (1.75 bpw, 5.9 GB, densely packed trits, smallest) and `PQ2_0` (2.13 bpw, 7.2 GB, faster prompt processing, what the scripts download). There is no mainline-compatible band; see the Q2_0 warning below. The MLX pack carries its own Hadamard-aware loader in `runtime/` plus the vision tower, and runs on stock mlx-vlm from `.venv-vlm`, not the fork in `.venv` |
| `ternary` / `27B` (previous generation) | GGUF ~6.7-7.1 GB + mmproj 0.9 GB; MLX 2-bit ~7.9 GB | Superseded by `bonsai2`. Two GGUF formats since the mainline rebase (`prism-b10658+`): `PQ2_0` (group 128, 6.66 GiB, smallest/fastest where supported: CUDA, Metal, CPU, ROCm) and official `Q2_0` group 64 (`Ternary-Bonsai-27B-Q2_g64.gguf`, 7.05 GiB, adds Vulkan/SYCL); smaller sizes use `*-Q2_0_g64.gguf` naming. The scripts pick per backend. Legacy `*-Q2_0.gguf` files (no `g64`) only load on old `prism-v5` releases; new binaries refuse them with an error |
| `bonsai` / `27B` | GGUF Q1_0 ~3.5 GB + mmproj 0.9 GB; MLX 1-bit ~4.8 GB | Smallest and fastest; fits on a modern iPhone without offloading |
| `8B` / `4B` / `1.7B` (both families) | smaller | Text-only, no tools wiring, legacy tested flag set |

### Do not run Bonsai 2 on stock llama.cpp

Every Bonsai 2 band stores its weights in a rotated basis and needs the activation transform that
only this demo's binaries have. `PQ2_0` and `PTQ1_0` are rejected outright by mainline as unknown
types, which is the safe failure. **`Q2_0` is the dangerous one: mainline loads it without a warning
and produces gibberish**, because `Q2_0` is a type upstream already knows and `qwen35` is a
supported architecture, so nothing in the file makes it refuse.

For that reason the Bonsai 2 `Q2_0` band is **not** in the model repo. It lives on its own at
[Ternary-Bonsai-2-27B-gguf-dev](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf-dev),
named `Ternary-Bonsai-2-27B-Q2_0-prism-fork-required.gguf`, published for testing and for the work
to upstream the Hadamard changes. Do not point a user at it for running the model. It moves into the
main repo once mainline can run it.

If someone reports gibberish from Bonsai 2, check which binary they used before anything else.

All three 27B families have the same capabilities (vision, tools, thinking, long context) —
they differ in size and speed. Context: 262,144 tokens max; FP16 KV cache is
64 KiB/token (~6.3 GiB at 100K), so 100K context fits on many consumer devices —
full peak-memory table in the README's Context Size section. All 27B repos:
https://huggingface.co/collections/prism-ml/bonsai-27b. All model repos are public;
no token needed.

## Knobs that matter (27B)

All extra args pass straight through the start scripts, e.g.
`./scripts/start_llama_server.sh --image-max-tokens 1024 -ub 1024`.

| Knob | What it does | Trade-off |
|---|---|---|
| `BONSAI_SPECULATIVE=1` (env, `start_llama_server.sh` only) | **Experimental.** Loads the paired DSpark drafter for speculative decoding (`--spec-type draft-dspark`). Post-migration (`prism-b10658+`) the drafter is the converted `*dspark-dflash*` sidecar (~0.6 GiB; embedding/lm_head shared with the target — one-time conversion documented in SPECULATIVE.md). Measured on an L40S: ternary 27B 1.8-2.4x decode (2.06x blended), 1-bit 27B 1.4-1.75x (1.60x blended). Off by default. | The drafter path is stable and fast on CUDA. On Apple Silicon (Metal) it only pays off for ternary code/math (~1.2x) and is a net slowdown on chat/reasoning and for the 1-bit family, so do not recommend it on Macs. Disables cross-request prompt-cache reuse (every turn re-prefills) and forces single-slot (`-np 1`); worse for multi-turn and the agentic Open WebUI path, which is why it is server-only and opt-in. The prebuilt binaries include both the dspark-capable `llama-server` and the CLI one-shot `llama-speculative-simple`. Details: SPECULATIVE.md. |
| `BONSAI_KV4=1` (env, `start_llama_server.sh` only) | **Experimental.** Q4_0 (4-bit) KV cache, ~3.5x smaller KV memory. Optional quality booster: `./scripts/make_kv_bias.sh` builds a model-specific mean-centering bias (tiny calibration corpus is enough; users can pass their own text) that the server picks up automatically. | Memory tool, not a speed tool: decode is slightly slower than F16 KV. The 27B's hybrid attention already keeps KV small, so only reach for this at very long contexts on tight machines. The bias is calibrated with K-rotation off and the script/server handle the matching flags automatically; llama.cpp backend only. Details: KV-CACHE.md. |
| `--reasoning-budget N` | Caps thinking at N tokens (default -1 = unlimited) | Middle ground; pair with `--reasoning-budget-message` |
| `--image-max-tokens N` | Downscales images to ~N vision tokens (1 token ~ 32x32 px). Model allows ~4096 (=4.2 MP) | The scripts default this to **1024 on Metal / Vulkan / CPU** and leave CUDA/ROCm uncapped; override with `BONSAI_IMAGE_MAX_TOKENS` (0 = uncapped). Loses fine detail (small text / OCR) on large images; images under the cap are unaffected |
| `BONSAI_NGL=N` (env) | Overrides GPU layer offload (auto-detect defaults to all layers when CUDA/ROCm/Vulkan/Metal tooling is present; 0 = CPU-only) | The auto-detect keys on installed tooling, not GPU capability: a CPU box with only an integrated GPU and Vulkan drivers will offload to that iGPU. Capable iGPUs (e.g. Strix Halo) genuinely benefit, weak ones are better on CPU; recommend `BONSAI_NGL=0` when decode is slower than expected on iGPU-only machines |
| `-ub N` | Prefill microbatch (default 512) | Sometimes faster prefill on Metal at 1024; measure |
| `BONSAI_CTX=N` (env) | Overrides the default context. The scripts default to a RAM-tiered size (8192 up to 131072 by machine memory) for predictable memory use; `0` (or unset) resolves to that same tiered size — it is *not* passed to llama.cpp as `-c 0` (which would use the model's full 262k training context and OOM constrained machines). Pass an explicit number to force full context | Bigger context = more KV memory at 64 KiB/token FP16; pair huge contexts with `BONSAI_KV4=1` |
| `BONSAI_MMPROJ_CPU=1` (env, 27B) | Adds `--no-mmproj-offload` so the vision projector stays in system RAM instead of VRAM. Off by default | Frees ~0.9 GiB of VRAM for KV/context on tight cards; cost is a slower image prefill (CPU vision encode, tens of ms to seconds/image). Token generation and text-only requests are unaffected, so it's a good trade when images are occasional |
| `--parallel N` | Server slots (default 4) | More slots = more concurrent users, same total context pool |
| `BRAVE_API_KEY` (env, Open WebUI) | Makes a Brave Search MCP server available next to the preconfigured Hugging Face + DeepWiki ones | Needs `npm i -g @brave/brave-search-mcp-server`; skipped otherwise. Limited to web/news/summarizer (~2.9k tokens); its full tool set is ~29k (brave_place_search alone ~20k). Override with BONSAI_BRAVE_TOOLS. |
| `BONSAI_CODE_INTERPRETER=0` (env, Open WebUI) | Disables the server-side Jupyter code interpreter (plots via matplotlib, data via pandas/numpy, market data via yfinance). On by default. | Off falls back to browser Pyodide: plots still work, but no yfinance/network. The Jupyter stack (`.venv-jupyter`) is built by setup.sh. |

Thinking is extracted into `reasoning_content` by default (collapsible in UIs).

## Built-in web UI (llama-server) — what it can do without any setup

Everything below is per-conversation, in Settings, no server restart:

- **Thinking on demand**: the message box has a **Reasoning effort** picker
  (lightbulb icon) - Off / Low (512) / Medium (2048) / High (8192) / Max (unlimited),
  per-conversation, overrides the server default. Under the hood these are reasoning
  budgets; the API also accepts `thinking_budget_tokens` per request
  (`0` = off, `N` = cap, `-1` = unlimited).
- **MCP tools with an agentic loop**: pre-configured for the 27B — the start scripts
  pass `--webui-config-file scripts/webui-config.json`, which seeds Hugging Face Hub
  and DeepWiki as admin defaults (verified served via `/props.webui_settings`).
  Config `enabled:true` only makes them **available in the message-box MCP selector**;
  tool schemas are sent only for chats that turn a server on. The new-chat screen's
  toggle becomes the user's default for future chats (stored in the browser) — that
  is the knob that trades the prompt tokens below for always-on tools. On a fast GPU
  always-on is cheap (one-time cost per server run). Add more servers in
  Settings -> MCP Client or in the JSON (add `"useProxy": true` if one rejects
  browser CORS — llama-server ships a `/cors-proxy`). `agenticMaxTurns` (default 10)
  bounds the tool loop.

  **MCP prompt cost** — every enabled server's tool schemas are rendered into the
  system prompt of each chat, so they add prefill before the first token. Measured
  from the live servers (2026-07; will drift as they update their tools):

  | MCP server | Tools | Approx. prompt tokens |
  |---|---|---|
  | Hugging Face | 8 | ~2,600 (biggest: `hf_fs` ~670, `hub_repo_details` ~490, `hub_repo_search` ~440) |
  | DeepWiki | 3 | ~400 |
  | Brave Search | 3 (web/news/summarizer) | ~2,900 (full 8-tool set is ~29k; `brave_place_search` alone ~20k - demo limits it via `--enabled-tools`) |

  The cost is paid once per server run, not per chat: the schemas are a stable prompt
  prefix and llama-server reuses the cached prefix across new chats as long as the
  enabled set (and order) does not change. On slow hardware prefer a small fixed
  subset over toggling servers between chats. Same logic in the Open WebUI demo:
  the servers are connected but not attached to the model - pick them per chat from
  the tool menu, or add `server:mcp:<id>` to the model's toolIds to make one permanent.
- **System message**: settable in Settings (useful to give the model a Bonsai identity).
  Can also be shipped as an admin default via `scripts/webui-config.json` (`systemMessage`).
- (A Pyodide Python interpreter setting exists but is NOT implemented in the current
  webui build — the Experimental settings section is a post-release TODO upstream.
  For code execution, use the Open WebUI demo's code interpreter instead.)
- **All sampling params** per conversation; empty fields fall back to the server
  flags (shown as placeholders read from `/props`).
- PDF attachments (optionally as images) and image upload for vision.

Note: UI settings live in the browser (localStorage), so they are per-user/per-machine —
server-side pre-seeding of tools is only available in the Open WebUI demo. This also
means browser state outlives config changes: if every NEW chat prefills thousands of
tokens, the user toggled an MCP server on from the new-chat screen (that saves it as
their default for future chats) - toggle it off there, or clear site data for
localhost:8080; an incognito window shows what a fresh browser would get.

**Agent: tell the user about the image-token cap.** If they are on Metal / Vulkan / CPU,
say that large images are downscaled to ~1024 vision tokens by default (fast, but fine
detail in big images is lost) and ask which they prefer:
- keep the cap — much snappier image answers;
- `BONSAI_IMAGE_MAX_TOKENS=0` — full image detail (best for OCR / screenshots / small
  text), noticeably slower per large image on consumer hardware.
On CUDA/ROCm there is nothing to decide — images run uncapped by default.

## Adding MCP servers

Full guide with entry examples: **TOOLS.md** (repo root). The essentials:

- Only streamable-HTTP servers work directly; stdio-only servers need a local HTTP
  bridge first (the Brave block in `start_openwebui.sh` is the pattern).
- llama-server webui: per-browser in Settings -> MCP Client (localStorage, wins over
  defaults), or shipped for everyone via the `mcpServers` JSON-string in
  `scripts/webui-config.json` (`"enabled":true` = listed in the per-chat selector,
  chats still opt in individually; `"useProxy":true` for CORS-rejecting servers).
- Open WebUI: edit `TOOL_SERVER_CONNECTIONS` in `scripts/start_openwebui.sh` and
  restart. Do NOT use the admin panel - `ENABLE_PERSISTENT_CONFIG=false` means panel
  edits are lost on restart; the script is the source of truth. For bearer tokens set
  `"auth_type":"bearer"` and inject the secret at runtime from an environment
  variable or a gitignored file (mirror the `.brave_key` handling below); never
  write a literal token into the tracked script. To attach a server to every chat,
  add `server:mcp:<id>` to the model `toolIds` in `scripts/openwebui/seed_openwebui.py`
  (mind the prompt-token table above).

**Optional web search (Brave)** - needs a Brave Search API key; the key stays local
(never committed). Install the bridge once: `npm i -g @brave/brave-search-mcp-server`.
- Open WebUI: put the key in a gitignored `.brave_key` file (or `BRAVE_API_KEY` env)
  and run `start_openwebui.sh` - it auto-starts the bridge on 127.0.0.1:8001 and adds
  the `brave` MCP (per-chat opt-in). Tell the user this is how they get web search.
- llama-server webui: no auto-bridge (out-of-box only). Run the bridge yourself
  (`BRAVE_API_KEY=... brave-search-mcp-server --transport http --host 127.0.0.1 --port 8001`)
  and add `http://127.0.0.1:8001/mcp` in Settings -> MCP Client. See TOOLS.md.
- Cost: ~2.9k prompt tokens (limited to web/news/summarizer; the full Brave set is ~29k, so start_openwebui.sh passes --enabled-tools).

## Behavior notes (from testing on Apple Silicon)

- Binary is the snappier demo; ternary trades speed for quality-per-bit. Both were
  tested working end to end (text, native tool_calls with round-trips, vision).
- With thinking on, most of a "slow answer" is reasoning tokens, not vision or prefill —
  before blaming the hardware, reach for a thinking budget. On slow machines prefer
  **capping** over disabling: `./scripts/start_llama_server.sh --reasoning-budget 2048`
  keeps most of the quality while bounding latency; the web UI's Reasoning-effort
  picker (Off ... Max) does the same per chat. (Default stays uncapped; these are
  user choices, not shipped defaults.)
- Image cost scales with resolution and is mostly a first-turn cost: the prompt cache
  makes follow-up questions about the same image near-instant.
- **Check macOS Low Power Mode** when speeds look far off — it throttles inference hard
  (System Settings -> Battery).
- Vision encode runs on GPU by default (`--mmproj-offload`).
- MLX: the **ternary 27B gets full vision + native tool calls** via mlx-vlm
  (`start_mlx_server.sh` uses the stock-mlx `.venv-vlm` that setup.sh creates;
  `BONSAI_MLX_VLM=0` opts out). The binary 27B MLX should support vision the same
  way (the vision tower is full precision in both packs) — it just hasn't been
  wired through / verified in these scripts yet.
- MLX has **no cross-request prompt cache** (mlx-vlm re-prefills the full
  conversation, image tokens included, on every turn), so follow-ups are much
  slower than llama.cpp (which caches the KV prefix). Steer multi-turn users to
  the llama.cpp backend; the image encoder itself is cached, it's the LM prefill.
- **M5 Macs on macOS 26.2–26.4:** if Metal init fails with `error compiling source` /
  command-buffer status 5, set `GGML_METAL_TENSOR_DISABLE=1` (README Appendix — FAQ has
  details). Keep `-ngl` on GPU; don't reach for `BONSAI_NGL=0`.
- Linux CUDA / Windows / CPU-only: not tested yet — extend these notes after testing.

## Quick verification commands

```bash
# server capabilities and effective context/slots
curl -s http://localhost:8080/props | python3 -m json.tool | head -30

# timing any request: read the "timings" object in the response
# (prompt_ms = encode+prefill, predicted_per_second = generation speed)
```

Tool calling: send an OpenAI `tools` array; expect `finish_reason: "tool_calls"`.
Vision: send an `image_url` content part (data URI works). Both verified on both
27B families.