@deepseek-ai/dsh-vision-bridge

English | 中文

Host plugin and Web settings page for routing image-bearing messages to a user-configured vision model when the active text model cannot accept image input.

Install

./install.sh            # installs into the default `web` profile
DSH_PROFILE=myprofile ./install.sh

The installer registers this repository as a profile bundle with dsh plugin, then applies the small dsh-host-apiproxy compatibility patch this plugin needs on dsh 0.1.0-rc.x: exposing the vision-bridge settings namespace and admitting new image prompts while the bridge is enabled. Restart dsh web after installing.

How it works

The bridge listens on the agent/pre-step waterfall. Before the loop appends claimed messages to the session log, it inspects each message for image content blocks:

1. If the active provider/model reports image input modality, the message passes through unchanged.
2. If the model explicitly reports text-only input and the settings namespace names a configured endpoint, the bridge:
   • asks the active text model to write one precise vision prompt from the accompanying message text;
   • assembles the same scoped system prompt the loop would use;
   • POSTs system prompt + written prompt + image data to the configured OpenAI-compatible endpoint;
   • replaces the image blocks with the returned text in a fresh user message.
3. The loop logs that text-only message, so the transcript remains reconstructable and the text model never receives raw image bytes.

Settings

The Web Settings page adds a Vision section under the vision-bridge settings namespace:

The plugin's cordis.yml config supplies the settings composition base, so a deployment can set defaults without writing the user document. Bridging requires all of enabled, url, apiKey, and model.

Endpoint protocol

POST <url>/chat/completions with Authorization: Bearer <apiKey> and an OpenAI chat-completions body:

{
  "model": "<settings.model>",
  "messages": [
    { "role": "system", "content": "<assembled harness system prompt>" },
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "<text-model-written prompt>" },
        { "type": "image_url", "image_url": { "url": "data:<mediaType>;base64,<bytes>" } }
      ]
    }
  ]
}

The response must be a chat completion whose first choice carries a string or text-part-array message.content.

Model Experience

Bridged conversation

What the model sees

When bridging activates, the text model sees the original message text plus one appended text block: [image analysis from vision model <model>] followed by the vision model's answer. The image block itself is removed. The text model's prompt-writing auxiliary call uses a fixed instruction and temperature: 0; it carries no session id.

Token effect

The original image blocks contribute no tokens to the text model. The appended vision answer contributes its full text token count, plus the fixed marker line and the prompt-writing call's input/output tokens.

KV Cache effect

System and tool sections are unchanged. The text-model prompt-writing call is a separate request with a fixed system prompt and does not reuse the conversation prefix. A changed vision endpoint, model, or enable state changes future conversation content and therefore future prefix caches.

Known Limitations and Deferred Work

• Bridging triggers only when the active adapter's resolveModel() explicitly omits image from inputModalities. Models with unknown modality metadata pass through untouched.
• The agent/request waterfall may replace the provider/model after pre-step; this plugin reads the agent's declared options and therefore does not follow such a late model switch.
• Images that already reached the session log through another path (for example a custom tool result) are not rewritten; only claimed pre-step inbox messages are bridged.
• Only OpenAI-compatible /chat/completions endpoints are supported.
• The settings page keeps the saved API key hidden and treats a blank key field as "keep the saved key"; clearing a stored key requires the raw settings document.