dsh-vision-bridge

A DeepSeek Harness plugin that lets text-only models receive and understand images. The vision work is done by a local model on your machine.

English · 简体中文

What is this?

DeepSeek Harness (dsh) is an open-source AI coding environment built entirely from plugins. Its chat models — deepseek-v4-flash and deepseek-v4-pro — are text-only: their API rejects image data. As a result, in Harness you cannot paste a screenshot into a session that uses them, attach an image to a message, or use the built-in read_image tool.

This plugin fixes that. It works in three layers:

1. A second model route that accepts images. The same DeepSeek models are registered again as a "twin" provider (deepseek-vision). Because the twin declares image support, the normal image features work: pasting produces a thumbnail and an image block, and read_image is allowed. Before each request is sent to the DeepSeek API, the plugin converts every image in the conversation into a text description produced by a local vision model. The API only ever receives text; the model answers as if it had seen the image.
2. Nine inspection tools. describe_image, extract_text, structured_scan, query_region, detect_elements, locate_object, compare_images, read_clipboard, and check_health let the model look at an image at different levels of detail — from a general description down to per-element coordinates — and let you do the same through chat.
3. Paste routing. When you paste an image, a small browser component asks the server whether the current model can handle images. If yes (twin route), the paste stays a normal image. If no (official text-only route), the image is saved to a private local file and the path is inserted as text, which the inspection tools can then read.

The vision model (Ollama + qwen2.5vl) runs on your machine. No image bytes are ever sent to DeepSeek's API or to any cloud vision service.

If you only use the official route, the plugin still helps: pasted images become local paths and the inspection tools work on them. If you only want the tools, you can ignore the twin route entirely.

Table of Contents

• What is this?
• The Problem
• The Solution
• Features
• Architecture
• Interaction Logic
• Installation
• Usage
• Configuration
• Vision Tools
• Project Structure
• Security & Privacy
• Requirements
• Troubleshooting
• License

The Problem

DeepSeek's flagship chat models (deepseek-v4-flash, deepseek-v4-pro) are text-only: their API endpoints cannot receive image bytes, and their model metadata declares inputModalities: ['text']. DeepSeek Harness builds on that metadata in three hard places:

Workarounds in the wild either fork presets (fragile), require cloud vision keys (Gemini etc.), or force you to switch to a weaker vision model for the whole session — losing DeepSeek's coding ability. None of them deliver what users actually want: DeepSeek, plus native image UX, plus the ability to see.

The Solution

One self-contained plugin, three cooperating layers:

1. Vision twin route (deepseek-vision) — the same DeepSeek models re-registered with inputModalities: ['text', 'image']. Every native gate opens: paste admission, thumbnails, durable image blocks, read_image. Under the hood the twin intercepts image blocks at the request layer, runs the local vision bridge, and forwards only text to DeepSeek's API.
2. Agentic vision tools — nine native tools (describe_image, structured_scan, query_region, ...) powered by a local Ollama vision model, with structured output, validation-with-retry, and content-addressed caching.
3. Paste router — a browser half that asks the host for a verdict based on real model metadata: vision-capable routes keep the native photo flow; text-only routes fall back to a local path so the bridge tools can take over.

Features

• Native image experience for text-only models — paste a screenshot, get a thumbnail, image block, and a DeepSeek that actually saw it, without a single image byte ever reaching the API.
• No cloud keys required — the vision engine is local Ollama (qwen2.5vl); the twin reuses your existing DEEPSEEK_API_KEY credential with the official route's own resolution logic.
• Request-layer transparency — no prompt hacks, no preset forks, no dynamic injection that can race; the interception happens in the adapter, exactly once, per request.
• Structured output with coordinates — element bounding boxes ([0,1000] normalized), region cropping, two-stage localization, image comparison, clipboard reads, schema-validated output with automatic retry on malformed responses.
• Repeated images are cached — repeated images hit the content-hash cache: zero extra inference, stable prefix-cache behavior.
• Self-contained & distributable — a single 29 KB tarball with no machine-specific paths; installs on any Harness via dsh plugin --profile web add.
• Coexists with the official route — the official provider stays untouched as the fallback; the paste verdict decides per session, from live metadata, which flow runs.

Architecture

┌──────────────────────────── Browser · client.js ────────────────────────────┐
│ paste / drop image                                                          │
│   → capture-phase listener (before the composer's own)                      │
│   → GET /vision-bridge/paste?model=<selector label>   (host verdict)        │
│        ├─ takeover:true   (confirmed text-only route)                       │
│        │    → POST bytes → host saves private temp file → path text         │
│        │      inserted into the composer; bridge tools take over            │
│        └─ takeover:false  (image-capable route)                             │
│             → native paste: image part + thumbnail, untouched               │
└─────────────────────────────────────┬───────────────────────────────────────┘
                                      ▼
┌──────────────────────── Host plugin · lib/index.js ────────────────────────┐
│  · POST /vision-bridge/paste → magic-byte sniff → 0600 temp file → {path}  │
│  · 9 tools registered into ctx.tools (native catalog, no mcp__ prefix)     │
│  · registerAdapter('deepseek-vision', VisionDeepSeekAdapter)               │
└──────────────┬───────────────────────────────────────────┬─────────────────┘
               ▼                                           ▼
     ┌────────────────────┐                  ┌──────────────────────────────┐
     │ vision-core         │                  │ VisionDeepSeekAdapter (twin) │
     │ · prepare/validate  │                  │ · listModels/resolveModel:  │
     │ · queue (serial)    │                  │   image-capable metadata     │
     │ · LRU cache         │                  │ · stream(): ImageBlock ──►   │
     │ · 9 tool handlers   │                  │   bridge analysis text ──►   │
     │                     │                  │   DeepSeek API (text-only)  │
     └──────────┬──────────┘                  └───────────────┬──────────────┘
                ▼                                              ▼
        Ollama · qwen2.5vl                        DeepSeek API · same endpoint,
        localhost · private · free                same credential as official

lib/core/ is the host-agnostic vision core shared with the author's MCP vision bridge (scripts/sync-core.mjs re-syncs it — one source of truth, two delivery forms).

Interaction Logic

1. Paste routing (per paste, decided by live model metadata)

The client caches verdicts per selector label (60 s TTL) and refreshes on focus, so the first paste of a session is already correct. Unknown metadata never hijacks a paste — the native path is the safe default.

2. Request-layer interception (vision twin only)

user pastes image ──► durable image block in session history
        │
        ▼
next model request ──► VisionDeepSeekAdapter.stream()
        │
        ▼
sanitize(): for every image block ──► attachments.readImage(bytes)
        │                                 │
        │                                 ▼
        │               write content-addressed file
        │               (~/.dsh/vision-bridge/images/<sha1>.png)
        │                                 │
        │                                 ▼
        │               vision-core.describe_image(path)   ← cache hit ⇒ zero inference
        │                                 │
        │                                 ▼
        │               text: [图片（视觉桥分析）] … + local path
        │
        ▼
text-only request ──► DeepSeek API (identical endpoint/credential as official route)
        │
        ▼
DeepSeek answers WITH the vision analysis; it may also call query_region / extract_text
on the embedded path for deeper, coordinate-accurate inspection.

3. Tool chain (evidence workflow)

structured_scan ──► element list with bboxes (heading/table/chart/button/…)
        │
        ├──► query_region(bbox) ──► the region is REALLY cropped and analyzed alone
        ├──► extract_text(with_coordinates) ──► OCR blocks with normalized coordinates
        └──► locate_object(desc) ──► coarse locate → crop ×1.3 → fine locate → full-image bbox

Installation

# 1. Prerequisites: Ollama running with a vision model (e.g. qwen2.5vl:7b),
#    and DEEPSEEK_API_KEY stored (for the twin route).

# 2. Install the plugin (any form works)
dsh plugin --profile web add .\dsh-external-dsh-vision-bridge-2.0.0.tgz   # tarball
#   dsh plugin --profile web add <directory>                             # checkout
#   dsh plugin --profile web add @yulee-314/dsh-vision-bridge            # npm

# 3. Restart dsh web (bundle layers load at boot)

Usage

1. In any session, open the model selector and pick:
   • DeepSeek (视觉桥) → DeepSeek-V4-Flash (视觉桥) — recommended: coding + native image UX, vision supplied by the bridge.
   • Official DeepSeek route — paste falls back to path text; bridge tools remain available.
2. Paste or drop an image. A thumbnail appears (twin) or a path is inserted (official).
3. Ask normally. DeepSeek answers from the bridge analysis; use query_region / extract_text for coordinate-level detail.

To make the twin the default for every new session:

# ~/.dsh/settings.yaml
agent-default-model:
  provider: deepseek-vision
  model: deepseek-v4-flash-vision
  reasoningEffort: max

Configuration

Tool rows are disabled by setting pasteToPath: false on the bundle row if the paste router is undesired on a specific deployment.

Vision Tools

Project Structure

dsh-vision-bridge/
├── package.json              # dsh.bundle + dsh.client manifests, self-contained deps
├── cordis.patch.yml          # loader row (bundle layer)
├── client.js                 # browser half: paste interception + verdict protocol
├── README.md / README.zh-CN.md
├── scripts/sync-core.mjs     # re-sync lib/core from the MCP vision bridge
└── lib/
    ├── index.js              # host plugin: tools, paste route, adapter registration
    ├── deepseek-vision.mjs   # vision twin route (extends the official DeepSeek adapter)
    └── core/                 # host-agnostic vision core (vendored, self-contained)
        ├── vision-core.mjs   #   tools + prompts + retry/validation orchestration
        ├── image.js          #   preprocessing, crop, data URLs
        ├── validate.js       #   schema validation + retry hints
        ├── cache.js          #   content-hash LRU
        ├── queue.js          #   serial inference queue
        ├── errors.js         #   error taxonomy (ollama_down / model_not_found / …)
        ├── grounding.js      #   two-stage localization math
        ├── compare.js        #   side-by-side comparison composition
        └── clipboard.js      #   Windows clipboard reader

Security & Privacy

• Images never leave your machine. The vision engine is localhost Ollama; the twin route sends only the bridge's text analysis to DeepSeek's API.
• Pasted bytes are magic-byte checked (PNG/JPEG/WebP/GIF), size-capped (25 MB), and stored 0600 in fresh unpredictable temp directories.
• The twin reuses the official route's credential resolution — no second key, no plaintext config.
• Paste hijacking is strictly evidence-based: without a positive text-only confirmation the native path stays untouched.

Requirements

Troubleshooting

License

MIT. The vision core originates from the author's MCP vision bridge project.