English

OmniVision is an OmniParser-powered GUI agent plugin for DeepSeek Harness. It turns the desktop or any image into structured elements (text + icons with pixel coordinates), so the model can see and operate the screen without multimodal vision.

Screenshots

Vision view (智能识别)	Recognition history (识别记录)

Features

Tools

The plugin registers these gui_* tools into the shared tools registry:

• gui_capture — capture the desktop screen and run OmniParser to extract interactive elements (text + icons with pixel coordinates), refreshing the shared vision state and saving the SOM-annotated overlay
• gui_act — perform a real mouse/keyboard action on the desktop: click / double-click / right-click / move / type / press / hotkey / scroll / drag
• gui_find — search the elements from the last capture by text or type
• gui_state — show the current vision state without parsing
• gui_verify — re-capture and check whether a text is present or absent, retrying until it settles
• gui_task — execute a scripted multi-step UI plan with re-parse between steps and optional assertions
• gui_open_app — launch an installed desktop app by name via its Windows AUMID
• gui_parse_image — parse a third-party image (from a conversation attachment) into the shared vision state

Installation

dsh plugin --profile web add dsh_omnivision
# or a local path:
dsh plugin --profile web add G:\deepseek\plugins\dsh_omnivision

Restart the web process after installing. The plugin loads as a profile bundle layer:

• Host half registers the gui_* tools into the shared tools registry and serves the browser half over an HTTP route (/dsh-omnivision/<method>)
• Client half mounts the OmniVision dock into shell.overlay and a toggle into the sidebar footer

Prerequisites

• Windows with an OmniParser FastAPI server on http://127.0.0.1:8000 (e.g. via G:\omni\OmniParser\start_server.bat)
• The Python venv needs pyautogui for screenshots and input automation

Usage

Call the gui_* tools from the model side; the OmniVision dock in the browser provides the visual surface:

Development

pnpm build      # tsc (host) + tsc (client) + tsdown (client bundle)
pnpm typecheck

License

MIT

中文

OmniVision 是 DeepSeek Harness 的 OmniParser 驱动的 GUI 智能体插件。它把桌面或任意图片变成结构化元素（文本 + 图标 + 像素坐标），让模型无需多模态能力就能「看见」并操作屏幕。

界面截图

智能识别视图	识别记录视图

功能

工具

插件向共享工具注册表注册以下 gui_* 工具：

• gui_capture — 以原生分辨率截取桌面，运行 OmniParser 提取可交互元素（文本 + 图标 + 像素坐标），刷新共享视觉状态并保存 SOM 标注图
• gui_act — 在桌面上执行真实鼠标/键盘操作：点击 / 双击 / 右键 / 移动 / 输入 / 按键 / 热键 / 滚轮 / 拖拽
• gui_find — 按文本或类型搜索上一次识别的元素
• gui_state — 查看当前视觉状态（不重新解析）
• gui_verify — 重新截屏解析，反复确认某文本出现或消失
• gui_task — 按脚本执行多步 UI 计划，步骤间重新解析并支持断言
• gui_open_app — 通过 Windows AUMID 按名称启动已安装的桌面应用
• gui_parse_image — 解析会话中的第三方图片（附件）进入共享视觉状态

安装

dsh plugin --profile web add dsh_omnivision
# 或本地路径：
dsh plugin --profile web add G:\deepseek\plugins\dsh_omnivision

安装后重启 web 进程。插件作为 profile bundle 层加载：

• Host 半把 gui_* 工具注册进共享 tools 注册表，并通过 HTTP 路由（/dsh-omnivision/<method>）为浏览器半提供数据
• Client 半把 OmniVision Dock 挂到 shell.overlay，侧栏底部挂开关按钮

前置依赖

• Windows + 运行在 http://127.0.0.1:8000 的 OmniParser FastAPI 服务（如 G:\omni\OmniParser\start_server.bat）
• Python venv 需安装 pyautogui（截图与输入自动化）

使用

模型侧直接调用 gui_* 工具；浏览器里的 OmniVision Dock 提供可视化操作：

开发

pnpm build      # tsc (host) + tsc (client) + tsdown (client bundle)
pnpm typecheck

许可证

MIT