dsh-win-computer-use

状态：✅ 已交付 · 发布件就绪（Status: delivered · publish-ready · not yet listed in the community catalog）

中文说明 →

Windows-native computer use for DeepSeek Harness. One batch tool runs a whole find → click → type → wait → screenshot sequence inside a single engine request, and drives windows through focus-free paths — so the agent works on this machine without taking over the desktop you are using.

What it saves

These are readings, not estimates. DeepSeek Harness records usage on every assistant message, so the numbers below come from its own session log.

The lever is the round trip, not the schema. Every request re-sends the entire conversation — that is what the provider bills as input on each turn, and it is why turning eight calls into one matters far more than shaving bytes off a tool definition.

And straight about the schema: merging eight single-purpose tools into two made that block bigger — 4,149 → 6,977 characters, because computer's step schema documents 41 fields. That block is a fixed prefix carried on every request and served from the prompt cache, and growing it is a deliberate trade: the field documentation is what lets the model write a whole steps array correctly on the first attempt. One avoided retry is worth far more than the prefix, since a retry is another full context re-send — the most expensive single request measured here was 386,879 uncached input tokens.

When to delegate (two experiments, measured)

An operation can be run by a subagent instead of by the conversation driving it. Whether that is worth anything depends on one thing — how many turns the main agent would have needed — and both cases were measured from the session logs.

1. A task the main agent can specify without looking — same machine, both arms:

Delegating cost 19% more: one batch call finished it in one turn, while a fresh child needed five. So "operations always go to a subagent" is wrong.

2. A cold, multi-turn task — the window shows a random code; nothing can be typed until it is read off the screen, so the main agent cannot finish in one turn:

Delegating saved 61%. Both runs verified themselves (status: OK 2776 / status: OK 7152) with no focus change on either side.

The two constants do all the work: the main agent carried ≈560,000 tokens per request, the child ≈20,000 — a factor of 28. Since the child's side barely grows with complexity while the main agent's grows linearly, the saving scales with the task:

The more complex the task, the more it saves — asymptotically ~96%.

The routing rule

1. Can the whole steps array be written without looking at the screen? (open X → type Z into Y → press OK → capture) → do it yourself, one call. Delegating is more expensive.
2. Otherwise — the next step depends on what the screen says, or the work has several stages → delegate the whole thing to a subagent, and do not scout it first: scouting pays the exploration cost once more, at the expensive agent's rates.

Reuse, measured

A background child is durable — the same operator was handed a second task with send_message and completed it (mirror=reuse-arm-ok, 6 steps, no focus change). Two tasks inside that one child cost 120,803 child tokens, and its per-request context only grew from 18,496 to 21,554: reuse does not inflate the child.

It still cannot pay for itself. A background child costs the parent one extra turn per result, and one parent turn is ~525,000 tokens — about 14× the two tasks' entire child budget. Reuse is worth doing for other reasons (the operator keeps what it has learned about this machine, and the parent never absorbs a second brief), not to save tokens.

The plugin ships this protocol as the computer-operator skill, so the guidance costs one catalog line until an agent actually loads it.

Why this one

Tools

Two tools, deliberately. A separate tool per verb costs schema tokens on every request.

// one call: new chat, type a message, send it, wait, capture the result
{"steps": [
  {"op": "click",    "window": "ChatGPT", "name": "新聊天"},
  {"op": "wait",     "window": "ChatGPT", "type": "Edit", "timeout_ms": 8000},
  {"op": "type",     "window": "ChatGPT", "type": "Edit", "text": "你好", "mode": "background"},
  {"op": "key",      "keys": "enter", "window": "ChatGPT"},
  {"op": "sleep",    "ms": 7000},
  {"op": "shot",     "window": "ChatGPT"}
]}

Reading is cheaper than looking

For text content — a chat reply, a log, a status line — read the window as text instead of capturing it:

{"op": "read", "window": "ChatGPT", "tail": 2000}                                  // newest 2000 chars
{"op": "wait", "window": "ChatGPT", "state": "text_stable", "stable_ms": 2500, "tail": 2000}

Measured against the ChatGPT desktop app: one long reply came back as 2,200 characters of quotable text in a single ~280 ms call, where a screenshot costs an image, cannot be quoted exactly, and captures whichever window happens to be on top. Document reading order puts the newest content last and the app's own chrome first, so a tail read is clean by construction.

It is also more reliable than element search inside browsers. At one point the ChatGPT window's UIA element tree had collapsed to 13 nodes — caption buttons and empty panes, the entire page gone — while read still returned 25,722 characters of page text. So a find returning nothing in a Chromium app does not mean the control is absent; it can mean the tree was never built.

wait state: "text_stable" replaces a guessed sleeps: it polls the text and returns it once it stops changing (measured: 4 polls, 2.6 s on a settled page). And when the element tree is gone, writing can still work — focus plus a physical type reaches the composer with real keystrokes.

Writing without the element tree

The other half of the same problem: when Chromium's tree collapses there is no element to address, and activating a window does not give its input box keyboard focus — keystrokes sent after a plain activation are dropped, Ctrl+A selects nothing, and Enter goes to whatever the app itself had focused. That last one is exactly how a "sent" message quietly never sends.

So type and key accept an x/y point and click it first, which is what actually transfers keyboard focus:

{"op": "type", "window": "ChatGPT", "mode": "physical", "x": 1436, "y": 1205,
 "text": "...", "verify": true}                       // click to focus, real keys, read back
{"op": "key", "keys": "enter", "window": "ChatGPT", "x": 1436, "y": 1205}

Measured on the ChatGPT app with its element tree down to 13 nodes: strategy: "physical.keystrokes", focusClick: true, verified: true, and the text present in the composer afterwards. Long text can go "via": "clipboard" — one clipboard write plus Ctrl+V instead of thousands of SendInput records (the previous clipboard text is saved and restored).

Combined, a chat round needs no element tree at all: click-and-type, Enter, wait text_stable, read.

Two more things that only showed up under real use:

• The foreground is returned once per call, not once per step. Restoring it between steps of the same batch hands the foreground to the user's window and the target app loses control-level focus — so the next step's Enter or Ctrl+A lands nowhere. That is exactly how a "send" quietly does nothing. Now the cursor is restored after every physical step and the foreground at the end of the call (reported as foregroundRestored); a step with restore: false opts out, which is how "leave this app in front" is expressed.
• Pure stability settles on "still working". wait state: "text_stable" was measured settling on the ChatGPT app's "正在回应" placeholder — a stable short string — and returning a reply that had not arrived. Pass guards: {"op":"wait","state":"text_stable","absent":"正在回应","contains":"ChatGPT 说"}.

How background operation works

mode defaults to "auto": try the focus-free layers first, fall back to physical input. mode: "background" refuses instead of silently taking your screen. mode: "physical" is the only mode that really takes focus — and it hands it back.

Every background attempt has to prove itself: text written through WM_SETTEXT is read back and compared before it counts as success.

Requirements

• Windows. (apply() hard-checks process.platform.)
• Windows PowerShell 5.1 (powershell.exe) as the engine — it ships UIAutomationClient and System.Drawing; PowerShell 7 does not.
• dsh web on the 0.1.x line.

Install

dsh plugin --profile web add github:wwwort/dsh-win-computer-use

Once it is published to npm, dsh plugin --profile web add dsh-win-computer-use will work too.

Build

lib/ is committed, so installing never builds anything. To rebuild:

DSH_CHECKOUT=<dsh source checkout> bash scripts/build.sh
node scripts/preflight.mjs        # verifies the dsh.bundle manifest and the real pack list

Measured (2026-09-19, 2560×1600, Windows 11)

The screenshots in this repository are real captures from that run, cropped to remove unrelated desktop content.

Known limits

• Windows only.
• key has no background implementation — use type (value write) or click (Invoke) when you need zero disturbance.
• Legacy controls (WinForms and friends) often expose no UIA patterns at all and appear as Pane; those go through the Win32 message layer. Chromium render widgets ignore WM_SETTEXT, so browser pages need UIA ValuePattern or physical input.
• A contenteditable field can accept SetValue yet keep reporting its placeholder. type therefore returns verified: true|false plus a note rather than implying success.
• Concurrency with a human: "move the pointer, then click" is not atomic, so the engine refuses to click when the pointer did not arrive, and reports where it actually is.
• UIA traversal can take hundreds of milliseconds on a complex window; uia truncates at max_nodes and says so.
• Window titles are matched by substring, and a zero-width character breaks the match — Edge's own title contains one.
• windows restore activates the window (SW_RESTORE); a minimized window cannot be captured meaningfully without restoring it first.

License

BSD-3-Clause.