dsh-web-fetch

Dual-source web content fetcher for DeepSeek Harness — CDP browser rendering + Tavily Extract, each as a standalone LLM tool.

English | 中文

Why dsh-web-fetch?

DeepSeek Harness's ctx.web.registerSearchProvider throws WEB_PROVIDER_AMBIGUOUS when multiple providers are registered for the same capability — the model can't choose.

dsh-web-fetch takes a different path: each data source is a standalone DSH tool (web_fetch_cdp / web_fetch_tavily) with its own description + schema. The LLM picks the right tool based on context, not hard-coded rules.

Both can be independently enabled/disabled — disabled tools are hidden from the LLM entirely.

Parameters (identical for both tools):

Renamed in v0.1.5-rc.1: this parameter used to be named query. That name collided with Tavily's own /extract request field query (a search term), and it invited callers to pass a search topic where a URL was required. It is now url. The old name is deliberately not accepted as an alias — it fails loudly with invalid arguments: missing required property "url", which is itself the migration hint.

Features

• Dual pluggable strategies — CDP + Tavily out of the box, add a new one with 1 file + 1 line
• Zero-intrusion — Cordis bundle plugin, no DSH core patch. All via ctx.tools.register / settings.installSection (dsh ≥ 0.1.2-alpha)
• Zero extra deps — only @deepseek-ai/dsh-settings, @deepseek-ai/dsh-tools, @deepseek-ai/schemastery
• Live config — settings UI card + ~/.dsh/settings.yaml hot-reload, no restart needed
• Concurrency-safe — isConcurrencySafe: true, supports AbortSignal

Architecture

dsh-web-fetch/
├── cordis.patch.yml        # bundle patch (install/uninstall)
├── package.json            # dsh.client injection
├── src/
│   ├── index.js            # Cordis apply() + 2 tool registrations
│   ├── types.js            # FetchStrategy契约 (JSDoc)
│   ├── helpers.js          # withTimeout / plainText / truncate
│   └── strategies/
│       ├── cdp.js          # CDP: raw http + manual WS frames, no `ws` dep
│       └── tavily.js       # Tavily Extract API
├── lib/client.js           # Settings card (React + locale zh/en)
└── tests/
    ├── entry.test.mjs           # host entry + manifest + engines table + key parity
    ├── host-integration.test.mjs # real Cordis + ToolRuntime + file settings provider
    ├── test-cdp-unit.mjs        # 21 tests (helpers 8 + factory 5 + router 8)
    ├── test-cdp-frames.mjs      # 5 tests (RFC6455 frame encode/decode)
    ├── test-tavily-unit.mjs     # 13 tests (availability, parsing, error paths, malformed payloads)
    └── client-smoke.mjs         # 15 checks (browser half: render + interaction + read-only)

Strategy contract (src/types.js):

// New source = implement this, then register once in src/index.js
export function makeMyStrategy(config) {
  return {
    id: "my",
    title: "My Fetcher",
    available() { return Boolean(config.apiKey) }, // cheap check, no I/O
    async fetch(req, signal) {
      return { sources: [{ url, title, snippet, content, provider: "my" }], truncated: false }
    },
  }
}

Quick Start

1. Install

# from source
git clone https://github.com/runfali/dsh-web-fetch.git && cd dsh-web-fetch
pnpm install          # DSH loader resolves deps from plugin dir, not host

dsh plugin --profile web add ./dsh-web-fetch
# or after publishing:
# dsh plugin --profile web add dsh-web-fetch

# restart DSH
# systemd: sudo systemctl restart dsh

Why pnpm install? Cordis plugin loader resolves imports from the plugin directory only. Without it you'll get ERR_MODULE_NOT_FOUND: @deepseek-ai/schemastery.

2. Configure (UI recommended)

Open Settings → Plugin Config → 通用 Web 内容获取（web-fetch）

• Enable toggles — CDP / Tavily checkboxes at the top
• CDP group — CDP Endpoint (default http://10.200.0.5:9222), Timeout ms (60000), Extra wait after load ms (2000)
• Tavily group — Endpoint (https://api.tavily.com/extract), API Key (leave empty to disable), Timeout ms (30000)

Saved to ~/.dsh/settings.yaml under web-fetch: and hot-reloaded.

- id: web-fetch
  config:
    cdpEnabled: true
    cdpEndpoint: 'http://10.200.0.5:9222'
    cdpTimeoutMs: 60000
    cdpWaitMs: 2000
    tavilyEnabled: false
    tavilyEndpoint: 'https://api.tavily.com/extract'
    tavilyApiKey: ''
    tavilyTimeoutMs: 30000

3. Use

The LLM will see the tools automatically. Manual test:

User: 用 web_fetch_tavily 提取 https://example.com 的正文
User: 用 web_fetch_cdp 抓取 https://example.com 这个需要渲染的页面

Tool output shape:

{
  "sources": [{ "url": "...", "title": "...", "snippet": "...", "content": "...", "provider": "cdp|tavily" }],
  "truncated": false
}

Adding a New Data Source

1. Create src/strategies/my.js implementing FetchStrategy
2. Register in src/index.js:

import { makeMyStrategy } from "./strategies/my.js"
ctx.tools.register(makeToolDef("my", makeMyStrategy, "myEnabled", current))

3. (Optional) Add fields to Config + FIELD_VIEWS in lib/client.js

No changes to router or core logic needed.

Development

pnpm test          # full suite
pnpm test:host     # real-host contract tests only

All tests are offline (no real browser, no real Tavily call). tests/host-integration.test.mjs drives the plugin on the real dsh objects (@deepseek-ai/cordis, dsh-tools, dsh-system-prompt, dsh-settings-file) rather than hand-rolled stubs, and skips loudly when those packages are unresolvable.

Requirements

Why the disjunction? npm semver satisfies a prerelease only from a range group that itself carries a prerelease with the same [major, minor, patch] tuple. The previous single >=0.1.2-alpha.3 <0.2.0 group therefore did not cover 0.1.5-rc.1 — "claims 0.1.5 support but fails its own claim". The same trap applies to every dependency range on a dsh package. tests/entry.test.mjs pins this with a 10-row decision table, a counter-proof against the old range, and a line-by-line cross-check against the host's real semver.satisfies.

Limitations & Notes

• tavilyApiKey is stored as plain text in ~/.dsh/settings.yaml (settings system, not credential vault). For sensitive envs, override via profile cordis.patch.yml.
• CDP uses Node's native http + hand-rolled WebSocket frames (no ws dep). permessage-deflate is not used — compatible with cloakbrowser default (compression off).
• Version compatibility: see the Requirements table above (machine-readable in package.json under dsh.engines.dsh).

Contributing

PRs welcome! Please:

1. Keep zero extra deps
2. Add a strategy under src/strategies/ with unit tests
3. Update both zh/en locales in lib/client.js if adding settings

License

MIT © 2025 dsh-web-fetch

中文说明

完整中文文档请见 README.zh-CN.md。

dsh-web-fetch 是 DeepSeek Harness 的通用网页内容获取插件，提供 双数据源、可插拔策略 能力。核心设计是规避 registerSearchProvider 的 WEB_PROVIDER_AMBIGUOUS 限制，将每个数据源注册为独立工具，由 LLM 自主决策。lib/client.js 内置完整中英文界面。