dsh-web-fetch-playwright

中文 · npm · GitHub

A DeepSeek Harness (DSH) plugin that gives the built-in web_fetch tool a Playwright/CDP backend: pages are rendered in a real browser, denoised with Readability + DOMPurify + Turndown + GFM, and returned as Markdown.

Features

• Real browser rendering — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
• Denoise pipeline — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped tool-web renderer. Inline data: images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as ![alt](data:image/png;base64,...8.9KB) so they cannot flood the body.
• Two backends — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
• Browser resolution — a configured path, a playwright CLI on $PATH, or the bundled playwright-core; CDP needs no local browser at all.
• Isolated or profile sessions (CDP) — every fetch is scoped to exactly one tab. Local launches close their browser per fetch; the CDP backend keeps one shared connection to the remote browser and each fetch opens a tab inside it, closed when done. By default that tab lives in the remote browser's real profile (its cookies, localStorage, and persistent logins apply — like playwright-cli open); unchecking Share the browser context switches to a throwaway isolated context per fetch.
• Live configuration — a settings card (设置 → 插件 → 插件配置) edits the backend, context mode, denoise toggle, and concurrency; changes apply to the next fetch without a restart.
• Budget-aware — per-fetch deadline (45s); concurrency is backend-priced (maxConcurrency, default 4 local browsers / 50 CDP tabs; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
• Bounded Cloudflare-challenge wait — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented cf-mitigated: challenge response header plus structural page markers), the fetch keeps the same tab and context and waits for the browser's own verification to clear it — tracking the last main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded and configurable (challengeWaitMs, default 15s; 0 restores the legacy first-response behavior), with a bounded same-tab retry (challengeRetries, default 1). When the budget runs out, the fetch fails with the distinct WEB_FETCH_CHALLENGE error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.

How it works

web_fetch (tool-web)
   └─ ctx.web.fetchProvider = playwright
        ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
        ├─ cdp:   connectOverCDP(endpoint)
        ├─ page.goto → settle (networkidle, best-effort) → page.content()
        ├─ denoise: jsdom → elide data-URI images → Readability → DOMPurify → Turndown(GFM)
        └─ Markdown (or raw HTML when denoise is off)

Requirements

• DSH web profile (dsh web), Node.js ≥ 20.
• For the local backend: a Playwright installation with Chromium, a Chromium-family browser binary, or playwright-core with a browser in the default cache.
• For the CDP backend: any browser already running with --remote-debugging-port (e.g. chromium --headless --remote-debugging-port=9222).

Installation

From the npm registry (prebuilt — no build permission needed):

dsh plugin --profile web add dsh-web-fetch-playwright

From a GitHub repository (source — pnpm runs the prepare build; allowlist the package in profiles/web/pnpm-workspace.yaml if pnpm blocks the build script):

dsh plugin --profile web add github:chendefine/dsh-web-fetch-playwright

Or through the DSH plugin marketplace (设置 → DSH插件市场) — the repo carries the dsh-plugin topic and is indexed automatically.

After a bundle plugin is added to the profile layer stack, restart dsh web for it to load; uninstall with dsh plugin --profile web remove dsh-web-fetch-playwright and restart again.

Configuration

The settings card (设置 → 插件 → 插件配置 → Playwright 网页爬取) edits the web-fetch-playwright settings section live:

Local backend resolution order:

1. The configured path (auto-detected as Playwright CLI or browser binary).
2. A playwright executable on $PATH (its package knows that installation's browser registry).
3. The bundled playwright-core — requires PLAYWRIGHT_BROWSERS_PATH or browsers in the default cache; otherwise the error suggests playwright install chromium.

Windows note — $PATH is scanned with the platform delimiter (;), but npm/pnpm global installs expose playwright as .cmd/.ps1 shims whose location does not walk up to the package root, so discovery may still land on step 3 (the bundled core). To drive a specific installation's browser registry, point playwrightPath at the playwright package directory or a browser executable.

CDP mode needs no local browser: the provider holds one shared connection for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a tab in the remote browser that closes when the fetch completes. Concurrency therefore counts tabs, which is why the CDP default is high (50). Unloading the plugin drops the shared connection (the remote browser itself is never closed).

The CDP context modes (share the browser's profile or not)

With Share the browser context checked (default, profile mode), each fetch is a tab in the remote browser's default context — the real profile. Cookies and localStorage come from and are written back to that profile, so sites the browser is logged into are fetched logged-in, exactly like a tab you open by hand. The shared context is never closed; resource filtering and popup guards attach to the fetch's tab only, so your other tabs are untouched. With it unchecked (isolated mode), every fetch gets a fresh incognito-like context — anonymous reads, nothing persists.

Profile-mode risk notes — it upgrades web_fetch from "anonymous read" to "acts as the browser's logged-in user":

• A fetched page that talks the agent into a GET-style state-changing URL (logout, settings change, API call) will send it with your session cookies.
• Concurrent fetches to the same site share one cookie jar; one fetch's logout or Set-Cookie affects the others.
• Output starts depending on the browser's history (A/B buckets, language preferences). The remote profile also keeps accumulating site data; the plugin never cleans it.

Persistent logins require a persistent user-data-dir. Headful (recommended — log in by hand once):

google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome-dsh-profile"

Headless server (pre-seed logins in a headful environment first): chromium --headless=new --remote-debugging-port=9222 --user-data-dir=/data/chrome-dsh-profile. Do not add --incognito or a throwaway user-data-dir — either defeats profile mode. Design notes and verified playwright-core facts live in docs/context-mode-profile.md.

Cloudflare challenge handling (bounded natural wait)

Some strict sites serve a Cloudflare interstitial before the real page. A real browser often passes the check on its own within a few seconds — but a fetch that only looks at the first response hands you the interstitial as if it were the page (the pre-0.2.5 behavior; reproduce it any time with challengeWaitMs: 0, or — from a repo checkout, after pnpm build — run node scripts/challenge-demo.mjs for a local simulated before/after, and node scripts/challenge-online.mjs <url> against a real site).

With the wait on (default):

1. Detection — a response carrying cf-mitigated: challenge (the documented signal for every challenge page type), or a 403/503 HTML document from a server: cloudflare edge, or the localized interstitial itself (title family like "Just a moment…", 请稍候…, "Минутку…", plus structural markers: /cdn-cgi/challenge-platform/ scripts, #challenge-* elements, cf-chl-widget- frames, window._cf_chl_opt). The content-level markers are the fallback tier and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (server: cloudflare / cf-ray) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. Cloudflare Bot Management's passive JavaScript-Detections telemetry (/cdn-cgi/challenge-platform/scripts/jsd/, injected into every normal page of a protected zone — e.g. openrouter.ai) is neutralized before the prefix scan, so protected 200 pages with real content never misclassify. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
2. Bounded wait, same tab and context — the fetch polls the live DOM (500ms interval) for the challenge to disappear while the browser runs its own verification; the last main-frame navigation response is tracked so the reloaded real document's status and headers are the ones reported. SPA-style clears (content swapped without any navigation) are caught by the same DOM probe.
3. Bounded retry — when a window runs out, the same tab re-navigates once (default; challengeRetries) with whatever clearance cookies the context already holds.
4. Clear failure — WEB_FETCH_CHALLENGE (a provider-specific code the web seam's open-string code allows) naming the site, the budget spent, and the last challenge status.

Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in isolated mode the clearance a fetch's browser earns dies with that fetch's context; in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by challengeWaitMs and the 45s per-fetch deadline; nothing blocks forever.

Development

pnpm install
pnpm typecheck   # tsc --noEmit
pnpm test        # vitest run (browser smoke self-skips without a browser)
pnpm build       # tsc declarations + tsdown (host ESM + client module-registration bundle)

Repository layout:

src/
├── index.ts               # host entry: registers provider + settings section
├── config.ts              # schemastery schema, CDP endpoint normalizer
├── provider.ts            # WebFetchProvider: navigation, deadline, semaphore, caps
├── markdown.ts            # denoise pipeline (Readability + DOMPurify + Turndown/GFM)
├── playwright-resolve.ts  # local backend discovery (path / $PATH / bundled core)
├── types.ts               # structural Playwright types (runtime module discovered dynamically)
└── client/                # browser half: settings card, form model, locales
tests/                     # unit + provider + browser integration (self-skipping)

See CONTRIBUTING.md for the development and release workflow, and SECURITY.md for the security model and reporting policy.

Security

Same stance as the built-in HTTP provider: no SSRF / private-network protection is implemented — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself — but note that in profile mode the requests (and any state changes a malicious page tricks the agent into) carry the remote browser's logged-in sessions (see the risk notes above).

License

MIT © 2026 chendefine