dsh-web-fetch-moli

中文 · GitHub

A DeepSeek Harness (DSH) plugin providing a Moli backend for the web_fetch tool. Built on Moli (an ultra-lightweight Rust headless browser), it renders dynamic micro-frontends, single-page applications (SPAs), and static sites with ~60MB memory footprint (>90% reduction compared to ~1GB for standard Chromium/Playwright), denoises pages via LinkeDOM + Readability + mdream, and returns clean, LLM-optimized GitHub Flavored Markdown.

Key Highlights

• Ultra-Lightweight Footprint — Consumes only ~50–60 MB RAM per local daemon, allowing high concurrency on resource-constrained servers without memory bloat.
• Enterprise Micro-Frontend & Dynamic SPA Compatibility — Bypasses Content Security Policy (Page.setBypassCSP), ignores certificate errors, and bypasses Service Worker caching. Drives native Moli layout tree computation via CDP Micro-Clip layout materialization (Page.captureScreenshot 1x1 micro-viewport) and triggers W3C standard scrollIntoView({ block: 'nearest' }) without any user-script monkey-patching, cleanly rendering complex micro-frontends (Alibaba Alfa, qiankun, single-spa) where other lightweight engines fail.
• Two Flexible Backends:
  • local (default): Automatically manages a local moli serve daemon over CDP, providing full SPA and micro-frontend execution with ~60MB memory footprint.
  • cdp: Connects to an existing remote Moli or Chromium CDP service over Chrome DevTools Protocol.
• High-Performance Two-Stage Denoise Pipeline — LinkeDOM + Mozilla Readability extracts primary content and eliminates noise (ads, nav bars, footers, forms), followed by mdream (Rust native with pure JS fallback) for high-fidelity, LLM-optimized Markdown conversion. Inline base64 images are elided to compact size placeholders.
• Cloudflare Challenge Resilience — Detects cf-mitigated: challenge interstitials and waits in the same page/context for natural clearance without artificial bot behavior.

Performance & Architecture Comparison

How it works

web_fetch (tool-web)
   └─ ctx.web.fetchProvider = moli
        ├─ local: resolves moli binary → manages `moli serve` daemon → connectOverCDP
        │           └─ setupPageHooks: Page.setBypassCSP + ignoreCertErrors + bypassServiceWorker
        ├─ cdp:   connectOverCDP(remoteEndpoint)
        ├─ page.goto → Micro-Clip layout materialization + native scroll loop → settle (networkidle) → page.content()
        ├─ denoise: LinkeDOM (DOM) → Readability (article extract) → mdream (Markdown)
        └─ Markdown (or raw HTML when denoise is disabled)

Requirements

• DeepSeek Harness (dsh web), Node.js ≥ 20.
• Zero manual setup: the plugin features automatic download on first use. If no moli binary is detected on your system, it automatically downloads and unpacks the latest release binary matching your OS and architecture (Linux, macOS, Windows x86_64 / arm64) into ~/.cache/moli.
• Existing installations of Moli v1.1.5+ are automatically detected on $PATH, ~/.local/bin/moli, or via the moliPath setting.

Manual installation (optional):

curl -sSL https://github.com/GreyRaphael/moli/releases/latest/download/moli-x86_64-unknown-linux-gnu.tar.gz | tar -xz -C ~/.local/bin
chmod +x ~/.local/bin/moli

Installation

# Option 1: Install directly via GitHub repository
dsh plugin --profile web add github:GreyRaphael/dsh-web-fetch-moli

# Option 2: Install via npm package name
dsh plugin --profile web add dsh-web-fetch-moli

After adding, restart dsh web to load the plugin.

Configuration

The settings card (设置 → 插件 → 插件配置 → Moli 网页爬取) configures web-fetch-moli in real time:

License

MIT