dsh-web-fetch-moli

中文 · GitHub

A DeepSeek Harness (DSH) plugin providing a Moli backend for the web_fetch tool. Built on Moli (an ultra-lightweight Rust headless browser), it renders dynamic micro-frontends, single-page applications (SPAs), and static sites with ~60MB memory footprint (>90% reduction compared to ~1GB for standard Chromium/Playwright), denoises pages via LinkeDOM + Readability + mdream, and returns clean, LLM-optimized GitHub Flavored Markdown.

Key Highlights

• Ultra-Lightweight Footprint — Consumes only ~50–60 MB RAM per local daemon, allowing high concurrency on resource-constrained servers without memory bloat.
• Enterprise Micro-Frontend & Dynamic SPA Compatibility — Bypasses Content Security Policy (Page.setBypassCSP) and automatically triggers IntersectionObserver sentinels for structure-first geometry, cleanly rendering complex micro-frontend sandboxes (Alibaba Alfa, qiankun, single-spa) where other lightweight engines (Lightpanda, Obscura) fail.
• Three Flexible Backends:
  • local (default): Automatically manages a local moli serve daemon over CDP, providing full SPA and micro-frontend execution.
  • cdp: Connects to an existing remote Moli or Chromium CDP service over Chrome DevTools Protocol.
  • cli: Directly executes one-shot moli fetch, zero background daemon, ultra-fast cold start.
• High-Performance Two-Stage Denoise Pipeline — LinkeDOM + Mozilla Readability extracts primary content and eliminates noise (ads, nav bars, footers, forms), followed by mdream (Rust native with pure JS fallback) for high-fidelity, LLM-optimized Markdown conversion. Inline base64 images are elided to compact size placeholders.
• Cloudflare Challenge Resilience — Detects cf-mitigated: challenge interstitials and waits in the same page/context for natural clearance without artificial bot behavior.

Performance & Architecture Comparison

How it works

web_fetch (tool-web)
   └─ ctx.web.fetchProvider = moli
        ├─ local: resolves moli binary → manages `moli serve` daemon → connectOverCDP
        │           └─ setupPageHooks: Page.setBypassCSP + IntersectionObserver sentinel hook
        ├─ cdp:   connectOverCDP(remoteEndpoint)
        ├─ cli:   spawns `moli fetch <url> --dump html` (zero daemon)
        ├─ page.goto → sentinel triggers → settle (networkidle) → page.content()
        ├─ denoise: LinkeDOM (DOM) → Readability (article extract) → mdream (Markdown)
        └─ Markdown (or raw HTML when denoise is disabled)

Requirements

• DeepSeek Harness (dsh web), Node.js ≥ 20.
• Zero manual setup: the plugin features automatic download on first use. If no moli binary is detected on your system, it automatically downloads and unpacks the latest release binary matching your OS and architecture (Linux, macOS, Windows x86_64 / arm64) into ~/.cache/moli.
• Existing installations of Moli v1.1.5+ are automatically detected on $PATH, ~/.local/bin/moli, or via the moliPath setting.

Manual installation (optional):

curl -sSL https://github.com/lexmount/moli/releases/latest/download/moli-x86_64-unknown-linux-gnu.tar.gz | tar -xz -C ~/.local/bin
chmod +x ~/.local/bin/moli

Installation

# Option 1: Install directly via GitHub repository
dsh plugin --profile web add github:GreyRaphael/dsh-web-fetch-moli

# Option 2: Install via npm package name
dsh plugin --profile web add dsh-web-fetch-moli

After adding, restart dsh web to load the plugin.

Configuration

The settings card (设置 → 插件 → 插件配置 → Moli 网页爬取) configures web-fetch-moli in real time:

License

MIT