TC635807/better-crawler4agent

better-crawler-4-agent is a stdio Model Context Protocol server that turns arbitrary web URLs into clean article text for AI coding agents, using a real headless Chromium via Playwright so that JavaScript-rendered pages and Chinese anti-bot sites such as Zhihu, CSDN, Juejin, Douban, and Tencent Cloud Community are handled correctly. Its core capability is a three-tier fallback (real browser, trafilatura, httpx with full browser fingerprint), plus font honeypot detection, fake-success detection that catches 200 responses which are actually login walls or "article deleted" illustrations, automatic skeleton-page retries, SSRF protection, and per-tier diagnostics returned with every response. Integration is explicitly cross-platform: the generic .mcp.json works in Claude Code, Cursor, Windsurf, Cline, Codex, and ZCode, while the repository root ships a package.json declaring dsh.bundle.patch plus a dsh-plugin/ directory with cordis.patch.yml and install.sh, so DeepSeek Harness can install the whole repo as a bundle via dsh plugin --profile web add github:TC635807/better-crawler4agent or the local ./dsh-plugin/install.sh script, registering three MCP tools (mcp__better-crawler__fetch_url, mcp__better-crawler__fetch_urls, mcp__better-crawler__crawler_status) through @deepseek-ai/dsh-mcp-client.

The typical workflow is straightforward and aimed at developers and researchers who need a coding agent to actually read pages inside a session. A single tool call passes one URL or up to twenty concurrent URLs (the README recommends batching no more than eight at a time because most MCP clients impose a 30-second call timeout); the server warms or reuses a singleton Chromium instance, performs tiered fetching, detects skeleton pages whose HTML exceeds 30 KB but whose body text is under 1500 characters, retries them once without retrying deterministic failures like 404 and 403, and finally returns the longest successful payload along with the hit tier, character count, and an optional warning. URL discovery is deliberately out of scope, so callers must pick the pages themselves, and an accompanying vendor-neutral skill at skills/web-to-text/SKILL.md documents when to fetch, which URLs to avoid (tag pages, user pages, video pages), and how to judge whether a result is trustworthy, which agents that support skill directories can simply copy.

Dependencies, limits, and first-run caveats are spelled out clearly. The project requires Python 3.10 or newer and locks playwright==1.61.0 because Playwright binds strictly to browser revisions (1.61 → Chromium revision 1228, 1.62 → revision 1234), so upgrading Playwright without rerunning python -m playwright install chromium will silently break browser reuse. The bundled python scripts/install.py creates an in-repo .venv, probes the system for an existing matching browser to avoid re-downloading roughly 700 MB, prints ready-to-paste MCP configuration for every supported client, and offers flags such as --print-config, --reuse-env, --browsers-path, and --no-browser; verification is done via python scripts/selfcheck.py (15 online checks including real fetches) or python scripts/selfcheck.py --offline (12 offline checks), and unit tests cover SSRF, honeypot recognition, body extraction, and error-page identification. All tuning goes through environment variables including BETTER_CRAWLER_TIMEOUT (25 s), BETTER_CRAWLER_MAX_CHARS (50000), BETTER_CRAWLER_ALLOW_PRIVATE, BETTER_CRAWLER_PYTHON, BETTER_CRAWLER_BROWSERS_PATH, and BETTER_CRAWLER_LOG_LEVEL; SSRF blocks file://, raw protocols, loopback, private RFC1918 ranges, and cloud metadata 169.254.169.254 by default and only opens them when BETTER_CRAWLER_ALLOW_PRIVATE=1 is set explicitly for local debugging. The license is MIT, and the README reminds users to respect each target site's robots.txt and terms of service, warning against using the tool for high-frequency bulk scraping.