xwh-01/dsh-mediacrawler

可安装的DeepSeek Harness配置文件包及有界MCP适配器,用于MediaCrawler。

Project Overview项目介绍

This is a DSH-native installable profile bundle and bounded stdio MCP adapter built exclusively for DeepSeek Harness. It connects DeepSeek Harness to a separately cloned MediaCrawler installation, adding support for searching content, fetching post and video details, accessing creator feeds, and scraping explicitly enabled comments across eight major Chinese content platforms. It does not fork or modify MediaCrawler’s source code, so it leaves the original project’s license terms completely unchanged. It exposes 12 distinct MCP tools to manage every step of the scraping workflow, from checking environment readiness to exporting final credential-redacted results.

This tool is intended for developers and researchers who need to collect structured public content data from major Chinese social and content platforms. After you complete installation, you just need to export the required environment variables that point to your local MediaCrawler root, its Python executable, and the adapter’s state directory. The packaged skill included with the bundle will then guide the DSH agent through every step of the process, from dependency validation to starting a collection run, polling its status, and exporting the final results. On your very first use, you just need to ask the agent to call check(deep=true), which validates all local dependencies before any scraping begins.

Before you can install and run this adapter, you need to prepare several required dependencies on your local system first. You must have Python 3.11 or newer, Node.js 22.19+ (or Node 24+), pnpm added to your system PATH, Google Chrome installed, and a separate working MediaCrawler checkout with its own dedicated Python environment. The adapter’s own code is released under the permissive MIT license, while MediaCrawler remains a separate project under its original non-commercial learning license. While credential fields are redacted in all exports, this is not full PII anonymization, so you should handle all exported data with appropriate care.

这是一款专为DeepSeek Harness(DSH)开发的原生配置文件捆绑包与MCP适配器,用于将DSH连接到独立安装的MediaCrawler项目,支持搜索内容、获取帖子/视频详情、创作者主页和已开启权限的评论,爬取范围覆盖小红书、抖音、快手、B站、微博、贴吧、知乎七大中文内容平台,通过12个MCP工具暴露全流程管控能力。

面向需要从主流中文内容平台采集公开数据的开发人员与研究人员,完成环境变量配置后,包内自带的Skill会引导DSH智能体依次完成运行时检查、启动小规模爬收任务、轮询运行状态、导出结果数据等全流程操作。首次使用时,需要让智能体调用check(deep=true)命令验证整个环境的就绪状态。

本项目仅作为适配器存在,不打包也不修改MediaCrawler的源码,用户需要提前自行准备好Python 3.11+、Node.js 22.19+、pnpm包管理器、Google Chrome浏览器,以及独立检出并配置好工作环境的MediaCrawler项目。本项目适配器代码采用MIT许可,MediaCrawler仍遵循其原有的非商业学习许可协议。

Pre-install check安装前体检Compatibility · Security兼容性 · 安全性 1 warning1 项注意
  • Only 2 stars - very few users, little community feedback星标只有 2,几乎没人在用,遇到问题缺少社区反馈
DSH walks through these 9 checksDSH 会逐条核对这 9 项

Compatibility兼容性

  • DSH, Node, OS and profile requirementsDSH 版本 / Node 版本 / 操作系统 / profile 是否满足要求
  • External dependencies and runtimes (Electron / Python / Docker, ...)外部依赖与运行时(Electron / Python / Docker 等)是否齐备
  • Conflicts with installed plugins: command names, skill / tool names, ports, duplicate MCP registration与已装插件是否冲突:命令名、skill / tool 重名、端口占用、重复 MCP 注册

Security安全性

  • Repo matches the facts registered here; archived or abandoned?仓库是否与页面登记一致,是否归档或长期停更
  • Safety of preinstall / install / postinstall and install.sh / setup.ps1preinstall / install / postinstall 与 install.sh、setup.ps1 是否安全
  • curl|bash, download-then-execute, obfuscation, unrelated domains → stop immediatelycurl|bash、下载即执行、混淆代码、无关域名 → 立刻停止
  • Typosquatting or unmaintained packages among the new dependencies新增依赖里有没有 typosquatting 或无人维护的包
  • Requested permissions vs. what the feature actually needs申请了哪些权限、是否超出功能所需(filesystem / network / shell / clipboard)
  • Any sudo / admin requirement, plus uninstall and rollback是否要求 sudo / 管理员权限,以及卸载与回滚方式

Anything uncertain must be marked unknown with a note on how to confirm it. This site's signal screen is a static snapshot, not a security audit.拿不准的必须标「未知」并说明要我怎么确认。本站的信号筛查是静态快照,不能替代安全审计。

Or use CLI install (for developers)或使用命令行安装(适合开发者)

CLI Install命令行安装

dsh plugin --profile web add github:xwh-01/dsh-mediacrawler

把 xwh-01/dsh-mediacrawler 加入你的 DSH 配置(web profile)即可启用。

READMEREADME

dsh-mediacrawler

CI Release

English | 中文

An installable profile bundle and bounded stdio MCP adapter that connects DeepSeek Harness to a separately installed MediaCrawler checkout.

It supports search, post/video detail, creator feeds, and explicitly enabled comments on Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. Each run is supervised, persisted, and exposed through twelve MCP tools.

This is an adapter, not a MediaCrawler fork. It does not copy or modify MediaCrawler source code, and it does not change MediaCrawler's license.

Quick start

1. Prepare the runtimes

Install the following first:

  • Python 3.11 or newer.
  • Node.js 22.19+ on the 22.x line, or Node.js 24+, with pnpm on PATH.
  • Google Chrome.
  • A separate MediaCrawler checkout with its own working Python environment.
  • DeepSeek Harness. The commands below pin the tested 0.1.0-rc.6 release through npx.

MediaCrawler and its browser dependencies are intentionally not vendored here.

2. Install the Python MCP runtime

Keep the adapter in its own virtual environment. In PowerShell:

$adapterVenv = Join-Path $HOME '.dsh\runtimes\dsh-mediacrawler'
python -m venv $adapterVenv
$env:DSH_MEDIACRAWLER_PYTHON = Join-Path $adapterVenv 'Scripts\python.exe'
& $env:DSH_MEDIACRAWLER_PYTHON -m pip install --upgrade pip
& $env:DSH_MEDIACRAWLER_PYTHON -m pip install "dsh-mediacrawler @ git+https://github.com/xwh-01/dsh-mediacrawler.git@v0.3.0"

On POSIX systems:

python3 -m venv "$HOME/.dsh/runtimes/dsh-mediacrawler"
export DSH_MEDIACRAWLER_PYTHON="$HOME/.dsh/runtimes/dsh-mediacrawler/bin/python"
"$DSH_MEDIACRAWLER_PYTHON" -m pip install --upgrade pip
"$DSH_MEDIACRAWLER_PYTHON" -m pip install "dsh-mediacrawler @ git+https://github.com/xwh-01/dsh-mediacrawler.git@v0.3.0"

Showing the opening section of the README — the full document lives in the repository以上为 README 开头摘要,完整文档在仓库内 · View the full README on GitHub →在 GitHub 查看完整 README →

← 上一个 Prev dsh-webui-launcher 下一个 Next dsh-chatops →