renjianguojinqianfan/dsh-skill-eval

Name: renjianguojinqianfan/dsh-skill-eval
Author: renjianguojinqianfan

Plugin插件 Native原生 ⭐ 2 MIT Prompts & Skills提示词与技能

这是对插件功能的描述，原文是：“DSH Plugin: Evaluate skill description trigger accuracy with LLM judge (under-trigger/over-trigger)”。翻译成中文如下：

Project Overview项目介绍

Edited by Zhiwei Agent (zhiwei)由知微Agent(zhiwei)编辑

dsh-skill-eval is a skill-trigger evaluation plugin built specifically for DeepSeek Harness (DSH), packaged and distributed as a native DSH extension through the dsh plugin --profile web add ./dsh-skill-eval command run from the repository root. After installation, users must declare a skill-eval block in their cordis.patch.yml overlay specifying the provider and model identifiers for the judge route, because the plugin validates that route against the DSH LLM runtime at startup and warns when the named provider has not yet been registered. The README makes no mention of Claude Code, Codex, Cursor, Gemini CLI, or any other agent platform, and the tooling aligns exclusively with DSH conventions such as the official dsh-tool-skill@0.1.0-rc.6 template and dsh-llm helper patterns, confirming its single-host design.

The typical workflow begins with the user invoking the /skill-eval <skill-name> [test-file] slash command in chat, or letting the model call the registered run_skill_eval(skill_name, test_file) tool, where the optional test_file argument defaults to the bundled examples/dsh-plugin-eval.json and any relative path is resolved against the plugin package directory. Internally the plugin enumerates every model-invocable skill from ctx.skills.snapshot, reproduces the official <system-reminder> plus <available_skills> catalog message verbatim, and asks the judge model a forced YES/NO routing question for each query, aggregating accuracy, precision, recall, false-positive and false-negative rates, and a confusion matrix. The intended audience is DSH integrators and skill authors who need a reproducible way to measure description quality, diagnose over- or under-triggering, and benchmark alternative judge models by swapping the configured provider.

Dependencies are minimal — pure Node.js with no external npm packages, and the README documents a complete verification suite including npm run check, npm test, npm run smoke (51 pure-function assertions), npm pack --dry-run, and a real DSH mount smoke script bash scripts/mount-smoke.sh, plus a fixture-refresh utility node scripts/refresh-catalog-fixture.mjs <path> that must be rerun after DSH upgrades to keep catalog fidelity pinned. The reported metrics measure the judge model's routing accuracy against the supplied skill description rather than live runtime behaviour, and the repository ships under the permissive MIT license with no cost, though first-time users should confirm the judge provider is registered in the DSH LLM runtime before issuing any evaluation request.

dsh-skill-eval 是一款面向 DeepSeek Harness（DSH）的技能触发评估插件，仓库本身即作为 DSH 原生扩展分发，安装需在仓库根目录执行 dsh plugin --profile web add ./dsh-skill-eval，随后在 cordis.patch.yml 中为 skill-eval 条目配置 provider 与 model 路由，DSH 启动时会对该路由进行校验，未注册的 provider 将被插件告警。该插件不涉及其他代码代理平台，README 中给出的命令、目录结构与官方 dsh-tool-skill 模板、dsh-llm 模式均以 DSH 为唯一宿主。

典型工作流为：在聊天中执行斜杠命令 /skill-eval <skill-name> [test-file]，或由模型调用 run_skill_eval(skill_name, test_file) 工具，test-file 可省略并将默认指向 examples/dsh-plugin-eval.json；插件会枚举 ctx.skills.snapshot 中的全部模型可调用技能，重现官方目录消息体，逐条交给 judge 模型作答 YES/NO 并产出准确率、精确率、召回率、混淆矩阵等指标。它适合希望量化技能描述路由质量、排查过触发与欠触发问题，并横向对比不同 judge 模型的 DSH 集成者与技能作者使用。

依赖与限制方面，插件以 Node.js 实现并通过 npm test、npm run check、npm run smoke 以及 bash scripts/mount-smoke.sh 完成校验，目录保真度夹具固定官方 dsh-tool-skill@0.1.0-rc.6 模板，DSH 升级后需用 node scripts/refresh-catalog-fixture.mjs 刷新夹具；评测结果反映的是 judge 模型本身的路由能力而非真实运行时；仓库以 MIT 协议发布，初次使用前请确保 provider 已在 DSH LLM 运行时中注册。

View on GitHub在 GitHub 查看

Send to your DSH发给你的 DSH 安装

请帮我安装这个 DSH 插件。安装前先完成【兼容性检查 + 安全性检查】，检查通过再动手。

插件：dsh-skill-eval（renjianguojinqianfan/dsh-skill-eval）
仓库：https://github.com/renjianguojinqianfan/dsh-skill-eval
本站详情页：https://www.yhbd.top/plugins/renjianguojinqianfan-dsh-skill-eval/
本站登记：类型 plugin · 归类 原生 DSH 插件 · 许可证 MIT · ⭐ 2 · 最近提交 2026-08-17 · 主语言 JavaScript

按下面顺序执行，每步先把结论告诉我，再进入下一步：

【1 兼容性检查】
① 我这边：DSH 版本、Node 版本、操作系统、当前 profile（web / desktop）。
② 读它的 README、package.json、插件 manifest，列出它要求的 DSH 版本 / Node 版本 / 操作系统 / 外部依赖 / 需要另外先装的运行时。
③ 逐条比对，结论只写「满足 / 不满足 / 未知」三种；不满足的给出可行替代方案。
④ 检查是否和我已装的插件冲突：命令名重复、skill / tool 重名、端口占用、重复注册的 MCP server。

【2 安全性检查】
① 仓库可信度：和上面「本站登记」是否一致；star / fork 数、创建时间、最近提交，是否归档或长期停更。
② 安装脚本：逐行看 package.json 的 preinstall / install / postinstall，以及 install.sh、setup.ps1 之类脚本。出现 curl|bash、下载后直接执行、混淆代码、访问与插件功能无关的域名，立刻停下来告诉我，不要继续装。
③ 依赖：列出新增依赖，标出无人维护、或与知名包拼写近似的可疑包（typosquatting）。
④ 权限与副作用：它会读写哪些目录、访问哪些域名、需要哪些 DSH 权限（filesystem / network / shell / clipboard 等），以及怎么卸载和回滚。
⑤ 如果它要求 sudo / 管理员权限，或权限明显超出功能所需，先停下来问我。

【3 安装】
上面两步没有「不满足」和「高危项」时才执行；用官方推荐方式安装，不要自行提权。

【4 汇报】
用表格输出：检查项 / 结论 / 依据 / 是否需要我决策。拿不准的一律写「未知」并说明要我怎么确认——不要猜，也不要替我决定。

Send this message to DSH in your current session: it verifies compatibility and security first (answering met / not met / unknown item by item) and only installs once everything checks out — it will stop and ask you if it finds a high-risk item. The box scrolls; the copy is the full prompt. CLI install commands may not be accurate across systems, so DSH is the safer route.把上面这条消息直接发给当前会话里的 DSH：它会先核对兼容性与安全性（逐条给「满足 / 不满足 / 未知」），确认没问题再安装，有高危项会停下来问你。框内可滚动，复制到的是完整提示词；安装命令不一定准确，发给 DSH 更稳。

Pre-install check安装前体检Compatibility · Security兼容性 · 安全性 1 warning1 项注意

Only 2 stars - very few users, little community feedback星标只有 2，几乎没人在用，遇到问题缺少社区反馈

DSH walks through these 9 checksDSH 会逐条核对这 9 项

Compatibility兼容性

DSH, Node, OS and profile requirementsDSH 版本 / Node 版本 / 操作系统 / profile 是否满足要求
External dependencies and runtimes (Electron / Python / Docker, ...)外部依赖与运行时（Electron / Python / Docker 等）是否齐备
Conflicts with installed plugins: command names, skill / tool names, ports, duplicate MCP registration与已装插件是否冲突：命令名、skill / tool 重名、端口占用、重复 MCP 注册

Security安全性

Repo matches the facts registered here; archived or abandoned?仓库是否与页面登记一致，是否归档或长期停更
Safety of preinstall / install / postinstall and install.sh / setup.ps1preinstall / install / postinstall 与 install.sh、setup.ps1 是否安全
curl|bash, download-then-execute, obfuscation, unrelated domains → stop immediatelycurl|bash、下载即执行、混淆代码、无关域名 → 立刻停止
Typosquatting or unmaintained packages among the new dependencies新增依赖里有没有 typosquatting 或无人维护的包
Requested permissions vs. what the feature actually needs申请了哪些权限、是否超出功能所需（filesystem / network / shell / clipboard）
Any sudo / admin requirement, plus uninstall and rollback是否要求 sudo / 管理员权限，以及卸载与回滚方式

Anything uncertain must be marked unknown with a note on how to confirm it. This site's signal screen is a static snapshot, not a security audit.拿不准的必须标「未知」并说明要我怎么确认。本站的信号筛查是静态快照，不能替代安全审计。

Or use CLI install (for developers)或使用命令行安装（适合开发者）

CLI Install命令行安装

dsh plugin --profile web add github:renjianguojinqianfan/dsh-skill-eval

把 renjianguojinqianfan/dsh-skill-eval 加入你的 DSH 配置（web profile）即可启用。

READMEREADME

3.4 KB · 缓存于 2026-09-25 22:08:37 · 3.4 KB

dsh-skill-eval

Skill-trigger evaluation plugin for DeepSeek Harness (DSH).

An LLM judge recreates the exact DSH skill catalog prompt and decides, for each test query, whether the target skill should be triggered. The plugin reports accuracy, precision, recall, false-positive/negative rates, and a confusion matrix — a reproducible measure of how reliably a skill description routes matching queries (and how often it over- or under-triggers).

Install

From the repo root:

dsh plugin --profile web add ./dsh-skill-eval

Then configure the judge model route in your profile/overlay cordis.patch.yml:

- id: skill-eval
  config:
    provider: <provider-id>
    model: <model-name>

The provider must be registered in the DSH LLM runtime (the same one your profile uses for chat). The plugin validates the route at startup and warns if the provider is not yet registered.

Usage

Slash command:

/skill-eval <skill-name> [test-file]

Model-callable tool:

run_skill_eval(skill_name="<skill-name>", test_file="examples/dsh-plugin-eval.json")

test-file is optional; it defaults to examples/dsh-plugin-eval.json inside the plugin package. Relative paths resolve against the plugin package directory.

Test-case format

A JSON array of { query, should_trigger } objects:

[
  { "query": "add a tool to the harness that persists across restarts", "should_trigger": true },
  { "query": "help me write a Python script for this CSV", "should_trigger": false }
]

category is optional and reserved for future use.

How it works

Enumerate the session's model-invocable skills (ctx.skills.snapshot).
Recreate the official catalog message verbatim (<system-reminder> + <available_skills> + normalized/truncated/escaped descriptions).
For each query, ask the judge model whether the target skill should be triggered, forcing a one-line YES/NO answer.
Compare against the expected label and aggregate metrics.

The evaluation measures the judge model's routing accuracy for the given skill description. Swap provider/model in the config to test other judges.

Showing the opening section of the README — the full document lives in the repository以上为 README 开头摘要，完整文档在仓库内 · View the full README on GitHub →在 GitHub 查看完整 README →

渲染基于仓库 renjianguojinqianfan/dsh-skill-eval 的 README.md。在 GitHub 查看原文 →