sfyyy/dsh-vision-bridge 预览 preview

sfyyy/dsh-vision-bridge

插件Plugin ⭐ 6 MIT toolsvisual

On-demand vision for text-only DeepSeek Harness (DSH) sessions: images become markers, and a vision_describe tool sends only image + question to an OpenAI-compatible vision model

编辑导读Editor's Brief

## 核心特性
- 该插件通过注册用于图像问题的 vision_describe 工具,为纯文本 DeepSeek Harness 会话提供按需视觉能力。
- 文本模型接收改写后的文本标记而非图像块,同时会话日志和 UI 保留原始图像。
- 每次视觉请求仅将所选图像和一个聚焦问题发送到兼容 OpenAI 的视觉端点,而不发送会话历史。
- 该插件会将用户上传和工具生成的截图(包括嵌套在工具结果中的图像)记录到附件索引中。
- 配置可通过“设置 → Vision Bridge”、~/.dsh/vision-bridge.json 或受支持的环境变量管理,并有明确的优先级。

## 使用场景
- Agent 可以检查 DSH 会话期间生成的截图,而无需将主对话切换到多模态模型。
- 用户可以在文本模型保持不变的情况下,询问已上传图像、示意图或图表的问题。
- 当提供的图像总数在一到四张之间时,可以在一次 vision_describe 调用中比较多张图像。
- 当 DSH 的沙箱感知文件服务可以访问本地图像文件时,可以通过绝对路径分析这些文件。

## 技术细节
- 该包以 @dsh-extension/dsh-vision-bridge 发布,并通过 DSH 插件命令从 npm 安装。
- agent/pre-step 钩子构建附件索引,同时经过包装的 session.deriveMessages() 会阻止图像块到达文本模型。
- DSH 内置的 llm-pi-ai 构建 OpenAI 多模态请求,该插件维护单个 vision-bridge provider 路由。
- 该插件仅在启用时绕过 DSH Web 的图像能力准入检查,在禁用或卸载后恢复原生准入行为。

## 注意事项
- 图像分析需要一个独立的兼容 OpenAI 的 /v1/chat/completions 视觉端点和有效的 API key。
- vision_describe 总共接受一到四张图像,可使用当前会话中的附件 ID,或 PNG、JPEG、WEBP、GIF 格式的本地绝对路径。
- 将插件添加到 web profile 后,必须重启 dsh web,以便 UI 加载新的客户端 bundle。

## Core Features
- The plugin gives text-only DeepSeek Harness sessions on-demand vision by registering a vision_describe tool for image questions.
- The text model receives rewritten text markers instead of image blocks, while the session log and UI retain original images.
- Each vision request sends only selected images and a focused question to an OpenAI-compatible vision endpoint, not conversation history.
- The plugin records user uploads and tool-produced screenshots, including images nested in tool results, into an attachment index.
- Configuration can be managed through Settings → Vision Bridge, ~/.dsh/vision-bridge.json, or supported environment variables, with defined precedence.

## Use Cases
- Agents can examine screenshots generated during a DSH session without switching the main conversation to a multimodal model.
- Users can ask questions about uploaded images, diagrams, or charts while the text model remains unchanged.
- Multiple images can be compared in one vision_describe call when the total number of supplied images is between one and four.
- Local image files can be analyzed through absolute paths when the DSH sandbox-aware file service can access them.

## Technical Details
- The package is published as @dsh-extension/dsh-vision-bridge and is installed from npm with the DSH plugin command.
- An agent/pre-step hook builds the attachment index, while a wrapped session.deriveMessages() prevents image blocks from reaching the text model.
- DSH’s built-in llm-pi-ai constructs the OpenAI multimodal request, and the plugin maintains a single vision-bridge provider route.
- The plugin bypasses DSH Web’s image-capability admission check only while enabled, restoring native admission behavior when disabled or uninstalled.

## Notes
- A separate OpenAI-compatible /v1/chat/completions vision endpoint and valid API key are required for image analysis.
- vision_describe accepts one to four images in total, using current-conversation attachment IDs or absolute PNG, JPEG, WEBP, or GIF local paths.
- After adding the plugin to a web profile, dsh web must be restarted so the UI loads the new client bundle.

安装Install

dsh plugin --profile web add @dsh-extension/dsh-vision-bridge

sfyyy/dsh-vision-bridge 加入你的 DSH 配置(web profile)即可启用。

READMEREADME

上一个 Prev ds-api-usage 下一个 Next dsh-plugin-terminal