sfyyy/dsh-vision-bridge
On-demand vision for text-only DeepSeek Harness (DSH) sessions: images become markers, and a vision_describe tool sends only image + question to an OpenAI-compatible vision model
编辑导读Editor's Brief
## 核心特性
- 该插件通过注册用于图像问题的 vision_describe 工具,为纯文本 DeepSeek Harness 会话提供按需视觉能力。
- 文本模型接收改写后的文本标记而非图像块,同时会话日志和 UI 保留原始图像。
- 每次视觉请求仅将所选图像和一个聚焦问题发送到兼容 OpenAI 的视觉端点,而不发送会话历史。
- 该插件会将用户上传和工具生成的截图(包括嵌套在工具结果中的图像)记录到附件索引中。
- 配置可通过“设置 → Vision Bridge”、~/.dsh/vision-bridge.json 或受支持的环境变量管理,并有明确的优先级。
## 使用场景
- Agent 可以检查 DSH 会话期间生成的截图,而无需将主对话切换到多模态模型。
- 用户可以在文本模型保持不变的情况下,询问已上传图像、示意图或图表的问题。
- 当提供的图像总数在一到四张之间时,可以在一次 vision_describe 调用中比较多张图像。
- 当 DSH 的沙箱感知文件服务可以访问本地图像文件时,可以通过绝对路径分析这些文件。
## 技术细节
- 该包以 @dsh-extension/dsh-vision-bridge 发布,并通过 DSH 插件命令从 npm 安装。
- agent/pre-step 钩子构建附件索引,同时经过包装的 session.deriveMessages() 会阻止图像块到达文本模型。
- DSH 内置的 llm-pi-ai 构建 OpenAI 多模态请求,该插件维护单个 vision-bridge provider 路由。
- 该插件仅在启用时绕过 DSH Web 的图像能力准入检查,在禁用或卸载后恢复原生准入行为。
## 注意事项
- 图像分析需要一个独立的兼容 OpenAI 的 /v1/chat/completions 视觉端点和有效的 API key。
- vision_describe 总共接受一到四张图像,可使用当前会话中的附件 ID,或 PNG、JPEG、WEBP、GIF 格式的本地绝对路径。
- 将插件添加到 web profile 后,必须重启 dsh web,以便 UI 加载新的客户端 bundle。
## Core Features
- The plugin gives text-only DeepSeek Harness sessions on-demand vision by registering a vision_describe tool for image questions.
- The text model receives rewritten text markers instead of image blocks, while the session log and UI retain original images.
- Each vision request sends only selected images and a focused question to an OpenAI-compatible vision endpoint, not conversation history.
- The plugin records user uploads and tool-produced screenshots, including images nested in tool results, into an attachment index.
- Configuration can be managed through Settings → Vision Bridge, ~/.dsh/vision-bridge.json, or supported environment variables, with defined precedence.
## Use Cases
- Agents can examine screenshots generated during a DSH session without switching the main conversation to a multimodal model.
- Users can ask questions about uploaded images, diagrams, or charts while the text model remains unchanged.
- Multiple images can be compared in one vision_describe call when the total number of supplied images is between one and four.
- Local image files can be analyzed through absolute paths when the DSH sandbox-aware file service can access them.
## Technical Details
- The package is published as @dsh-extension/dsh-vision-bridge and is installed from npm with the DSH plugin command.
- An agent/pre-step hook builds the attachment index, while a wrapped session.deriveMessages() prevents image blocks from reaching the text model.
- DSH’s built-in llm-pi-ai constructs the OpenAI multimodal request, and the plugin maintains a single vision-bridge provider route.
- The plugin bypasses DSH Web’s image-capability admission check only while enabled, restoring native admission behavior when disabled or uninstalled.
## Notes
- A separate OpenAI-compatible /v1/chat/completions vision endpoint and valid API key are required for image analysis.
- vision_describe accepts one to four images in total, using current-conversation attachment IDs or absolute PNG, JPEG, WEBP, or GIF local paths.
- After adding the plugin to a web profile, dsh web must be restarted so the UI loads the new client bundle.
安装Install
dsh plugin --profile web add @dsh-extension/dsh-vision-bridge
把 sfyyy/dsh-vision-bridge 加入你的 DSH 配置(web profile)即可启用。
READMEREADME
@dsh-extension/dsh-vision-bridge
On-demand vision for text-only DeepSeek Harness (DSH) sessions.
A DSH plugin that gives a text-only DeepSeek session on-demand multimodal capability: the session stays on its text model for every turn, and only when the model actually needs to look at pixels — a screenshot, an uploaded image, a diagram, a chart — does it call the vision_describe tool, which sends only the image(s) + a focused question to an OpenAI-compatible vision model.
- No long context ever reaches the vision model — a 300k-token conversation history is never sent; each vision call is just image + question, keeping cost minimal.
- Session log and UI keep the original images — only the model input is rewritten to text markers.
- Bring your own vision endpoint — any OpenAI-compatible
/v1/chat/completionsservice (OpenAI, DeepSeek, Gemini proxy, local vLLM/One-API, …).
❤️ Sponsors
Want to appear here? — sponsor this project with an API donation.
![]() |
🎉 Thanks to xiaoyaoapi for donating their API to this project! xiaoyaoapi is an OpenAI-compatible AI API aggregation gateway for developers, built on New API with a unified admin dashboard. It offers unified key management, transparent usage tracking, and multi-channel access to mainstream large models under a single endpoint — letting developers integrate leading LLM services at lower cost and with greater convenience, ready to use as the vision endpoint of this plugin. |
How it works
User / tool produces an image ──► image stays in the session and UI
│
▼ (model input layer)
image is rewritten to a text marker
(marker carries the attachment id and hints
the model to call vision_describe)
│
▼
text model calls vision_describe(attachmentIds / paths, question)
│
▼
vision model (receives only image + question) → text answer
→ returned as a normal tool result
- The
agent/pre-stephook records every image attachment that appears in the session (user uploads and tool-produced screenshots, including ones nested insidetool-result), building an attachment index thatvision_describeuses to resolve bytes by id. session.deriveMessages()is wrapped so that no text-model request ever contains image blocks (the native DeepSeek adapter rejects them); images are replaced by text markers. The session event log and UI keep showing the original images.- DSH's built-in
llm-pi-aibuilds the OpenAI multimodal request; the plugin only maintains a singlevision-bridgeprovider route and does not re-implement a protocol adapter.
Image admission
DSH Web runs an image-capability check before a message enters the agent, based on the current DeepSeek model. This plugin keeps an admission bypass so images can enter the session first; the marker rewrite then guarantees no text-model request carries image blocks. When the plugin is disabled or uninstalled, the native admission check is restored.
Check GET /_dsh/vision-bridge/settings for the live value.admissionBypass and dependency-service status.
Installation
Install from the npm registry (not a local checkout) — one command:
# if you already have the `dsh` CLI on PATH:
dsh plugin --profile web add @dsh-extension/dsh-vision-bridge
# or, if you have been using npx all along:
npx @deepseek-ai/dsh@0.1.0-rc.6 plugin --profile web add @dsh-extension/dsh-vision-bridge
The
--profileflag targets the profile you boot (webis the browser UI profile). Omit it or adapt it if your profile has a different name.After a new client bundle is added, restart
dsh webonce so the UI picks it up.
Configuration
Configure it in Settings → Vision Bridge (DSH Web), or edit ~/.dsh/vision-bridge.json:
{
"enabled": true,
"baseUrl": "https://api.openai.com/v1",
"apiKey": "sk-xxxx",
"apiKeyEnv": "",
"model": "gpt-5.6-terra"
}
baseUrlaccepts an API root, a.../v1base, or a full.../chat/completionsURL (the plugin normalizes it).apiKeyandapiKeyEnvare mutually exclusive. A directly entered key is synced to the DSH credential store and referenced asDSH_VISION_BRIDGE_API_KEY.- The plugin maintains exactly one
vision-bridgeroute inside DSH'sllm-pi-ai.providersand never touches other providers. enabled: falsedisables the whole chain: no tool registration, no image rewriting, no admission bypass (native behavior restored).
Precedence (highest wins): Settings page (with schema defaults) → environment variables → config file.
Environment overrides: DSH_VISION_BRIDGE_BASE_URL, DSH_VISION_BRIDGE_API_KEY, DSH_VISION_BRIDGE_API_KEY_ENV, DSH_VISION_BRIDGE_MODEL, DSH_VISION_BRIDGE_ENABLED.
vision_describe tool
- Arguments
attachmentIds: image attachment ids from the current conversation (shaped likesha256:...), one or several;paths: absolute local image file paths (png/jpeg/webp/gif) — use either or both, 1–4 images in total;question: required — a focused, specific question about the image(s).
- Behavior: resolves the images → sends image(s) + question to the vision model via the
vision-bridgeroute → returns the text answer as a tool result. - Multi-image comparison is supported: put several images in the same user message.
- Attachment ids must come from the current conversation (user uploads or tool output);
pathsgo through DSH's sandbox-aware file service.
Verify
npm test
The suite covers: image-marker rewriting (including nested tool-result), both id- and path-based resolution, event-log attachment indexing, full-chain shutdown when disabled, and text-only sessions staying untouched.
Development
From a local checkout:
dsh plugin inject /path/to/dsh-vision-bridge
Search keywords
deepseek · deepseek-harness · dsh · plugin · vision · multimodal · vision-language-model · VLM · image understanding · screenshot · OCR · image analysis · OpenAI-compatible · text-only-llm · on-demand vision · LLM agent

amruthpillai/reactive-resume
tt-a1i/archify
strukto-ai/mirage
liustack/modlens
omdsh-dev/DSH-better-sidebar
dsh-market/dsh-market
GanyuanRan/Aegis