bobjia/dsh-context-milvus

Plugin插件 ⭐ 2 MIT Sessions & Context会话与上下文

claude-context-milvus like plugin for Deepseek Harness (DSH)

Project Overview项目介绍

This is an open-source plugin for DeepSeek Harness (DSH), built on Milvus vector database. It provides a full semantic code search pipeline with indexing and search. Use it on large codebases to improve DSH agent performance by cutting token usage and context pollution. It requires a running Milvus instance to work.

这是DeepSeek Harness基于Milvus的开源插件,提供完整的语义代码搜索索引与查询管线,支持代码索引、语义搜索、调用链追踪等功能。适用于大型代码库,可减少token消耗和上下文污染,提升DSH代理效率。注意需要预先部署可用的Milvus实例。

Or use CLI install (for developers)或使用命令行安装(适合开发者)

CLI Install命令行安装

dsh plugin --profile web add dsh-context-milvus

bobjia/dsh-context-milvus 加入你的 DSH 配置(web profile)即可启用。

READMEREADME

dsh-context-milvus

npm version Listed on dsh-plugin.org

English | 简体中文

A DSH plugin that provides semantic code search over a Milvus vector database, with a complete index ↔ search pipeline.

dsh-context-milvus = equips your DSH Agent with a dedicated codebase semantic search engine. Milvus handles high-speed vector retrieval, transforming "needle-in-a-haystack grep" into precise recall of relevant code snippets — reducing tokens, minimizing tool calls, and improving coding agent quality on large repositories.


Why dsh-context-milvus?

dsh-context-milvus is an open-source code semantic search plugin for DeepSeek Harness (DSH) coding agents, built on Milvus as the vector database and registered as a Cordis Plugin. Its core purpose: solve the high token consumption, excessive tool calls, context pollution, and poor large-codebase comprehension that plague native DSH Agent grep workflows.

Native DSH Agent workflow: encounter a problem → repeatedly search_code (grep) → read files → search again, flooding the prompt with irrelevant text, exploding tool calls, increasing token costs, and making it easy to miss dependencies in large repositories.

Solving the key pain points of native grep search

Native grep workflow pain point dsh-context-milvus solution
Literal string matching only — semantically related but differently named code is missed Vector semantic search — matches by code meaning, not just keywords
Multiple tool call rounds, reading many irrelevant files, token explosion Returns only truly relevant code snippets, split by AST at function/class boundaries for precision
Flooding context with grep output and irrelevant source code, causing context pollution and degraded model reasoning Milvus pre-built index, Agent gets concise effective context in one tool call without search noise in the prompt
Thousands of files in large repos, Agent traversal is extremely inefficient Milvus vector DB enables fast retrieval over millions of code blocks, supports incremental index updates without full repo rescanning
Can only search already-open or known-path files After full-repo indexing, can semantically search any code location regardless of file path knowledge

Features

  • search_code — Semantic code search: natural language query, returns matching code snippets
  • index_code — Index codebase: AST parsing + chunking → Embedding → Milvus storage
  • index_status — View index status: file count, last index time, hash statistics
  • find_callers — Code relationship analysis (impact analysis): find all references to a symbol, with cross-file import resolution
  • trace_call_chain — Call chain tracing: BFS expansion from entry symbol (impact/dependency analysis), with cross-file resolution disambiguation
  • Hybrid search — BM25 keyword + vector semantic dual-path retrieval, RRF fusion, hybridMode toggle
  • Chunk overlap — AST chunks include surrounding context lines (chunkContextLines, default 2) for better recall
  • Query expansion — expands natural-language queries with code synonyms before embedding (queryExpansion, default on)
  • Two-stage reranking — retrieves a topK×3 pool then applies proportional term-overlap & name-match boosts, keeping the Milvus score primary (rerankEnabled, default on)
  • Ignore pattern system — Three-layer gitignore-style ignore rules (default + codebase + global)
  • Incremental indexing — Merkle SHA-256 hash tracking, processes only changed files
  • Workspace isolation — Independent Merkle state files per workspace, no interference
  • ADR decision memory system — Records design rationale behind code changes (Architecture Decision Records), supports semantic search, CRUD, constraint injection, and consistency checking
  • Code relationship analysis — Extracts symbol references from AST during indexing (references, language-specific syntax nodes), supports cross-file exact matching
  • Cross-file import resolution (V2) — Scans import/export statements using tree-sitter AST during indexing, builds a persistent bidirectional Import Map, enabling find_callers/trace_call_chain to perform precise cross-file symbol matching (same-name disambiguation, cross-module tracing)
  • Native telemetry (opt-in)search_code / index_code / index_status write one JSONL line per execution (disabled by default, no source code captured), with an analysis script for descriptive stats + Bootstrap CI + correlation

Codex CLI support

The retrieval engine behind this plugin is published separately as dsh-context-milvus-core, so the same code also runs as a stdio MCP server for OpenAI Codex CLI (and any other MCP client) from the codex-context-milvus package. It exposes the five retrieval tools — search_code, index_code, index_status, find_callers, trace_call_chain — and shares the same Milvus collection and per-workspace index state as the DSH plugin. Start it with ADR_ENABLED=true and the 8 ADR decision-memory tools are registered as well; the four that write to disk stay gated behind CONTEXT_MILVUS_ADR_WRITE.

Shortest setup:

codex mcp add context-milvus -- npx -y codex-context-milvus mcp

See packages/codex/README.md for the init wizard, the environment variable reference, the error-code table, and current limitations (ADR tools are off by default, no runtime config hot-reload).

Offline / air-gapped install

An offline machine cannot resolve the ~230-package production closure, so the payload has to be built on a connected one. Seed a scratch install with the package as its only production dependency, then carry either the npm cache or the whole node_modules:

mkdir ctxmilvus-offline && cd ctxmilvus-offline
npm init -y
npm pkg set dependencies.codex-context-milvus=0.2.0
npm install --omit=dev --cache ./npm-cache

tar czf ctxmilvus-offline.tgz npm-cache package.json package-lock.json   # method A
tar czf ctxmilvus-tree.tgz node_modules                                  # method B

On the target, method A installs with npm ci --omit=dev --offline --cache ./npm-cache (--offline makes npm fail on a missing tarball instead of reaching for a network that is not there), while method B only needs the tree extracted and the binary invoked directly. Two traps: the generated config.toml starts the server with npx -y, which is a registry call on every Codex launch, so point command/args at the local bin/mcp.js; and the tree-sitter* prebuilds for linux/darwin/win32 × x64/arm64 come inside the tarballs, so one bundle is cross-platform — any other triple needs a compiler. The full recipe, including the prebuild-pruning trick and the exact TOML, is in packages/codex/README.md → Offline install.


Effectiveness Evaluation

A reproducible statistical evaluation suite (see scripts/eval/) quantifies how dsh-context-milvus improves retrieval quality and end-to-end agent efficiency. It covers offline retrieval quality, end-to-end agent evaluation, and native telemetry — using nonparametric statistics (Wilcoxon, Bootstrap CI, Cliff's Δ) with a unified file-level relevance standard. Run instructions and full reports live in scripts/eval/*/output/report.md.

Offline retrieval quality — 21 annotated queries × 19-file multi-language corpus

Three retrieval strategies compared: G (grep keyword), R (naive RAG: sliding-window + pure vector), P (plugin: AST chunking + BM25 hybrid + RRF + chunk overlap + query expansion + two-stage reranking).

Metric G (grep) R (naive RAG) P (plugin)
recall@10 0.9524 1.0000 0.9524
MRR 0.6754 0.9524 0.8452
nDCG@10 0.7446 0.9610 0.8725
hit@1 0.4762 0.9048 0.7619
precision@10 (file-level) 0.4203 0.1095 0.2730
precision@10 (chunk-level) 0.3619

Note on precision@10: Two metrics are reported. File-level precision@10 de-duplicates results by file path (each file counted once), measuring how many unique relevant files appear in the top-K. Chunk-level precision@10 counts each result independently, matching the classic IR definition: "of the 10 entries the Agent sees, how many are from relevant files?" The chunk-level metric is higher because the Agent benefits from multiple chunks of the same relevant file clustering in the top results.

Key findings:

  • Two-stage reranking lifts hit@1 by 6.7% (from 0.714 to 0.762 vs P0 baseline): proportional term-overlap (+30%) and name-matching (+15%) boosts improve first-hit accuracy. The gap to naive RAG (0.905) narrowed from 0.14 to 0.14.
  • Chunk-level precision@10 = 0.362: the Agent sees ~3.6 relevant entries per 10 results. This is bounded by the corpus (each query has only 1-2 relevant files, each producing few chunks) — the ceiling is determined by the number of chunks per relevant file, not search quality.
  • AST chunking + query expansion + chunk overlap drive precision: P vs R precision@10 +0.1635 (p=0.00013, Cliff's Δ=0.868 — large effect). Function/class-boundary chunks with surrounding context lines are far more focused than fixed sliding windows.
  • Semantic search beats keyword grep on ranking: P vs G MRR +0.1698 (p=0.108), nDCG@10 +0.1280 (p=0.100) — relevant files rank higher, with near-significant p-values.
  • grep precision is high but recall is brittle: G has the best precision@10 (0.4203) but the worst hit@1 (0.4762) — keyword-only search misses semantically-related code (e.g. "retry with exponential backoff" never matches withRetry).

End-to-end agent evaluation — 8 tasks × 3 runs × 3 strategies

Group Average pass rate Token consumption
G (grep) 37.5% baseline
R (naive RAG) 50.0% −928 vs G
P (plugin) 62.5% −2109 vs G

Key findings:

  • Highest task pass rate: P 62.5% vs G 37.5% vs R 50.0%.
  • Significant token reduction: P vs G Δmean −2109 tokens/task (95% CI [−2325, −1864]), Wilcoxon p=0.014, significant after Holm correction, Cliff's Δ=−1.0. P also beats R by −928 tokens/task (p=0.014).

Native telemetry (opt-in)

search_code / index_code / index_status record execution metrics (query, result count, top score, duration, files/chunks indexed, etc.) as one JSONL line per call — disabled by default (telemetryEnabled: false), no source code content captured. Run node scripts/eval/telemetry/run.mjs to generate a descriptive statistics + Bootstrap CI + correlation report from ~/.milvus-index/telemetry.jsonl.


What role does Milvus play, and why Milvus?

  1. Stores AST-chunked code vectors: dsh-context-milvus uses tree-sitter AST to split code at function/class/method boundaries, generates embeddings, and stores them in Milvus — avoiding cutting a function in half.
  2. High-performance vector search: Encodes the query and performs vector similarity search with low latency, suitable for real-time Agent tool calls.

    Note: BM25 keyword fusion is already implemented — Milvus native BM25 full-text search + vector semantic dual-path retrieval, RRF fusion (hybridMode enabled by default).

  3. Supports self-hosted Milvus / Zilliz Cloud, two deployment options; teams can control data; supports incremental indexing after code changes without full rebuild.
  4. Specifically adapted for code RAG: Supports path-scoped filtering (search_code path parameter), allowing directory-limited searches — ideal for codebase scenarios.

DSH plugin architecture advantages

It is not a standalone MCP service, but a DSH plugin (Cordis Plugin) embedded directly into the DSH Agent process:

  • Zero network overhead: Plugin and Agent share the same process, tool calls don't go through HTTP, latency far below MCP
  • Naturally shares DSH resource configuration: Reuses DSH's config management, environment variable injection, and logging system — no additional configuration needed
  • DSH Web GUI integration: Visual configuration through Settings → Plugins interface, no YAML hand-editing
  • DSH ecosystem compatibility: Shares the tool registry with other DSH plugins (bash, agent-loop, web-search, etc.), Agents can freely combine them

Core Workflow

Registered DSH tools

Tool Function Key Parameters
search_code Semantic code search query (natural language), topK (result count), path (search scope)
index_code Index codebase mode (full/incremental), path (target path)
index_status View index status path (view per-workspace status)
search_adr Semantic ADR search query (natural language), status, topK
search_adr_by_file Find ADRs by file path file_path (code file path), status
create_adr Create new ADR title (required), requirement, change_type
update_adr Update existing ADR adr_id (required), content, status
list_adrs List ADR records status, change_type, limit
load_constraints Load active ADR constraints adr_ids, format
check_adr_consistency Check ADR-code consistency file_path, fix
find_callers Find all references to a symbol for impact analysis, supports cross-file import resolution symbol (required), direction, maxResults, sourceFile, resolve
trace_call_chain BFS call chain tracing from entry symbol (impact/dependency analysis), supports import resolution disambiguation entry (required), direction, maxDepth, maxResults, resolve

Workflow

  1. Run index_code: Parse the project, split code blocks via tree-sitter AST → call Embedding API to generate vectors → store in Milvus collection.
  2. Agent encounters a coding problem, calls search_code for hybrid search (vector semantic + BM25 keyword, RRF fusion).
  3. Milvus returns the most relevant code snippets, injected into the Agent's context.
  4. Agent debugs, refactors, or develops based on precise context — no more frantic grep file reading.
  5. After code changes, run index_code mode=incremental to incrementally re-index only changed files.
  6. Check index status anytime with index_status (indexed files, total code blocks, last index time).
  7. Before modifying code, use find_callers for impact analysis: see which places reference the symbol to avoid missing cascading effects. For same-name symbols across files, use the sourceFile parameter to disambiguate by definition file.
  8. Understand call chains with trace_call_chain: BFS expansion from entry function, direction=backward traces callers, direction=forward traces downstream dependencies. resolve: false falls back to V1 name-matching mode.
  9. Cross-file reference analysis: find_callers and trace_call_chain enable import resolution by default (resolve: true). The Import Map built during indexing automatically maps import { X } from './foo' to foo.ts's exports, eliminating same-name ambiguity and supporting cross-module call chain tracing. Falls back to V1 name matching when the import map is not built.

ADR decision memory workflow

The ADR decision memory system records the "why" behind code changes (design decisions, trade-offs, constraints), enabling the Agent to not only read code but understand its evolution:

Note: ADR functionality is disabled by default. To enable it, set adrEnabled: true in the DSH config panel (Settings → Plugins → dsh-context-milvus).

  1. Before modifying code with ADR coverage, use search_adr_by_file to check if the file has decision records, avoiding violation of existing decisions.
  2. When making design decisions, use create_adr to record the context, alternatives, and rationale, and use update_adr to maintain code_anchors linking to code locations.
  3. When needing to understand constraints, use load_constraints to load active ADR constraints into the context.
  4. After creating or updating ADRs, use check_adr_consistency to verify ADR-code consistency, with fix for auto-repair.
  5. Use search_adr for semantic search of historical decisions, understanding "why this was done this way."

Spec Document Fusion

When the brainstorming skill produces specification documents, they can be linked to the codebase through the following steps:

  1. Write spec documents: brainstorming output saved to docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md
  2. Generate anchors: Call index_specs to automatically detect code references in the document and generate frontmatter + code_anchors
  3. Index: index_code automatically scans docs/superpowers/specs/ and docs/superpowers/plans/ directories
  4. Discover: search_adr returns both ADR and spec document results (with docType annotation)

Configuration

Field Default Description
specRoot docs/superpowers/specs Spec document directory (relative to indexRoot)
planRoot docs/superpowers/plans Implementation plan directory (relative to indexRoot)

Spec document fusion follows the adrEnabled toggle — no additional configuration needed.


Prerequisites

1. Install Ollama (Embedding service)

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start Ollama service
ollama serve

Or use any OpenAI-compatible Embedding API service (OpenAI, Alibaba Cloud Bailian, etc.) by configuring embeddingEndpoint and embeddingApiKey.

2. Install Embedding model

# Pull nomic-embed-text model (default)
ollama pull nomic-embed-text

# Or other supported Embedding models:
ollama pull bge-m3
ollama pull mxbai-embed-large

3. Install Milvus (vector database)

Docker (recommended):

# Pull and start Milvus standalone
docker run -d --name milvus \
  -p 19530:19530 \
  -p 9091:9091 \
  milvusdb/milvus:latest

# Verify connection
docker ps | grep milvus

Milvus cluster mode (Docker Compose):

# Download docker-compose file
wget https://github.com/milvus-io/milvus/releases/latest/download/milvus-standalone-docker-compose.yml -O docker-compose.yml

# Start
docker compose up -d

Or use Zilliz Cloud managed service — no self-hosting required.

Verify installation

# Verify Ollama
curl http://localhost:11434/api/tags

# Verify Milvus
docker run -it --rm \
  -e MILVUS_URL=localhost:19530 \
  milvusdb/milvus-sdk-node:latest \
  node -e "const {MilvusClient} = require('@zilliz/milvus2-sdk-node'); \
  new MilvusClient({address:'localhost:19530'}).listCollections().then(r=>console.log(r))"

Install to DSH

Method 1: From npm (recommended)

The plugin is published to the npm registry. Install directly via DSH CLI:

dsh plugin --profile web add dsh-context-milvus

The npm package includes pre-built dist/ output — no build step required during installation, avoiding the ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED error.

Method 2: From local tarball (offline / local development)

Build and package as a tarball, then install directly:

# 1. Build
npm run build

# 2. Package as tarball
pnpm pack

# 3. Install to profile
dsh plugin --profile web add ./dsh-context-milvus-0.1.3.tgz

pnpm pack produces a tarball containing the compiled dist/ output — no build step required during installation, so pnpm won't raise ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED.

Method 3: From Git (requires additional configuration)

dsh plugin --profile web add git+https://github.com/bobjia/dsh-context-milvus.git

dist/ output is not committed to git. The plugin uses the prepare script to automatically run tsc during installation.

pnpm 10 limitation: pnpm 10 blocks execution of build scripts by default. If you see:

ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED
The git-hosted package "dsh-context-milvus@0.1.2" needs to execute build scripts
but is not in the "onlyBuiltDependencies" allowlist.

Add to your profile's pnpm-workspace.yaml:

# ~/.dsh/profiles/<profile-name>/pnpm-workspace.yaml
onlyBuiltDependencies:
- dsh-context-milvus

Then re-run the install command. Or run pnpm approve-builds and select dsh-context-milvus.

To avoid this authorization, use Method 1 (npm) or Method 2 (tarball).

Configure the plugin

After installation, edit cordis.patch.yml under your profile:

# ~/.dsh/profiles/<profile-name>/cordis.patch.yml
- id: dsh-context-milvus
  config:
    milvusAddress: localhost:19530
    milvusCollection: code_embeddings
    milvusDim: 768
    embeddingEndpoint: http://localhost:11434/api/embed
    embeddingModel: nomic-embed-text
    indexRoot: /path/to/your/code
    indexExtensions: .ts,.tsx,.js,.py,.java,.go,.rs,.cpp,.cs,.scala,.php
    hybridMode: true
    bm25RrfK: 60

Restart DSH after configuration.

Build from source (local development)

If using a local development version:

1. Install dependencies

cd /mnt/home/bobjia/workspace/dsh-context-milvus
npm install --legacy-peer-deps

2. Create symlinks for @deepseek-ai packages

# Link DSH runtime packages (npm install may break these links)
ln -sf /mnt/home/bobjia/.npm-global/lib/node_modules/@deepseek-ai/dsh/node_modules/@deepseek-ai/cordis \
  node_modules/@deepseek-ai/cordis
ln -sf /mnt/home/bobjia/.npm-global/lib/node_modules/@deepseek-ai/dsh/node_modules/@deepseek-ai/dsh-tools \
  node_modules/@deepseek-ai/dsh-tools
ln -sf /mnt/home/bobjia/.npm-global/lib/node_modules/@deepseek-ai/dsh/node_modules/@deepseek-ai/schemastery \
  node_modules/@deepseek-ai/schemastery

3. Register with DSH

# Install as local dependency
dsh plugin --profile web add file:/mnt/home/bobjia/workspace/dsh-context-milvus

dsh plugin add automatically adds the plugin to dsh.profile.bundles — no need to manually edit package.json.

4. Configure plugin

Edit ~/.dsh/profiles/<profile-name>/cordis.patch.yml (same as above) and restart DSH.


Configuration System

Priority (highest → lowest)

  1. Cordis Config (set via cordis.patch.yml or DSH Web GUI)
  2. Environment variables (fallback)
  3. Defaults (e.g., localhost:19530)

Configuration fields

Field Environment Variable Type Default Description
milvusAddress MILVUS_ADDRESS string localhost:19530 Milvus server address
milvusToken MILVUS_TOKEN string (secret) empty Milvus auth token
milvusCollection MILVUS_COLLECTION string code_embeddings Collection name
milvusDim MILVUS_EMBEDDING_DIM number 768 Vector dimension
embeddingEndpoint EMBEDDING_ENDPOINT string http://localhost:11434/api/embed Embedding API URL
embeddingApiKey EMBEDDING_API_KEY string (secret) empty Embedding API key
embeddingModel EMBEDDING_MODEL string nomic-embed-text Embedding model name
indexRoot INDEX_ROOT string process.cwd() Code repository root path
indexExtensions INDEX_EXTENSIONS string all supported extensions File extensions to index (comma-separated)
hybridMode HYBRID_MODE boolean true Enable hybrid search (BM25 full-text + vector semantic, RRF fusion)
bm25RrfK BM25_RRF_K number 60 RRF fusion parameter k
indexIgnoreDirs INDEX_IGNORE_DIRS string dist, build, target, vendor, ... Directories to skip during scan
ignorePatterns IGNORE_PATTERNS string (textarea) empty Custom gitignore-style ignore rules
merkleFilePath MERKLE_FILE_PATH string ~/.milvus-index/merkle-{name}-{hash}.json Merkle state file path

Tool Reference

search_code

Semantic code search. Automatically invoked when the user asks about code functionality, logic, or needs to find code by natural language.

Parameters:

Parameter Type Required Default Description
query string yes Natural language query
topK number no 5 Maximum results to return
path string no (configured root) Path scope for search

Return format:

[
  {
    "filePath": "src/auth/login.ts",
    "content": "export async function loginUser(credentials) { ... }",
    "score": 0.92,
    "language": "typescript",
    "chunkType": "function_declaration",
    "name": "loginUser",
    "startLine": 42,
    "endLine": 68
  }
]

index_code

Index the codebase. Supports two modes:

  • full — Full index of all files
  • incremental — Incremental index (only changed files, based on Merkle hash)

Parameters:

Parameter Type Required Default Description
mode string no incremental Index mode: full or incremental
path string no (configured root) Path to index

index_status

View index status, including file count, total code blocks, last index time, etc.

find_callers

Find all references to a symbol (function/variable/class) in the codebase, for impact analysis. V2 adds cross-file import resolution: use sourceFile to disambiguate same-name symbols across files.

Parameters:

Parameter Type Required Default Description
symbol string yes Symbol name to find (function, variable, class)
direction string no backward backward=who references me (impact); forward=who I reference (dependency)
maxResults number no 20 Maximum results
sourceFile string no Definition file path (explicit disambiguation: only return callers that import from this file)
resolve boolean no true Whether to enable import resolution (false falls back to V1 name-matching)

Return format:

{
  "chunks": [
    {
      "filePath": "src/auth/login.ts",
      "content": "export async function loginUser(credentials) { ... }",
      "startLine": 42,
      "endLine": 68,
      "chunkType": "function_declaration",
      "name": "loginUser",
      "resolution": {
        "status": "resolved",
        "targetFile": "src/auth/session.ts",
        "exportedAs": "loginUser"
      }
    }
  ]
}

resolution field: status is resolved (resolved to a cross-file import), local (defined in the same file), or unresolved (fallback to V1 name-matching). Only present when import resolution is enabled and the Import Map is built.

trace_call_chain

Starting from the entry symbol, BFS-traverses the call chain along reference relationships. direction=backward for impact analysis (find who calls the entry), direction=forward for dependency analysis (what the entry calls). Uses a visited set to prevent cycles. V2 supports import resolution disambiguation (resolve: true by default), using filePath:symbol composite keys for cross-file call chain tracing.

Parameters:

Parameter Type Required Default Description
entry string yes Entry symbol name
direction string no backward Traversal direction
maxDepth number no 3 Maximum recursion depth
maxResults number no 10 Maximum results per level
resolve boolean no true Whether to enable import resolution (false falls back to V1)

Return format:

{
  "chain": [
    {
      "depth": 0,
      "symbol": "main",
      "filePath": "src/index.ts",
      "startLine": 1,
      "endLine": 5,
      "callers": ["runApp"]
    },
    {
      "depth": 1,
      "symbol": "runApp",
      "filePath": "src/app.ts",
      "startLine": 10,
      "endLine": 20,
      "callers": ["initConfig"]
    }
  ]
}

Code Chunking

Language Extensions Chunking method Covered AST node types
TypeScript .ts, .tsx, .mts, .cts tree-sitter function_declaration, method_definition, class_declaration, interface_declaration, enum_declaration, type_alias_declaration, arrow_function, generator_function, getter, setter
JavaScript .js, .jsx, .mjs, .cjs tree-sitter function_declaration, method_definition, class_declaration, arrow_function, generator_function, getter, setter
Python .py tree-sitter + regex fallback function_definition, class_definition, async_function_definition, decorated_definition
Java .java tree-sitter + regex fallback class_declaration, interface_declaration, enum_declaration, method_declaration, constructor_declaration, record_declaration
Go .go tree-sitter + regex fallback function_declaration, method_declaration, type_declaration, type_spec
Rust .rs tree-sitter + regex fallback function_item, impl_item, trait_item, struct_item, enum_item, macro_definition
C++ .cpp, .cxx, .cc, .hpp, .h, .hh tree-sitter + regex fallback function_definition, class_specifier, namespace_definition, struct_specifier, enum_specifier
C# .cs tree-sitter + regex fallback method_declaration, class_declaration, interface_declaration, struct_declaration, enum_declaration
Scala .scala tree-sitter + regex fallback class_definition, function_definition, trait_definition, object_definition, constructor_definition
PHP .php regex fallback function_definition, class_declaration, interface_declaration, trait_declaration, enum_declaration

All languages except PHP (regex-only) use tree-sitter AST parsing as the primary method. Python, Java, Go, Rust, C++, C#, and Scala automatically fall back to regex when tree-sitter parsing fails; TypeScript / JavaScript have no regex fallback — if tree-sitter parsing fails, the file is skipped (no index entry).


Ignore Pattern System (IgnoreMatcher)

Three-layer gitignore-style file ignore rules, ensuring only the code files that need analysis are indexed:

Three rule layers

  1. Built-in defaults: Automatically excludes node_modules/, dist/, build/, .git/, __pycache__/, *.log, *.min.js, and 30+ common build artifacts and dependency directories
  2. Codebase ignore files: Automatically reads .gitignore, .ignore, .xxxignore, etc. from the codebase root
  3. Global ignore file: Reads ~/.context/.contextignore (user-level global rules)

Automatic hidden path protection

Automatically ignores path segments starting with . (e.g., .git/, .vscode/, .env), preventing hidden directories and files from being indexed.

Backward compatibility

The indexIgnoreDirs config (comma-separated directory names) is automatically converted to gitignore-style patterns (e.g., dist**/dist/**), maintaining compatibility with older versions.


Incremental Indexing & Workspace Isolation

Incremental Indexing (Merkle hash tracking)

  • Uses SHA-256 hash tracking for each file's content changes
  • Only re-indexes new or modified files; skips unchanged files
  • Deleted files are automatically removed from Milvus
  • State is persisted to a local JSON file

Workspace Isolation

  • Different workspaces use independent Merkle state files
  • State file paths are generated based on the workspace path's SHA-256 hash
  • Indexing different workspaces does not interfere with each other
  • The path parameter in tool calls specifies the workspace, automatically using the corresponding state file

When to Use (and When Not To)

✅ Suitable Scenarios

  • Codebases from tens of thousands to millions of lines, using DSH Agent for refactoring, bug localization, or cross-file reading
  • Want to reduce token overhead and minimize Agent grep tool loops
  • Need an open-source, self-hostable solution, avoiding closed-source indexing services
  • Already using the DSH framework and want to enhance Agent code comprehension
  • Need incremental indexing — code changes frequently but don't want full rebuilds every time

❌ Not Suitable / Caveats

  1. Requires an embedding API (OpenAI / Ollama, etc.), code snippets are sent to the embedding service during indexing; for high-privacy requirements, use Ollama local embeddings
  2. Adds Milvus / Zilliz Cloud as a dependency, increasing operational complexity; small codebases (a few hundred files) may not see significant benefit
  3. It is a retrieval augmentation tool, not a replacement for the model's context window — it filters high-quality context to solve "signal overload," not to infinitely expand the window
  4. Requires DSH environment (v0.6+), cannot run independently of DSH

Comparison: DIY Code RAG vs dsh-context-milvus

If you build your own code RAG for DSH Agent: you'd need to handle AST chunking, vector search tuning, incremental sync, DSH tool wrapping, result ranking, and ignore file systems. dsh-context-milvus packages all of this engineering into a plug-and-play solution, specifically tuned for code scenarios.

Dimension DIY Code RAG dsh-context-milvus
AST Chunking Integrate tree-sitter yourself, configure per language Built-in 10-language tree-sitter chunking, auto fallback to regex
Semantic Search Call embedding service and tune parameters yourself Built-in vector semantic search, plug-and-play (BM25 keyword fusion)
Incremental Indexing Implement file hash comparison and state management yourself Built-in Merkle file state tracking, SHA-256, incremental updates
Workspace Isolation Handle multi-workspace state conflicts yourself Automatic path-hash-based isolation, no interference
Ignore Files Implement .gitignore parsing yourself Built-in three-layer ignore rule system (default + codebase + global)
DSH Tool Wrapping Wrap DSH tools yourself (defineTool) 13 native DSH tools (5 code tools + 8 ADR tools), one-click registration, formatted output
Configuration UI Build yourself or hand-write YAML DSH Web GUI visual configuration, 13 config fields
Config Sources Single source Three-source merge (Cordis Config > env vars > defaults)
Index Status Build yourself Built-in index_status tool, real-time index status

DSH Web Configuration

After installation, go to the DSH Web interface (http://127.0.0.1:3080) Settings → Plugins to see dsh-context-milvus and its configuration form, supporting:

  • Text inputs (standard fields)
  • Password inputs (secret fields like milvusToken, embeddingApiKey)
  • Number inputs (number fields like milvusDim)
  • Toggles (boolean fields like hybridMode)
  • Field descriptions / help text

Architecture

┌──────────────────────────────────────────┐  ┌──────────────────────────────────────┐
│      DSH Agent / Web UI (13 tools)       │  │   OpenAI Codex CLI / any MCP client  │
│  search_code │ index_code │ index_status │  │        (5 tools, MCP stdio)          │
│  find_callers │ trace_call_chain         │  │  search_code │ index_code │ ...      │
│  8 × ADR tools (decision memory)         │  │   ADR tools need ADR_ENABLED         │
└────────────────────┬─────────────────────┘  └───────────────────┬──────────────────┘
                     │                                            │
       packages/dsh (Cordis adapter)              packages/codex (MCP adapter + CLI)
       tools.ts / adr-tools.ts /                   server.ts / handlers.ts /
       constraint-injector.ts                      init-wizard.ts / doctor.ts
                     │                                            │
                     └────────────────────┬───────────────────────┘
                                          ▼
                    packages/core — dsh-context-milvus-core (framework-agnostic)
   ┌──────────────────────────────────────────────────────────────────────────────┐
   │  chunker (AST+regex) → embedding → milvus-service        merkle (SHA-256 Δ)  │
   │  code-relations (BFS findCallers/traceChain)             import-resolver     │
   │  query-expansion → reranker                              ignore-matcher (3层) │
   │  telemetry (JSONL, opt-in)                               logger port         │
   │  ADR engine: frontmatter/chunker/anchors/service/indexer/bundle              │
   └──────────────────────────────────────────────────────────────────────────────┘
                                          │
                          ┌───────────────┴───────────────┐
                     ┌──────────┐                   ┌──────────┐
                     │  Milvus  │                   │Embedding │
                     │(vector DB)│                  │   API    │
                     └──────────┘                   └──────────┘

Both adapters depend on the core package; they never depend on each other. The core boundary is machine-enforced: it may not import @deepseek-ai/*, @modelcontextprotocol/* or zod, and may not call console.* directly (logging goes through the injected Logger).

Module dependency graph

Core (packages/core/src/, imported by adapters only through the index.ts barrel):

index.ts (barrel)
  ├── config.ts     — Config resolution (adapter config > env vars > defaults)
  │     └── DEFAULT_IGNORE_PATTERNS — Built-in gitignore-style ignore rules
  ├── milvus-service.ts — Milvus vector DB client wrapper (CRUD, search, ADR collection)
  │     ├── embedding.ts — OpenAI-compatible Embedding API client
  │     ├── query-expansion.ts / reranker.ts — retrieval quality stages
  │     └── logger.ts — Logger port (consoleLogger / silentLogger)
  ├── merkle.ts     — SHA-256 hash tracker (incremental indexing, persisted to JSON)
  ├── code-relations.ts — Code relationship analysis engine (BFS call chain + dedup)
  │     └── import-resolver.ts — Cross-file Import Map (tree-sitter AST import/export scan)
  ├── ignore-matcher.ts — gitignore-style pattern matching (file exclusion)
  └── indexer.ts    — Indexing pipeline orchestration
        └── chunker.ts — tree-sitter AST chunking + regex fallback (references extraction + language import/export config)

DSH adapter (packages/dsh/src/plugins/dsh-context-milvus/):

index.ts        — Cordis entry: bootstrap, settings panel, register 13 tools
tools.ts        — DSH tool definitions, formatting, workspace-aware tracker creation
adr-frontmatter.ts — YAML frontmatter parsing
adr-chunker.ts     — Markdown section chunking
adr-anchor-index.ts / adr-anchor-generator.ts — code_anchors index + generation
adr-service.ts     — ADR CRUD + state management
adr-indexer.ts     — ADR indexing pipeline
adr-tools.ts       — 8 ADR tools
constraint-injector.ts — System prompt injection + re-injection

Codex adapter (packages/codex/src/): workspace-resolver.tscontext.ts (stderr logger) → workspace-services.ts (per-root cache) → handlers.ts (5 tools) → server.ts (MCP wiring) + result-format.ts / schemas.ts, plus init-wizard.ts and doctor.ts for the CLI.


Testing

# Run all tests (single root Jest project across the three packages)
npm test

# Test coverage
npm run test:coverage

# Single test file — Jest must run under --experimental-vm-modules here,
# so plain `npx jest <file>` fails with "Cannot use import statement outside a module"
node --experimental-vm-modules node_modules/.bin/jest packages/core/test/dsh-context-remdb.spec.ts

# Code relationship analysis tests
node --experimental-vm-modules node_modules/.bin/jest packages/core/test/code-relations.spec.ts

# Cross-file Import Resolution tests
node --experimental-vm-modules node_modules/.bin/jest packages/core/test/import-resolver.spec.ts

# core boundary guard + DSH contract freeze
node --experimental-vm-modules node_modules/.bin/jest packages/core/test/core-boundary.spec.ts
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/public-surface.spec.ts

# ADR module tests
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/adr-frontmatter.spec.ts
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/adr-chunker.spec.ts
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/adr-anchor-index.spec.ts
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/adr-service.spec.ts
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/adr-indexer.spec.ts
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/adr-tools.spec.ts
node --experimental-vm-modules node_modules/.bin/jest packages/dsh/test/constraint-injector.spec.ts

# MCP server smoke test (spawns the built bin over real stdio — build first)
npm run build && node --experimental-vm-modules node_modules/.bin/jest packages/codex/test/mcp-smoke.spec.ts

Development

# Install (see the note below about the peer conflict)
npm install --legacy-peer-deps

# Build all packages in order: core → dsh → codex
npm run build

# Type check all packages (builds core first, since adapters resolve its .d.ts)
npm run typecheck

# Run tests (verbose)
node --experimental-vm-modules node_modules/.bin/jest --no-cache --verbose

@deepseek-ai/dsh-llm and @deepseek-ai/dsh-settings require incompatible @deepseek-ai/dsh-brand versions, so npm needs --legacy-peer-deps (or npm ci --legacy-peer-deps). This predates the workspace split and is unrelated to it.


Dependencies

Core (packages/coredsh-context-milvus-core):

  • @zilliz/milvus2-sdk-node — Milvus Node.js SDK
  • ignore — gitignore-style pattern matching
  • tree-sitter — AST parsing engine
  • tree-sitter-typescript — TypeScript/JSX grammar
  • tree-sitter-python — Python grammar
  • tree-sitter-java — Java grammar
  • tree-sitter-go — Go grammar
  • tree-sitter-rust — Rust grammar
  • tree-sitter-cpp — C++ grammar
  • tree-sitter-c-sharp — C# grammar
  • tree-sitter-scala — Scala grammar

DSH adapter (packages/dsh), all provided by the DSH runtime:

  • @deepseek-ai/cordis — DSH framework
  • @deepseek-ai/dsh-tools — DSH tool registration API
  • @deepseek-ai/schemastery — Config schema definition
  • @deepseek-ai/dsh-settings — settings panel (installSection)
  • @deepseek-ai/dsh-llm — agent access used by constraint re-injection

Codex adapter (packages/codex):

  • @modelcontextprotocol/sdk — MCP server + stdio transport
  • zod — MCP tool input schemas (kept out of core on purpose)

License

MIT

上一个 Prev dsh-models-radar 下一个 Next deskpet-bridge