dsh-auto-evolve

A self-evolving plugin for DeepSeek Harness (dsh). It observes how the agent runs, proposes improvements to its own assets via the LLM, validates each proposal inside a sandboxed trial agent, and applies only verified mutations — with a versioned ledger and automatic rollback on regression.

中文

The idea. Instead of shipping a static skill set, the plugin owns a small genome of evolvable assets (skills, tool-result post-processors, prompt sections, guard policies). Every runtime signal (tool failures, repeated calls, request errors) is collected; when a threshold crosses, a proposal cycle drafts a mutation, a sandboxed agent replays the failing episode with and without the change, and the change is applied only if the trial shows measurable improvement. Every mutation is versioned in a durable ledger and can be rolled back.

How it works

Observe ──▶ Propose ──▶ Validate ──▶ Apply ──▶ (observe again, for regressions)
  │             │            │            │
  tools/result  ctx.llm      sandboxed    ctx.skills.register + ledger
  request-error (stream)     sub-agent    disposer kept for rollback

The evolution loop

Safety boundaries

• The mutation vocabulary is code, the content is model-generated. The LLM never invents asset kinds or operators; it only fills in payloads that pass the closed zod schema.
• Validation is by execution, not by self-claim. A proposal is applied only when a sandboxed trial beats the baseline on observable metrics.
• Cost is capped per cycle and per day. The budget gate (maxCostPerCycle / dailyBudget) bounds the proposal call and every trial replay in auto-apply; an exhausted cap pauses the loop instead of silently burning tokens.
• Rollback is first-class. Every applied mutation keeps its disposer and its parent content; regression reverts the exact previous state.
• Observe-only is the default. In observe mode the plugin never proposes — it just collects signals and logs triggers.

Installation

The plugin is a dsh bundle: install it with the official CLI in one command — no manual cordis.yml editing required.

# Install into the web profile (the default UI profile)
dsh plugin --profile web add dsh-auto-evolve

# Or into the TUI profile
dsh plugin --profile tui add dsh-auto-evolve

dsh plugin add initializes the profile if needed, installs the package, and automatically adds dsh-auto-evolve to the profile's bundle stack (dsh.profile.bundles). The bundled cordis.patch.yml registers the plugin row with the defaults below; restart dsh and the plugin is live.

Local development / source build:

git clone https://github.com/lispking/dsh-auto-evolve.git
cd dsh-auto-evolve
pnpm install
pnpm build
# Install your local checkout into a profile
dsh plugin --profile web add /absolute/path/to/dsh-auto-evolve

Custom configuration

The bundle applies a default config; to change it, override the row in your own profile patch (applied after every bundle layer):

# ~/.dsh/profiles/web/cordis.patch.yml
- id: self-evolve
  config:
    mode: auto-apply        # observe | propose | auto-apply
    observation:
      toolFailureThreshold: 3
      repeatThreshold: 3
      requestErrorThreshold: 3
      windowMs: 300000
    proposal:
      maxProposalsPerTrigger: 1
      maxEpisodesPerProposal: 5
      maxPromptChars: 24000
      maxTokens: 2000
    budget:
      maxCostPerCycle: 0       # 0 disables the per-cycle cap
      dailyBudget: 0           # 0 disables the daily cap
    validation:
      maxTrialMs: 30000
      maxToolCalls: 20
      maxTrialSteps: 12
      maxTrialTokens: 8000
    evolution:
      stallThreshold: 3
      stallPauseMs: 1800000
      cooldownMs: 600000

Note: a patch replaces the row's whole config rather than merging into it, so specify every field you want to keep. The plugin's peer services (storage, LLM, tools, skills) come from the dsh-base/web bundles — no extra setup.

Modes

Operator tools

The plugin registers a small set of agent-callable tools (dsh exposes no chat-command framework; in an agent harness the operator surface is tools). Ask the agent to inspect or drive the loop — no manual API calls needed:

Availability by mode: all five tools are registered in every mode; evolve_apply / evolve_rollback need the applier (mounted in propose and auto-apply), and evolve_cycle needs an LLM provider and rejects in observe mode.

Configuration

Programmatic API

import { SelfEvolveStore, SelfEvolveApplier, runProposalCycle, validateMutations } from 'dsh-auto-evolve'

// Run one proposal cycle (persists candidate assets).
const materialized = await runProposalCycle(ctx, store, { provider, model, maxTokens: 2000 })

// Validate a candidate: baseline vs trial replay, returns the verdict.
const { baseline, trial, comparison } = await validateMutations(ctx, {
  provider,
  model,
  episode: 'replay of the failing scenario',
  mutations: [candidateAsset],
  bounds: { maxTrialMs: 30_000, maxToolCalls: 20 },
})

// Apply a validated candidate (registers the skill live) or roll it back.
await applier.applyCandidate(candidate.id, trialId, 'validated')
await applier.rollback(candidate.id, 'regression observed')

Development

pnpm build   # tsc + tsdown → lib/
pnpm test    # vitest (unit + integration over a memory storage backend)

The test suite covers the pure decision logic (metrics comparison, mutation schema, thresholds) and the full wiring (durable store, observation collector, proposal cycle with a scripted LLM adapter, apply/rollback with the real skill registry).

License

MIT