dsh-plugin-turn-memory

The active implementation is being rebuilt from the researched agreements in design.md. The previous implementation is retained under obsolete/ for investigation only.

The plugin now has two coordinated layers:

• completed-turn compression uses a parent fork and lands an N→M transcript rewrite;
• optional long-thought preprocessing starts asynchronous small-model recall hints as soon as each complete long reasoning block arrives, then gives successful hints to the normal main-model turn worker without replacing its raw parent context;
• long root turns receive a model-visible handoff reminder and can continue in a fresh turn after their completed-turn compression lands;
• optional root-session compaction provides ctx.compaction, processes a selected completed-history range through sequential main-model workers, and commits one standard durable checkpoint only after the host-owned working draft is complete.
• compressed image blocks can become durable <memory-image ref="..." /> text references; read_memory_image reloads the original pixels on demand instead of placing them in every later model request.
• read_session_history can recover one completed turn's original append-only user, assistant, reasoning, and tool-result nodes when compressed memory omitted a costly detail.

The master plugin remains disabled by default. Completed-turn compression is on when the plugin is enabled unless turnCompression: false. Session compaction is separately opt-in through sessionCompaction.enabled: true.

Observed completed user-conversation turns receive a durable pending marker and are recovered after an agent cold resume until their final landing exists. The plugin does not backfill marker-free history merely because an old session is opened after installation. UI-created lineage forks are user conversations and participate normally; internal subagents do not. A legacy turn-memory no-op whose current surface still contains raw reasoning is also migrated once.

Install

Add the package to a DSH profile from GitHub:

dsh plugin --profile web add github:vilicvane/dsh-plugin-turn-memory

For local development, link the working tree instead:

dsh plugin --profile web add link:/path/to/dsh-plugin-turn-memory

Then register the plugin in the profile's cordis.patch.yml. For the web profile this is normally $DSH_HOME/profiles/web/cordis.patch.yml. Restart dsh web after installing the package or changing the profile configuration.

Configuration

Completed-turn compression only

This mode leaves the profile's existing whole-session compaction backend untouched:

- insert:
    - id: turn-memory
      name: dsh-plugin-turn-memory
      config:
        enabled: true
        turnCompression: true
        turnContinuation:
          enabled: true
          reminderIntervalNodes: 30
        thoughtHints:
          enabled: true
          provider: ollama
          model: your-small-model-id
          minimumChars: 16000
          maxTokens: 1024
          timeoutMs: 120000
        sessionHistory:
          enabled: true
          maxReadChars: 160000
          catalogTurns: 40

Completed-turn and whole-session compaction

Only one plugin may provide ctx.compaction. Disable compaction-basic and remove or disable any other session-compaction provider before enabling this layer. Re-enable command-compact when the Web profile has disabled it; its /compact command will call turn-memory's engine.

- id: compaction-basic
  disabled: true

- id: command-compact
  disabled: false

- insert:
    - id: turn-memory
      name: dsh-plugin-turn-memory
      config:
        enabled: true
        turnCompression: true
        sessionCompaction:
          enabled: true
          auto: true
          thresholdRatio: 0.8
          retainRatio: 0.16
          segmentTokens: 32000
          workerAttempts: 2

Set turnCompression: false in the same configuration to use only the whole-session layer. Set sessionCompaction.auto: false to disable pressure triggering while keeping manual /compact available.

The main tuning options are:

Each session segment first uses the parent's provider/model through fork. If it stops without an authoritative finish, continuation workers use the same main model through fresh spawn, with the assigned canonical source and accepted host-owned revision embedded in the prompt. A provider error, timeout, max-token stop, or missing finish consumes the budget only when that worker produced no accepted revision. Turn, session, and thought-hint prompt files are read for each new request, so prompt edits do not require a Web restart.

When thought hints are enabled, each root-session assistant/chunk reasoning block-end at or above minimumChars immediately starts an independent ctx.llm.stream() call. The model sees only the fixed prompts/thought-hints.md system prompt followed by one user message containing the raw reasoning block. Several blocks may run concurrently. At durable turn/end, compression waits for the calls that belong to assistant messages actually present on that turn's current surface; a cold resume regenerates missing calls from those durable reasoning blocks. Failed or empty hints are omitted and never block compression because the parent fork still inherits the complete original reasoning. Hints are advisory recall indexes only, are never persisted as transcript nodes, and every original assistant node carrying reasoning must still be rewritten before landing.

At each multiple of turnContinuation.reminderIntervalNodes reached by one root turn, its dynamic runtime context gives the model a short direct action notice to stop, prepare a handoff, and call continue_after_turn_compression. The notice and automatic follow-up use ordinary text without XML wrapper tags; structured tags remain only where compression protocols need explicit data boundaries or durable references. The notice remains plugin context rather than a fabricated durable assistant message. The model may keep the current turn only when the whole task will finish in the next few actions; an in-progress atomic mutation is finished before handing off. The counter includes append-origin model-visible messages in the open turn and excludes replaceable runtime snapshots. The notice includes the current node count, estimated token size of the entire canonical context, and next reminder milestone. Durable runtime snapshots remember which milestones were already shown, including across a restart, and the recovery parser still recognizes older tagged snapshots. The successful durable native tool pair or Code Mode sub-dispatch records completed work, current state, unresolved dependencies, and exact next work, then concludes the turn. The plugin first lands and flushes normal completed-turn compression, then submits a direct automatic plugin-sourced follow-up as a separate ordinary turn. If compression has not succeeded, continuation does not start; cold-resume recovery can finish the missing compression and dispatch the still-pending request without duplicating an already inserted follow-up.

The plugin exposes read_memory_image to the main conversation and both compression-worker types when the profile has an attachment store. Compression retains the content-addressed attachment id rather than a local path, and the tool accepts only ids that really occur in the current session or its active root parent. The original pixels are loaded only for the tool request and only when the active model declares image input.

The main tool catalog also exposes read_session_history by default. With no turn it lists recent completed turns and their visible/reasoning sizes; with turn=<n> it reads that turn's original append-only message nodes, including reasoning shadowed by later surface replacement. Reads are limited to the calling agent's own session and are paged by character offset. This is an on-demand recovery path: it does not restore raw history to the standing surface or make it part of every request.

Provider-confirmed context overflow has a fixed one-shot recovery policy: compact once and replay the failed main-model request only after a durable surface replacement was committed. If that replay also overflows, the provider error is preserved rather than starting another compaction loop. This is independent of sessionCompaction.compactionRetries, which controls additional successful pressure compactions when one checkpoint still leaves the session above the configured threshold.

Verification workflow

Run the fast deterministic checks after every edit:

pnpm check

Run the real end-to-end smoke when the change touches plugin wiring, prompts, tools, surface landing, or persistence:

pnpm e2e

Run the dedicated session-compaction E2E after changing pressure selection, segmentation, session tools, worker orchestration, checkpoint rendering, or the compaction transaction:

pnpm e2e:session

It disables the stock backend in the test overlay, creates three completed turns, injects the same post-turn assistant replacement produced by turn-memory, and first proves that upstream token-meter rejects that lifecycle shape. It then forces the first two turns into separate segments and calls the custom engine manually through its canonical-surface pricing fallback. With a one-failure budget, the first segment's fork is deliberately stopped after an accepted mutation; the test requires a fresh-spawn continuation to finish it, then requires the next segment to begin with a fork. Finally it verifies standard event adjacency, checkpoint provenance, the untouched retained tail, persistence, and a cold resume of the exact session.

When production prompt behavior changes, also run the qualitative steer regression:

pnpm e2e:prompt

Repeat the same independent real-model scenario to measure observed stability:

TURN_MEMORY_PROMPT_EVAL_RUNS=3 pnpm e2e:prompt:repeat

The repeat wrapper retains every session and surface path, then reports the pass rate and a Wilson 95% confidence interval under .tmp/.

Run the broader production-prompt matrix to cover typo consolidation, a genuine evidence-driven objective change, and implementation-detail retention:

TURN_MEMORY_PROMPT_EVAL_RUNS=2 pnpm e2e:prompt:matrix

Every scenario performs real parent work, automatic turn compression, a cold resume, full surface replay, and sessionQuery comparison. The matrix accepts variable valid node counts but requires causal user/assistant boundaries and scenario-specific continuation details. It fingerprints the production prompt, renderer, editor, and plugin entry before and after each session so concurrent edits fail the run instead of mixing prompt versions. Use TURN_MEMORY_PROMPT_EVAL_SCENARIOS=goal_change to isolate one scenario.

This creates a real multi-step turn whose first /repo attempt fails and whose later user steer says playground. Unlike the deterministic smoke protocol, it uses the production compression prompt and requires the two user messages to become one corrected intent whose provenance includes both sources. Assistant work may remain in more than one semantic node, but together it must retain both the failed detour and the final 6-type/9-device conclusion. It checks live folding, persistence, cold loading, and session-query projection, then writes .tmp/prompt-eval-surface-<session-id>.json for inspection.

The e2e workflow uses the dedicated ~/.dsh/profiles/test-turn-memory profile and the configured real model. It creates a fresh parent session, forces a multi-node tool turn, intentionally leaves its live turn/end uncompressed, flushes persistence, destroys the agent, and cold-resumes the same session. The recovery scan must then start the missed compression. The runner interrupts its first worker after an accepted n* -> r* mutation and verifies that a fresh-spawn recovery worker continues those r* nodes through the final rewrite. It checks live foldSurface() and deriveMessages(), flushes, cold-resumes a second time, and verifies that durable recovery detection neither duplicates nor changes the replacement identities. Success ends with E2E_RESULT=PASS, prints the retained test session id, and writes the exact sessionQuery.readSurface() snapshot to .tmp/e2e-surface-<session-id>.json. The E2E_SURFACE_PATH output contains its absolute path for direct inspection.

The runner requests shutdown through DSH's bounded appExit service rather than calling process.exit(), so persistence and plugin teardown are drained.

Every enabled turn-memory instance also flushes and writes its complete folded surface after a successful replacement. By default the latest snapshot for each session is written to .tmp/e2e-surface-<session-id>.json beside this plugin; surfaceDumpDir changes the destination but leaving it unset does not disable the dump.