Ottohere-Mourn/TeachReplay
Teach once, replay anywhere — harness-agnostic Teach-by-Demonstration engine (Record → Compile → Replay → Verify) with OpenMausBot and DeepSeek Harness integrations.
安装Install
dsh plugin --profile web add github:Ottohere-Mourn/TeachReplay把 Ottohere-Mourn/TeachReplay 加入你的 DSH 配置（web profile）即可启用。
READMEREADME
TeachReplay
Teach once, replay anywhere.
TeachReplay is a harness-agnostic Teach-by-Demonstration engine: show a workflow once — clicks, typed values, shell commands — and it becomes a reusable, parameterized skill that can be replayed later with different inputs and an explicit success/failure verdict on real computers.
Teach → Compile → change parameters → Replay → Success · 26 s · watch in HD
What is TeachReplay?
Teach → Record → Compile → Replay → Verify
- Teach — start a recording and demonstrate the task once on a computer.
- Record — the recorder watches the computer's semantic state (URLs, visible text, element roles and values) and diffs it into a versioned trajectory. Password fields stay masked; every value passes secret redaction.
- Compile — a deterministic compiler turns the trajectory into a parameterized skill: your demonstrated values become inputs with your values as defaults. GUI steps are grounded semantically (role + name, never raw coordinates); shell commands become real CLI steps with their recorded exit codes.
- Replay — the engine re-snapshots per step, matches targets semantically, acts, and verifies each step's effect. One bounded retry, an optional model-assisted recovery hook, then explicit failure.
- Verify — success requires the recorded condition (URL or confirmation text) to hold. "The engine stopped" is never success.
Package overview
| Package | Purpose |
|---|---|
@teachreplay/core |
trajectory schema · recorder · parameterized skill compiler · parameter substitution · replay engine · verifier · file stores · createTeachRuntime orchestration |
@teachreplay/remote |
generic SSH Linux computer backend (Xvfb + Chrome DevTools) with GUI + shell channels — not provider-specific |
@teachreplay/mock |
deterministic in-memory demo computer for tests and local demos |
@teachreplay/adapter-dsh |
DeepSeek Harness plugin registering teach_* tools |
Architecture
The core owns all Teach-by-Demonstration logic and depends on nothing but Node.js — an enforced invariant (independence test). Adapters contribute only backends, stores, and event sinks:
| Integration | Status |
|---|---|
Standalone (createTeachRuntime + file stores + mock/remote backends) |
✅ included — see the standalone demo |
| OpenMausBot (TeachReplay-OpenMausBot) | ✅ thin adapter — the v0.1 integration rebuilt on the core |
| DeepSeek Harness (deepseek-ai/deepseek-harness) | ✅ adapter targets the real DSH plugin API — see below |
TeachReplay for DeepSeek Harness
@teachreplay/adapter-dsh adds teach-by-demonstration tools to
DeepSeek Harness agents:
teach_start / teach_stop / teach_compile / teach_replay /
teach_shell — the same Record → Compile → Replay → Verify engine, no core
logic duplicated.
- Targets the real DSH plugin API:
@deepseek-ai/cordis/@deepseek-ai/dsh-toolsarepeerDependencies, pinned asdevDependencies(4.0.1/0.1.1-rc.2, the exact pair DSH resolves atdsh-v0.1.1-rc.2) so this repo's own build compiles the plugin against the realdefineTool/Plugin.Objecttypes — no local shims. A prior DSH-native rewrite verified inside a full DSH workspace checkout (see the adapter README) is what this version is based on - Installation:
pnpm add @teachreplay/adapter-dsh— DSH's sub-packages are published to npm under thenextdist-tag while DSH is in developer preview; see the adapter README (⚠️ DSH is in developer preview with compatibility-breaking changes; re-verify per release) - Demo:
examples/dsh-demorunsteach_start → demonstrate → teach_stop → teach_compile → change parameters → teach_replay → successlocally — against the mock computer throughDshTeachSessiondirectly, not through DSH's real tool runtime
(DSH does not accept external pull requests while in developer preview —
see its own CONTRIBUTING.md — so this ships as an independent package
rather than an upstream PR.)
Quick start
git clone https://github.com/Ottohere-Mourn/TeachReplay.git
cd TeachReplay
pnpm install
pnpm build
pnpm demo # standalone: record → compile → change parameters → replay → verify
pnpm test # 50 tests, including the zero-harness-dependency invariant
pnpm typecheck
The demo runs entirely on the built-in demo computer — no credentials, no network, no harness.
Real evaluation
scripts/mini-benchmark.mjs runs 8 tasks / 14 replays against a real remote Linux computer
(GUI + CLI, SSH env-configured). Every success is checked against ground truth on the machine,
not just the replay's own verdict. Latest real run (results in
scripts/teach-benchmark-results.json):
| Metric | Result |
|---|---|
| Verdicts correct | 14/14 |
| Normal replay success | 12/12 |
| Changed-parameter replays | 6/6 |
| GUI + CLI workflow | success, processed output matches |
| Severe UI drift / missing element | detected as explicit failures (2/2) |
| Ground-truth mismatches | 0 |
An honest small-scale sanity check — not a claim of state of the art.
Limitations
- Recording observes state transitions (polled, ~500 ms), not input events — very fast actions can coalesce. Slow, deliberate demonstration is the intended input.
- Clicks are inferred from state changes; ambiguous transitions record no click rather than a wrong one.
- Model-assisted recovery is an optional hook (
ModelBackend) — adapters wire the agent. - Replay speed is SSH-round-trip bound (~4 s/step on the evaluation box).
- DSH is in developer preview with compatibility-breaking changes — re-verify per release.
Attribution and license
Apache-2.0 (see LICENSE). TeachReplay was extracted from TeachReplay v0.1, developed inside OpenMausBot (Apache-2.0) — see NOTICE for the derived-work attributions, including the CDP helper derived from OpenMausBot's computer-use tooling. Record-to-skill is not a new idea — TeachReplay's value is the working, verified combination of demonstration → structured trajectory → parameterized skill → executable replay → explicit verification across GUI + CLI.
相关推荐Related
- deepseek-ai/deepseek-harness DeepSeek Harness: Everything is a Plugin.
- deepseek-ai/deepseek-harness DeepSeek Harness (dsh) — 由 DeepSeek AI 开发的开源 agent harness（智能体框架）。一切皆插件：核心 + 插件 + 技能 = 可热插拔的 AI 编程工作台。DeepSeek Harness: …
-
ruvnet/ruflo
🌊 The original agent meta-harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build c…
-
amruthpillai/reactive-resume
A one-of-a-kind resume builder that keeps your privacy in mind. Completely secure, customizable, portable, open-source a…
-
volcengine/OpenViking
Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
-
Molunerfinn/PicGo
:rocket: The Ultimate Image Uploader for Efficient Creators. Supports Obsidian, Typora, VS Code etc. and 60+ image hosti…
-
titanwings/colleague-skill
将冰冷的离别化为温暖的 Skill，欢迎加入数字生命1.0！Transforming cold farewells into warm skills? It's giving rebirth era. Welcome to Digital …
-
nocobase/nocobase
NocoBase is an open-source AI + no-code platform for building business systems fast. Instead of generating everything fr…