d3vmeh/dsh-llm-gate

dsh-llm-gate is a native cordis plugin built specifically for DeepSeek Harness (DSH) that solves the problem of provider requests being silently dropped or timing out when a backend can only serve a limited number of concurrent calls, such as a local llama.cpp server started with --parallel 1. Instead of letting the request sit at the HTTP layer until the Node default of 300 seconds elapses and a terminated error appears, the plugin parks surplus requests inside a FIFO queue before any network call is dispatched, so no timeout clock is running while a request waits. Installation is performed with dsh plugin --profile web add dsh-llm-gate, after which the operator enables per-provider gating inside ~/.dsh/profiles/web/cordis.patch.yml by listing route names from llm-pi-ai.providers (or another adapter) together with maxConcurrent, maxQueued, and queueTimeoutMs; dsh web is then restarted and the composed configuration can be inspected with dsh --profile web --dump-config.

The intended audience is anyone running DSH against a constrained local model where subagents, the main agent, compaction, and session-title generation can collide on the same provider in the same turn. The plugin attaches to the llm/stream waterfall so every model request the host issues goes through it, including agent calls, subagent calls, compaction passes, and title generation. When a request must wait, a single line is printed to the dsh terminal showing the provider, the session identifier, the queue depth, and either a queued or dispatched event with the wait time in milliseconds; an auxiliary purpose=compaction or purpose=session-title tag is added so the operator can tell those requests apart. Requests that obtain a slot immediately emit no log line at all.

Dependencies are minimal: the host must have the llm service running and an adapter such as llm-pi-ai already configured with named providers, because the gate only intercepts the provider keys explicitly listed in the patch. Operators should set maxConcurrent to match llama.cpp's --parallel slot count and, when seeking throughput rather than strict serialization, raise it together with --parallel 2 --kv-unified. Queue full and queue timeout failures end the turn immediately and are not retried by dsh-llm-retry, while waiting time inside the gate is not billed against streamIdleTimeoutMs because the adapter is not invoked until a slot is acquired; that timer must still be large enough for prompt processing. Cancellation must use the request's abort signal, otherwise a dropped stream stays queued until a slot frees, at which point it dispatches and is closed right away. The project is released under the MIT license.