Xidong-AI/dsh-rate-limiter

dsh-rate-limiter

English | 中文

A proactive rate limiter plugin for DeepSeek Harness (dsh): it controls the request rate per provider (token bucket) before model requests are issued, and queues the request with a delay instead of failing when the limit is exceeded — avoiding upstream 429s.

It complements the official dsh-llm-retry (exponential backoff after failure): rate limiting comes first (prevention), backoff comes last (safety net); the two do not interfere with each other.

Features

• Per-provider token bucket, enforced before the request is sent (proactive prevention)
• Over-limit requests are queued with a delay instead of rejected (no 429s, no lost requests)
• Unconfigured providers pass through untouched (zero intrusion)
• Queued waits honor the abort signal: stopping the user interrupts the wait immediately
• Hand-written reservation-based token bucket (concurrency-safe), zero third-party rate-limiting dependencies
• Mounts on agent/request, coexists naturally with dsh-llm-retry

Installation

Install from npm:

dsh plugin --profile web add @xidong-ai/dsh-rate-limiter

npm registry URLs are case-sensitive; use the lowercase package name.

Or install directly from GitHub:

dsh plugin --profile web add github:Xidong-AI/dsh-rate-limiter

For local development, add the checkout directly:

dsh plugin --profile web add .

After installing, dsh --profile web --dump-config should show the plugin entry:

- id: rate-limiter
  name: @xidong-ai/dsh-rate-limiter
  config:
    enabled: true
    providers: {}

Configuration

Configure the token bucket per provider in the profile's cordis.patch.yml (or this plugin's cordis.patch.yml):

- id: rate-limiter
  config:
    enabled: true
    providers:
      nvidia:
        rate: 0.5        # tokens/second (long-term average QPS)
        burst: 1         # bucket capacity (allowed burst requests)
      sensenova:
        rate: 0.02778
        burst: 1

• rate: refill rate (tokens/second), i.e. the long-term average request rate.
• burst: bucket capacity, the number of burst requests allowed.
• Providers not listed are not rate-limited; requests pass through untouched (zero intrusion).
• enabled: false disables the plugin entirely.