dsh-llm-rate-limiter

English | 中文

Per-model LLM call rate limiter for DeepSeek Harness with queue/reject support and interactive GUI configuration.

Features

• Per-model rate limiting — independent concurrency, RPM, and burst limits for each provider/model
• Two algorithms — Token Bucket (allows bursts) or Sliding Window (smooth, strict RPM)
• Queue mode — throttled requests wait in queue and are released when a slot opens
• Reject mode — throttled requests fail immediately (integrates with dsh-llm-retry for auto-backoff)
• Interactive GUI — collapsible card in DSH Settings → Plugins → Configurable
• Live status panel — real-time counters, per-model progress bars and an event log (v0.2.0)
• Hot-reload — settings changes take effect immediately, no restart needed
• Every request checked — intercepts llm/stream waterfall, covering every LLM call in every agent turn

Status Panel (v0.2.0)

Expanding the card shows a live panel at the top of its body:

📊 实时状态                          ● 实时   [清零]
 请求 42 · 通过 38 · 拒绝 2 · 超时 1 · 中止 1 · 平均等待 214ms
 deepseek/deepseek-chat   [令牌桶]  ▓▓▓▓▓▓▓░░░ 7.5/10   并发 2/5   排队 1
 openai/gpt-4o            [滑动窗口] ▓▓▓▓▓▓▓▓▓▓ 3/3 rpm  并发 1/5
 12:00:03  timeout   openai/gpt-4o              等待 1m
 12:00:01  rejected  openai/gpt-4o
 12:00:00  granted   deepseek/deepseek-chat     等待 4.2s

Carrier fallback (DSH 0.1.5-rc.3 onward)

DSH 0.1.5-rc.3 changed @deepseek-ai/dsh-client-connection's own inject from ["webServer", "credentials"] to ["credentials"], but its HostConnectionService.register() still dereferences owner.webServer. Cordis rebinds a cross-fiber service's ctx to the reader's fiber, so connection.rpc.handle() throws: still present in 0.1.7-rc.1 — see below.

cannot get property "webServer" without inject

The plugin now detects that and mounts the same channel itself:

Both carriers speak the identical wire protocol, so the browser half is unchanged — the panel cannot tell which one is live. If connection.requestRejection() is unavailable, the plugin refuses to mount rather than publishing an unauthenticated route.

Installation

Option 1: npm (recommended)

dsh plugin add <your-profile> @leaf233/dsh-llm-rate-limiter
# or, inside the profile directory:
pnpm add @leaf233/dsh-llm-rate-limiter

Option 2: local path (development)

dsh plugin add <your-profile> ./path/to/dsh-llm-rate-limiter
# or
dsh plugin add ./path/to/dsh-llm-rate-limiter   # default profile

The plugin must be added as a dependency in the profile's package.json. The bundle entry (cordis.patch.yml) is auto-detected by reconcilePlugins.

0.3.0 requires DSH 0.1.7-rc.1 or newer. Pin @leaf233/dsh-llm-rate-limiter@0.2.x for DSH 0.1.5. See the compatibility table.

Option 3: from GitHub

dsh plugin add <your-profile> github:Leafyezi233/dsh-llm-rate-limiter

⚠️ Important: Git-hosted plugins are blocked by pnpm's allowBuilds restriction on first install. If the install fails, check the error message for the exact key pnpm suggests, then add it to your profile's pnpm-workspace.yaml:

pnpm:
  allowBuilds:
    - '@leaf233/dsh-llm-rate-limiter'

Then re-run the install command.

Configuration

Via GUI

1. Open DSH Web UI (dsh web)
2. Go to Settings → Plugins and open the Official group
3. Find the LLM 调用限速 card (data-plugin-item="llm-rate-limiter") and open it
4. Configure defaults, per-model overrides, and throttle behavior — changes save through the page's own form, so the live status panel is right there

The card lives in the Official group because DSH lists every plugins.item registrant there; the plugin's order is 900, so it follows the five official settings pages.

Via file

Changes persist into the profile's Cordis patch, in the entry's own config (0.1.7 reads configuration from the entry's exported Config; a legacy settings.yaml is imported by DSH once at first start and then renamed). The shape is:

llm-rate-limiter:
  enabled: true
  strategy: token-bucket       # "token-bucket" | "sliding-window"
  defaults:
    maxConcurrent: 5
    maxRpm: 60
    burstSize: 10               # token-bucket only
    refillRate: 1               # token-bucket only (tokens/sec)
  models:
    "deepseek/deepseek-chat":
      maxConcurrent: 8
      maxRpm: 120
    "openai/gpt-4o":
      maxConcurrent: 2
      maxRpm: 10
      burstSize: 3
    "anthropic/claude-3-5-sonnet":
      enabled: false            # skip rate limiting for this model
  onThrottled: queue            # "queue" | "reject"
  maxQueueWaitMs: 60000

Settings Reference

Note: When a model overrides maxRpm without explicitly setting refillRate, the refill rate is automatically derived as maxRpm / 60 (tokens per second). This ensures "set maxRpm=3" actually limits to 3 requests per minute.

Algorithm Comparison

How It Works

Agent Turn
  → LLM Call (e.g. deepseek/deepseek-chat)
    → ctx.on("llm/stream") interceptor
      → Resolve rate limiter for this provider/model
      → Token bucket: has tokens + concurrency room?
      → If YES: consume token, acquire slot, forward to API
      → If NO (reject mode): return RATE_LIMIT error immediately
      → If NO (queue mode): park in waiters[], wait for token refill
    → Request completes → release slot → drain waiting requests
  → dsh-llm-retry catches RATE_LIMIT → exponential backoff → retry

Development

# Clone
git clone https://github.com/Leafyezi233/dsh-llm-rate-limiter.git
cd dsh-llm-rate-limiter

# Install deps
pnpm install

# Run the full suite (375 assertions across 6 files)
pnpm test

# Individual suites
node test-strategies.mjs       # 19 — rate-limit algorithms
node test-status-rpc.mjs       # 61 — channel handler + counters
node test-client-bundle.mjs    # 106 — real client bundle on a miniature React runtime
node test-host-integration.mjs # 80 — real apply(ctx, config) wiring + volatile refs
node test-status-route.mjs     # 73 — self-registered route + the 0.1.5 regression
node test-config-schema.mjs    # 36 — Config volatile contract the settings page needs

# Live HTTP proof against a real DSH install (real socket, real webserver)
pnpm run verify:live           # 19 — exits 2 (skipped) when DSH is absent

# Run E2E rate-limit test
node test-3rpm.mjs

# Install into a DSH profile for testing
dsh plugin add <your-profile> .

The plugin uses a live symlink when installed via link: — edits to lib/ take effect on browser hard-refresh (Ctrl+Shift+R) without reinstalling.

Compatibility

Breaking change in 0.3.0

DSH 0.1.7 removed the two client services this plugin used for its settings UI:

0.3.0 only supports DSH 0.1.7-rc.1+. Use 0.2.x for 0.1.5.

The four breakpoints this migration rests on are documented inline in lib/index.js and lib/types/config.js, and the reasoning is recorded in CHANGELOG.md under [0.3.0].

License

MIT