dsh-plan-checkup · Plan Checkup

English · 简体中文 · 繁體中文

A plugin for DeepSeek Harness (dsh). When the agent hands in a plan in plan mode, Plan Checkup reads it step by step before you approve it, and marks what it finds on the plan review card.

It only advises. It never blocks, rewrites or approves a plan, and nothing leaves your machine until you set up a judgment engine.

What you see

1. In dsh web, type /plan to turn on plan mode, then send your request.
2. When the agent hands in its plan, a Plan checkup badge appears in the review card's toolbar, next to the link that opens the full plan. Rule flags show at once; the badge shows progress (for example Checking 3/8) while the engine works through the steps, then the number of flags in each color, or No issues found.
3. Click the badge for the details: each flagged step, the flags with their probabilities, the commands a rule matched, and the notes about the whole plan. Each flag has 👍 / 👎 buttons; your votes are kept in a local file and can be used to recalibrate thresholds later.
4. To have the agent revise the plan, click Put in the message box, then Request changes on the card. The message box then holds a ready-made list of the problems, which you can edit and send. Copy feedback copies the same text.
5. Approve runs the plan as usual.

If the engine can't be reached, the badge says Rules only, the panel gives the reason, and the review goes on normally.

Requirements

• dsh 0.2.x. Tested on dsh 0.2.0-rc.2 (Windows 11, Node.js 24.21) with the web profile.
• Node.js 22.19 or later in the 22 line, or 24 and later (the same as dsh).
• For anything beyond the command rules, a judgment engine (next section).

Install

dsh plugin --profile web add dsh-plan-checkup

If you run dsh through npx, use npx @deepseek-ai/dsh plugin --profile web add dsh-plan-checkup. Restart dsh web afterwards. Until a judgment engine is configured, only the command rules run and the badge says Rules only.

Update: run dsh plugin --profile web update dsh-plan-checkup (through npx, npx @deepseek-ai/dsh plugin --profile web update dsh-plan-checkup), then restart dsh web.

• dsh installs plugins with pnpm, and pnpm picks a newly published version only once it is a day old. For the first day after a release, update stays on the version you have; to update right away, name the version, for example dsh plugin --profile web add dsh-plan-checkup@0.1.1.
• update stays within the minor version you have (0.1.x). To move to a new minor version such as 0.2.0, run dsh plugin --profile web add dsh-plan-checkup@latest; the one-day wait applies here too.
• The releases page lists each version and what changed.

To remove it, run dsh plugin --profile web remove dsh-plan-checkup and delete the plan-checkup entry from your profile's cordis.patch.yml if you added one.

Set up a judgment engine

Settings go in your profile's cordis.patch.yml (for the web profile, ~/.dsh/profiles/web/cordis.patch.yml). An entry with id: plan-checkup replaces the plugin's whole config; keys you leave out keep their defaults.

Option 1: an OpenAI-compatible endpoint (no Jev key needed)

Each question becomes a lettered multiple-choice question. The model outputs a single token, and the plugin reads the probability of each option letter from the logprobs; it never parses free text. The endpoint must support logprobs and top_logprobs, for example vLLM, SGLang, llama.cpp, or Ollama 0.12.11 and later.

A vLLM server on your LAN over plain http:

- id: plan-checkup
  config:
    engine: llm
    llm:
      endpoint: http://10.0.0.20:8000/v1/chat/completions
      model: Qwen3.8-27B
      allowEgress: true            # plan text is sent to this host
      allowHttpHosts: [10.0.0.20]  # allow plain http to this host
      extraBody:
        chat_template_kwargs:
          enable_thinking: false   # turn thinking off on reasoning models

Loopback endpoints (127.0.0.1, localhost) don't need allowEgress. If the endpoint needs a key, store it as a dsh credential or an environment variable and name it in llm.apiKeyEnv.

When thinking is left on, the first token is <think> instead of an option letter; the checkup falls back to the rules, and the panel says the engine's response was in the wrong format.

Before connecting dsh, you can check an endpoint from a clone of this repository:

npm run probe -- --endpoint http://10.0.0.20:8000/v1/chat/completions --model Qwen3.8-27B \
  --allow-egress --allow-http-host 10.0.0.20 --extra '{"chat_template_kwargs":{"enable_thinking":false}}'

It runs a built-in 7-step plan (or --plan <file>) and prints the probabilities and flags for each step.

Option 2: TypeSafe Jev, or a server with the same API

TypeSafe cloud: store your key as TYPESAFE_API_KEY (a dsh credential or an environment variable), then allow sending:

- id: plan-checkup
  config:
    jev:
      allowEgress: true

A Jev-compatible server such as Laya or openjev: point the endpoint at it. Loopback servers don't need allowEgress.

- id: plan-checkup
  config:
    jev:
      endpoint: http://127.0.0.1:8791/v1/systemone

In our evaluation, pretrained Laya scored close to chance on these questions; fine-tune it on your own data before relying on it. The default thresholds were calibrated on Qwen3.8-27B and have not been checked against Jev.

What leaves your machine

Settings

The full defaults are in cordis.patch.yml. The common keys:

Where data is stored

In $DSH_HOME/plan-checkup/ (by default ~/.dsh/plan-checkup/):

• results/<session>_<callId>.json: the result for each plan, so it is still there when you reopen the plan.
• ledger.jsonl: for each checkup, the engine, the probability of each answer, latency and token counts, the plan title, your decision (approve or request changes) and your 👍 / 👎 votes. It doesn't contain the step text.

The plugin never writes to the session log.

How well it works

We evaluated it on synthetic plans with deliberately planted problems (code and data in eval/): 210 plans to tune the questions and thresholds, then 90 held-out plans from 6 new scenarios. Precision / recall of the card's flags on the held-out set:

With Qwen3.8-27B on a vLLM server, a plan took 2.1 s at the median and 2.9 s at p95, with about 26 requests. These plans are cleaner than real ones and every one has planted problems, so read the numbers as an upper bound. Plans from a capable agent rarely contain these slips, so on real work expect No issues found most of the time.

Feedback

This is the first public version, and feedback decides what comes next. Open an issue with the feedback form. The Send feedback link at the bottom of the details panel opens the same form with the plugin version and the engine filled in; nothing is sent until you submit it yourself. Useful things to include:

• the step and the flag that was wrong, or the problem that wasn't flagged;
• your dsh version and judgment engine;
• optionally, an excerpt of the plan (remove anything private) and the matching lines from ledger.jsonl.

Known limitations

• It looks for specific slips: dangerous commands, unrequested work, vague steps, missing tests. It does not judge whether the approach is right or whether the facts a plan relies on are true.
• The questions are written for coding plans. For research or writing plans, most of them don't apply.
• Steps are split from the plan's top-level list or headings; nested items fold into their step, a step longer than 1,200 characters is cut, and sections that aren't steps (goals, background) are not checked.
• If the judgment engine is the same model that wrote the plan, it shares that model's blind spots.
• Thresholds were calibrated on synthetic data with Qwen3.8-27B. Other engines and models need their own check; the probabilities and votes in ledger.jsonl can be used for that.
• May miss part of the request is the weakest check; treat it as a soft reminder.
• Plans from sub-agents never get a review card (sub-agents can't ask the user), so they are not checked.
• Request changes cancels the review, so the feedback is sent by you, not by the plugin.
• It is an advisory tool, not a security boundary: dsh's sandbox and approvals work as before.

Development

npm test                  # unit tests, no network or keys needed
npm run dev:llm           # mock planner that always hands in a 7-step plan (port 18765)
npm run dev:jev           # mock Jev that validates requests and returns probabilities (port 18766)
npm run dev:logprobs      # mock logprob endpoint for the llm engine (port 18767)
npm run probe -- …        # test a judgment engine with one plan (see above)

dev/overlay.yml (mock Jev) and dev/overlay-llm.yml (mock logprob endpoint) start a separate test setup where every service runs on loopback and no key is needed:

DSH_HOME=.m0/home M0_MOCK_API_KEY=mock dsh web --patch dev/overlay-llm.yml --port 3190 --no-open

node dev/screenshot.mjs <dsh-url> <lang> <request> <out.png> [--feedback <out.png>] retakes the screenshots against a running dsh web, using Edge or Chrome and no extra packages; MOCK_PLAN_LANG=en or zh-CN makes the mock planner hand in its plan in that language.

The mocks can simulate errors: MOCK_JEV_MODE=401|402|429|529|slow and MOCK_LLM_MODE=401|429|503|slow|nologprobs|think. The browser half, lib/client.js, is written by hand in the dsh client-module format, so there is no build step. Notes on the dsh hooks it relies on are in m0/README.md.

License

MIT