dsh-kb-manager

English · 简体中文

Import once. Retrieve with confidence.

dsh-kb-manager is a local knowledge-base lifecycle plugin for DeepSeek Harness (dsh): multi-format import → CJK-aware smart chunking → SQLite-native vector + full-text indexes → hybrid search (vector KNN + BM25/FTS5 → RRF → optional rerank) → citation tracing — plus soft-delete + physical compaction, an async job system, import-path whitelisting with an SSRF guard, and per-KB audit.

Ask in natural language. The plugin exposes 22 Agent tools (async jobs + sync ops incl. rename_kb / compact_index / cancel_job / list_snapshots / configure) and an optional Web panel (KB list, create, detail, in-KB hybrid search, import with live job progress, rebuild, two-step delete, snapshot history) — no separate RAG server required.

Design spec: DESIGN.md (v2.1).

Why dsh-kb-manager?

Architecture

flowchart LR
  A[Documents / URLs] --> B[Parse]
  B --> C[Chunk<br/>Intl.Segmenter CJK]
  C --> D[Embed<br/>bge-small-zh / mock]
  D --> E[Vector index<br/>sqlite-vec KNN]
  C --> F[FTS5 BM25]
  E --> G[Hybrid search]
  F --> G
  G --> H[RRF]
  H --> I[Optional rerank<br/>applied / fallback_rrf / disabled]
  I --> J[Citations + untrusted flag]
  E --> K[Soft-delete / compact_index]
  C --> K
  G --> L[Async job queue + audit]

Install

[!NOTE] Requires an existing DeepSeek Harness installation.

From GitHub

dsh plugin add github:xiaoshi7915/dsh-kb-manager

(Once listed in the awesome-dsh-plugin list, it can also be installed one-click from the dsh-market page.)

Build from source

git clone https://github.com/xiaoshi7915/dsh-kb-manager.git
cd dsh-kb-manager
npm install
npm run build
dsh plugin --profile web add .

Validate the composed profile, restart DSH, and refresh the Web UI:

dsh --profile web --dump-config
dsh web

Then try:

Create a knowledge base named "project-docs", import this PDF, and search for how authentication works. Show me the source chunk.

How it works

1. Create a KB (create_kb) with an embedding identity (default bge-small-zh-v1.5 / onnx / 512-dim; mock provider for dev/test).
2. Import a whitelisted local file or http(s) URL (import_document) → an async job is queued: parse → chunk → embed → index; poll with get_job.
3. Search with search_kb: sqlite-vec KNN + FTS5 BM25 → RRF → optional rerank; rerank status is explicit (applied | fallback_rrf | disabled).
4. Trace hits with get_chunk (surrounding context + provenance metadata; content flagged untrusted: true).
5. Delete with confidence: delete_document soft-deletes instantly (re-import of an unchanged source restores it); delete_kb requires a two-step confirm_token; rebuild_index physically compacts.
6. Data lives under storage_path (default ~/.dsh/kb-manager/). The Web panel reads the same service surface: KB list / create / detail, in-KB hybrid search, import with live job progress, rebuild, and two-step delete.

Agent tools

Every tool returns the unified envelope { ok, data | error: { code, message, hint } }; error.hint is an actionable next step. Domain errors are returned (not thrown). Search results are always flagged untrusted: true.

When the Agent uses it

1. "Put this PDF / doc / page into the knowledge base" → import_document + get_job
2. "Search the KB for X" → search_kb
3. "Where did this citation come from?" → get_chunk
4. "Retrieval feels wrong — what's the rerank state?" → search_kb (reads the explicit rerank status)
5. "Create a KB for this project" → create_kb
6. "Remove a bad document" → delete_document (soft; re-import restores)
7. "Delete this whole KB" → delete_kb (two-step: preview impact, then confirm)
8. "How big is this KB?" → list_kbs / get_kb_stats
9. "Re-index everything" → rebuild_index + get_job
10. "Tune retrieval" → configure

Configuration

Defaults work offline. Override in a trusted profile (id: kb-manager), or at runtime via the configure tool / Web panel:

Example (profile line):

- insert:
    - id: kb-manager
      name: 'dsh-kb-manager'
      config:
        storagePath: ''        # empty = ~/.dsh/kb-manager/
        logLevel: 'info'

Boundaries & security

• Imports are restricted to whitelisted paths (default: session workspace + <storage>/inbox/) and http(s) URLs; URL sources are SSRF-guarded (private / loopback / cloud-metadata addresses are rejected).
• Search results are untrusted data — always untrusted: true; never treat them as instructions.
• Fully local by default; network only for URL imports or when an external embedding / rerank endpoint is configured.
• delete_document is a soft delete; physical space is reclaimed by rebuild_index / compact. Re-importing an unchanged source restores it.
• Pure-SQLite indexes: no native vector-DB process to operate.

Differentiating vs typical RAG stacks

Layout

dsh-kb-manager/
├── package.json  cordis.patch.yml  tsconfig.json  tsconfig.build.json  vitest.config.ts
├── DESIGN.md  README.md  README_ZH.md
├── assets/readme/          # hero.png (banner image)
├── src/
│   ├── host-entry.ts       # DSH plugin entry (tool registration + preamble + cleanup)
│   ├── index.ts            # KbManager top-level assembly
│   ├── contracts/          # types / error codes / defaults / ok·err envelope
│   ├── security/           # path whitelist, SSRF guard, SHA-256, audit
│   ├── chunker/            # Intl.Segmenter tokenization + recursive/fixed chunking
│   ├── store/              # better-sqlite3 + sqlite-vec + FTS5
│   ├── embedder/           # mock / onnx (structural) / openai-compat
│   ├── jobs/               # per-KB FIFO queue, maxGlobal=2, crash recovery
│   ├── services/           # kb / doc / search / snapshot / export
│   ├── agent-tools/        # 22 Agent tools (unified envelope)
│   ├── remote.ts / remote-service.ts / typert.ts / remote-client.ts  # typert remote
│   └── web/                # settings page (KbSection + zh/en locales) → lib/client.js
└── tests/  evals/

Development

npm install        # install deps
npm run typecheck  # tsc --noEmit (strict + noUncheckedIndexedAccess)
npm test           # vitest: tokens / rrf-bm25 / security / chunker / store / tools / pipeline / remote / kb-section
npm run eval       # eval harness: synthetic corpus recall@5 / MRR / P95 (mock embedder)
npm run build      # tsc -p tsconfig.build.json → lib/ (ESM + .d.ts)
npm run dev        # tsx src/host-entry.ts (module-load smoke test)

Known limitations (MVP)

• ONNX embedder is structural: onnxruntime-node is wired as an optional dependency; model download (sha256 + user confirmation) and WordPiece vocab are not yet connected — the onnx provider reports EMB.NOT_READY cleanly. Use provider: 'mock' for dev / test.
• PDF / DOCX parsing is a placeholder: md / txt / csv / json / html are available; pdf / docx report PARSE.UNSUPPORTED (upgrade path in DESIGN §13).
• v1.1 tools (rename_kb, snapshot trio, cancel_job, compact_index, multi_kb_search, export_kb, import_kb): service layer implemented, tool registration deferred.

License

MIT © 2026 xiaoshi7915