dsh-chinese-search

Chinese full-text search plugin for DeepSeek Harness memory plugins.

Problem

SQLite FTS5's default unicode61 tokenizer splits text by whitespace, which completely fails for Chinese (no word boundaries):

Input:  "电力监控系统"
FTS5:   ["电力监控系统"]  ← one giant token
Search: "电力" → 0 results ❌

Solution

Uses jieba (Rust-based, no native build) for proper Chinese word segmentation:

Input:  "电力监控系统"
jieba:  ["电力", "监控", "系统"]
Search: "电力" → ✅ hit

Installation

dsh plugin add dsh-chinese-search

Usage

With memory plugins

Works automatically with sage-mem, dsh-memory, and other memory plugins. Just install and it enhances search:

dsh plugin add dsh-chinese-search
dsh plugin add sage-mem  # or any other memory plugin

Programmatic API

import { ChineseSearchEngine } from 'dsh-chinese-search'
import Database from 'better-sqlite3'

const db = new Database('memory.db')
const engine = new ChineseSearchEngine(db)

// Create index
engine.createIndex('memory', 'content')

// Index documents
engine.indexDocument(1, '电力监控系统内生安全研究')
engine.indexDocument(2, 'DeepSeek V3 model architecture')

// Search (Chinese + English)
const results = engine.search('电力监控', 'memory')
// → [{ id: 1, content: '...', score: 0.95, matchedTerms: ['电力', '监控'] }]

Benchmark

Tested with 10,000 Chinese memory entries:

How it works

1. Tokenization: jieba segments Chinese text into words
2. Dual index: FTS5 for English + jieba token table for Chinese
3. Hybrid search: Combines results from both indexes
4. Scoring: Ranks by term frequency and match coverage
5. Fallback: LIKE query for partial/fuzzy matches

Configuration

# dsh config
chinese-search:
  enabled: true
  dictPath: ./custom_dict.txt  # Optional custom jieba dictionary
  autoIndex: true
  fuzzy: true
  limit: 20

License

MIT