dsh-bilibili

中文 | English

A DeepSeek Harness tool plugin that gives agents a bilibili_extract tool. Send a Bilibili link and the agent extracts the video's text information (transcript / comments / danmaku), captures keyframes on demand, and produces a summary.

This plugin bundles no third-party binaries or models; the open-source projects and services it invokes are listed in THIRD_PARTY_NOTICES.md.

---

✨ Features

• Full text extraction: metadata, complete timestamped transcript (long transcripts are truncated with a full-text time index), hot comments (with replies), danmaku (top repeated messages + density-peak timeline samples — opening spam no longer dominates); videos without a subtitle track are auto-transcribed — Bijian ASR by default (the same anonymous capability behind Bilibili's "live AI subtitles", 24h cache), or switch to local engines — sherpa-onnx (Chinese, SenseVoice) or whisper.cpp — fully offline; one failing source never breaks the rest (each degrades to empty with a note);
• Optional frame vision descriptions: vision-less main models can still "see" frames — send each frame to a local Ollama / llama.cpp backend (Qwen3-VL 2B/4B/8B tiers) or any OpenAI-compatible vision API for a text description; the report cites images only when needed;
• Automatic frame selection (picture-driven only): scene-change detection (sampled pass beyond 20 minutes) + even-interval backfill, 5s dedupe; no keyword guessing — deciding "which transcript moments are incomplete and need visuals" is semantic analysis, left to the main agent's two-pass prompts;
• Sharp-frame preference: within ±1.5s of each target time, FFmpeg blurdetect scores every frame and the sharpest one wins — motion-blurred animation entrances and fade frames are skipped;
• Two-pass workflow: the agent reads the transcript first (instant, zero download), then requests frames with explicit timestamps — each frame is captioned with its nearby subtitle; the agent reports needed moments in a fixed [建议抓帧] mm:ss format; 24h video cache reuse across passes;
• Replaceable output template: a concise shareable summary template is bundled; summaryTemplate can point to any custom template file;
• Download-first capture: the video is downloaded locally before frame extraction (≤30 min / ≤800MB), with automatic fallback to remote per-frame extraction;
• Robust: exponential backoff on Bilibili 412 rate limits; login-required subtitles are detected with a SESSDATA hint; ffmpeg runs pipe-free, so it works in any environment.

---

🚀 Installation

Prerequisites: Node 18+, ffmpeg on PATH, pnpm.

# Option 1: install from GitHub (recommended)
dsh plugin --profile web add git+https://github.com/CZX2244/dsh-bilibili

# Option 2: local directory (development; link mode applies changes instantly)
dsh plugin --profile web add ./dsh-bilibili

# Restart the web profile (dsh web); bilibili_extract becomes available in new sessions

After installation the tool joins the agent's toolchain: when the user sends a Bilibili link (bilibili.com/video/BV..., a b23.tv short link, or a bare BV id), the model can call it for analysis. The plugin also registers bilibili_login (QR-code login) and bilibili_doctor (environment check, below).

---

🩺 Environment check (first-run probe)

Before the first extraction the plugin runs a three-layer environment probe automatically; the report is cached for about an hour (invalidated when relevant config changes). It reports only what is missing, so nothing gets installed twice and no bandwidth is wasted:

Layer	Checks	Behavior on failure
Local dependencies	ffmpeg / whisper-cli / sherpa-onnx binaries, model and tokens files, vision endpoint, output dir writability	ffmpeg missing → video download and frame capture are skipped (no more 800 MB download before a raw spawn error); a not-ready local ASR engine is skipped before downloading the audio stream
Config	asrProvider validity, sherpa/whisper required fields, visionBaseUrl resolution	Invalid/missing items are marked error/warn with fix hints
Cloud	Bilibili main API, Bijian ASR, login service reachability; SESSDATA validity (verified via nav isLogin, not just file existence)	Unreachable/expired items are marked unreachable/warn; an unreachable Bilibili main API is flagged as "plugin unusable" (local ASR engines get their audio stream from playurl too)

• The report is attached to every extraction result (an Environment check section); the agent relays the fix hints to the user, or a single line when everything passes;
• bilibili_doctor re-checks on demand; refresh=true forces a fresh probe (use after installing/updating dependencies or changing config);
• Probes only check existence/reachability: no real ASR tasks are submitted and no media is downloaded; a failing check degrades, never blocks extraction;
• The report never contains raw credentials (SESSDATA validity only).

---

🎨 Custom output templates

The output format is a replaceable part of the plugin: the tool provides the data (transcript / frames / danmaku / comments), the template decides what it looks like.

• Bundled default: templates/summary.md — a concise "time-saver" summary (one-sentence takeaway → timestamped key points → worth-watching segments → shareable closing lines);
• Bundled alternative: templates/timeline.md — a generic time-axis format (title → hook → timestamped sections with inline points → image anchors → conclusion); add images only when they help, never force them;
• Switch templates: set summaryTemplate: 'C:/path/my-template.md' in the config;
• Change the default: edit templates/summary.md directly in the plugin directory;
• An invalid custom path falls back to the bundled template, so the tool never breaks because of a template.

For richer output formats (study notes, review tables, timelines, Q&A cards, etc.), install the companion skill bilibili-video-analyzer (an A–K format catalog) and the agent picks by user intent.

---

🧠 Recommended workflow (two-pass; prompt-driven is the primary path)

Frame selection is not keyword guessing — the main agent uses the analysis prompts injected into the system prompt to read the transcript and decide which moments are incomplete without the picture. The built-in "content-completeness check" teaches the agent five gap types — dangling reference / conclusion without data / unspoken operations / silent demos / visual comparisons — and asks it to report them explicitly:

① bilibili_extract(url, extract_frames: false)      # text only: instant, zero download
② the agent scans the transcript for information gaps and picks the moments that need visuals
③ the agent lists them as [建议抓帧] mm:ss reason, then calls again with timestamps —
   the plugin aligns each moment to the most changed frame within ±4s (FFmpeg scene detection),
   then picks the sharpest frame within ±1.5s (blurdetect): semantic targeting → picture refinement → sharpness gate
④ the agent decides which images to cite in the report using description/citation_hint (or read_image)

Fallback: only a single call (no timestamps) uses automatic selection — purely picture-driven: scene changes (sampled pass beyond 20 min) + even-interval backfill. That's insurance for "the model skipped the two-pass flow", with no keyword guessing.

Precision note: captured frame times may differ from the requested time by up to about ±0.5s (fast seek depends on keyframe spacing), and more after scene alignment and sharp-frame refinement — the exact requested moment is shown per frame as 原始请求 in the report.

---

🔧 Configuration

Defaults live in cordis.patch.yml; override any field in $DSH_HOME/profiles/web/cordis.patch.yml (later layers win per row):

- override:
    - id: bilibili
      config:
        sessdata: ''                 # optional Bilibili SESSDATA (logged-in subtitles / more comments)
        commentLimit: 20             # max comments to fetch
        maxFrames: 6                 # max keyframes
        extractFrames: true          # false = text-only mode
        downloadVideo: true          # download video locally before capture (recommended)
        keepVideo: false             # true = keep downloaded video files
        maxVideoMinutes: 30          # videos longer than this are captured remotely per frame
        maxDownloadMb: 800           # download size cap (MB)
        quality: 32                  # 16=360p 32=480p 64=720p 80=1080p
        detectScenes: true           # scene-change detection (sampled pass beyond 20 min)
        sceneThreshold: 0.4          # scene threshold 0-1, higher = stricter
        sharpFrames: true            # sharp-frame preference: blurdetect picks the sharpest frame within ±1.5s
        asrProvider: 'bcut'          # ASR engine: bcut (default) | sherpa-onnx (Chinese) | whisper-local | auto | none
        sherpaBin: ''                # sherpa-onnx-offline binary path
        sherpaModel: ''              # sherpa model onnx path (SenseVoice/Paraformer)
        sherpaModelType: 'sense-voice'  # sense-voice | paraformer | zipformer2-ctc
        sherpaTokens: ''             # sherpa tokens.txt path
        sherpaThreads: 0             # sherpa CPU threads (0 = auto)
        whisperBin: 'whisper-cli'    # whisper.cpp binary (PATH or absolute path)
        whisperModel: 'medium'       # small / medium / large-v3, or a ggml-*.bin path
        whisperModelDir: ''          # model dir; empty = <whisperBin dir>/models
        whisperLanguage: 'zh'        # transcription language
        whisperThreads: 0            # whisper CPU threads (0 = auto)
        visionProvider: 'none'       # frame vision: none (default) | ollama | llama-cpp | openai-compatible
        visionBaseUrl: ''            # vision endpoint; empty + ollama = http://localhost:11434/v1
        visionModel: 'medium'        # low(2B) / medium(4B) / high(8B), or an explicit model name
        visionApiKey: ''             # cloud vision API key (empty for local)
        visionPrompt: ''             # vision prompt (empty = built-in per-model default)
        visionPromptByModel: {}      # per-model prompt overrides (explicit model name / low / medium)
        visionMaxFrames: 6           # max frames to describe (aligned with maxFrames)
        framesDir: ''                # frame output dir; empty = system temp/dsh-bilibili/<bvid>
        summaryTemplate: ''          # template path; empty = bundled templates/summary.md
        timeoutMs: 300000            # overall tool timeout (ms)

Local ASR transcription (optional; sherpa-onnx recommended for Chinese)

No-subtitle videos default to Bijian (zero config, anonymous, China-friendly). To go fully offline or when Bijian fails, switch to a local engine. The plugin ships no models — only the interface; models and binaries are downloaded by the user (no API keys, quotas, or fees involved).

Recommended: sherpa-onnx (Chinese, SenseVoice)

Bilibili is mostly Chinese content, and SenseVoice beats Whisper on Chinese accuracy while being faster and smaller; official models are hosted on ModelScope (fast in China).

Tier	Recommended model	Size (approx)	For
Low	SenseVoiceSmall (int8)	~230 MB	low-end machines
Mid	SenseVoiceSmall (fp32)	~900 MB	mainstream (recommended)
High	Paraformer-large	~2.5 GB	high-end / maximum accuracy

Steps:

1. Download the sherpa-onnx-offline binary for your OS from sherpa-onnx;
2. Download a model (model.onnx + tokens.txt) — SenseVoice models are on ModelScope or the sherpa-onnx model list;
3. Set asrProvider: 'sherpa-onnx' and fill sherpaBin / sherpaModel / sherpaTokens (sherpaModelType defaults to sense-voice).

Alternative: whisper.cpp (general / English)

Tier	whisperModel	Model file	Size	For
Low	small	ggml-small.bin	~466 MB	low-end / fast drafts
Mid	medium	ggml-medium.bin	~1.5 GB	mainstream
High	large-v3	ggml-large-v3.bin	~3 GB	high-end

1. Download whisper-cli from whisper.cpp;
2. Download the matching ggml-*.bin model into a models/ directory;
3. Set asrProvider: 'whisper-local' and fill whisperBin / whisperModel.

Note: asrProvider: 'auto' falls back in order Bijian → sherpa-onnx → whisper-local. For Chinese, use at least medium (whisper) or pick SenseVoice (sherpa) directly. sherpa-onnx CLI flags vary slightly across versions — check --help of your build if something errors.

🔍 Frame vision descriptions (optional)

When the main model has no vision, enable this feature: each captured frame is sent to a vision model and returned with a description field, so the main model can decide which images to cite in the report — cite only when visual confirmation matters (charts / UIs / demo details); pure talking-head frames are not cited. Off by default; a vision failure never breaks the main flow (frame paths are still returned).

Local (recommended): install Ollama and pull a model — no keys, offline, free:

Tier	visionModel	Ollama model	RAM (approx)	For
Low	low	qwen3-vl:2b	~2 GB	ultra low-end
Mid	medium (default)	qwen3-vl:4b	~4 GB	low-end to mainstream (recommended)
High	high	qwen3-vl:8b	~6-8 GB	mainstream, best quality

Larger models can be passed as explicit names (e.g. qwen3-vl:32b) — they are just no longer a default tier.

Mid-tier alternative MiniCPM-V 4.0 (OpenBMB, 2026; officially claims to surpass GPT-4.1-mini, runs on phones; official GGUF/int4 releases — check the official library for its Ollama tag). visionModel also accepts explicit model names (Ollama tags or cloud model ids).

Other non-Qwen models (verified on the Ollama library, 2026-08): minicpm-v:8b (OpenBMB MiniCPM-V 2.6, strong Chinese OCR), moondream (1.9B, English-first), gemma3n (Google, English-first). Kimi-VL / InternVL / GLM-4V are not in the official Ollama library — use community GGUFs via llama.cpp or cloud OpenAI-compatible APIs (e.g. Moonshot / Zhipu). MiniCPM-V 4.0's official GGUF works on the llama-cpp route.

Selection rationale: this task is understanding frame content + emitting a citation hint, not OCR transcription — the weights are on Chinese scene understanding and instruction-following, so the default tiers use the Qwen3-VL family (consistent behavior, shared prompts); MiniCPM-V 4.0 for maximum edge efficiency.

llama.cpp (local alternative): run llama-server with a vision GGUF (model + mmproj); it exposes an OpenAI-compatible API, and visionModel is simply the --alias you set at launch — matching the alias to a tier keyword reuses the tier config directly:

llama-server -m qwen3-vl-8b-q4_k_m.gguf --mmproj mmproj-qwen3-vl-8b.gguf --port 8080 --alias qwen3-vl:8b
# plugin config: visionProvider: 'llama-cpp' + visionModel: 'medium'

The three tiers are Qwen3-VL-first: low Qwen3-VL-2B, mid Qwen3-VL-4B (default), high Qwen3-VL-8B. Verified GGUFs: official Qwen/Qwen3-VL-4B/8B-Thinking-GGUF (with mmproj), community unsloth/Qwen3-VL-4B-Instruct-GGUF, etc. If your llama.cpp build doesn't support the Qwen3-VL architecture yet, fall back to Qwen2.5-VL-7B-Instruct-GGUF (official on ModelScope). llama.cpp also supports MiniCPM-V (incl. 4.0), InternVL, GLM-4V, LLaVA, gemma3n, moondream2, and more.

Cloud: any OpenAI-compatible endpoint via visionProvider: 'openai-compatible' + visionBaseUrl + visionModel + visionApiKey. Single-frame description doesn't need flagship multimodal models — budget tiers suffice: GLM-4V-Flash (free quota for Chinese) / GPT-4o-mini / SiliconFlow Qwen-VL.

Per-model prompts: every built-in prompt's task is understanding the frame's content (what's happening, what's shown) — visible text is paraphrased as key points only, never transcribed. The plugin picks prompts automatically per model (MiniCPM-V family gets a dedicated prompt, moondream2 gets English, small low-tier models get a shorter prompt); override with visionPrompt (global) or visionPromptByModel (per explicit model name or low/medium tier).

Citation quality gate: every vision description must end with a single line 「配图建议：适合/不适合」 (suitable = clear, informative, helps readers understand; unsuitable = talking head, blurry, or uninformative). Frames carry a citation_hint field, and reports cite only suitable frames, at most 1-2 per section.

2B measured results (2026-08, llama.cpp b10428 + Qwen3-VL-2B-Instruct-Q4_K_M, 16-thread CPU, 7 ground-truth test images): 100% citation-tail stability; chart values (120/240/180/300) and poster numbers (32% / 5M / three rounds) matched exactly; talking-head frames correctly marked unsuitable; 4-9s per frame. The low-tier short prompt was tuned from these live runs (anti-hallucination + citation criteria).

Note: describing several frames on a CPU takes tens of seconds to minutes (faster on GPU); visionMaxFrames caps the count.

---

📁 Project structure

dsh-bilibili/
├── lib/
│   ├── index.js        # Cordis plugin entry: tool registration + system prompt + config schema
│   ├── extractor.js    # extraction layer: Bilibili API + downloads + scene detection + ffmpeg capture
│   ├── keyframes.js    # pure functions: automatic frame selection (picture-driven), time formatting
│   └── format.js       # pure functions: extraction result → model-facing text digest
├── templates/summary.md  # bundled default output template (replaceable)
├── test/                 # unit tests (node --test)
├── cordis.patch.yml      # bundle patch layer (recognized by the plugin system)
└── package.json          # dsh.bundle.patch declaration + peer dependencies

---

🔌 Plugin standard

This plugin follows the DeepSeek Harness plugin standard: the npm package declares dsh.bundle.patch → dsh plugin add reconciles it into dsh.profile.bundles → the Cordis loader mounts it after a profile restart. See the deepseek-harness repo for the standard.

---

🛠️ Local development (link mode)

dsh plugin add installs local directories via link: (changes take effect immediately). Because ESM resolves dependencies by real path, the plugin directory needs a junction pointing at the profile's node_modules:

New-Item -ItemType Junction -Path ".\node_modules\@deepseek-ai" `
  -Target "$env:USERPROFILE\.dsh\profiles\node_modules\@deepseek-ai"

Restart the web profile after changes.

---

⚠️ Limitations

• Multi-part videos: only part 1 is handled for now;
• Videos without subtitles are transcribed via Bijian ASR by default (local sherpa-onnx / whisper.cpp available); transcripts may contain recognition errors and are labeled as such in the result;
• Bijian ASR is an anonymous endpoint and may rate-limit rapid repeated calls (returns errors); for frequent/stable transcription prefer asrProvider: 'auto' or a local sherpa-onnx setup;
• Frame images are not auto-cleaned (so the model can read_image anytime) — that costs disk space;
• Node's fetch doesn't read system proxy env vars; proxied networks are pending support;
• Scene detection switches to a sampled pass beyond 20 minutes (full decode cost).

---

📄 License

MIT