dsh-anydoc-markdown

Document → Markdown + vision image description plugin for DeepSeek Harness (dsh).

Note: This plugin has only been tested against Deepseek harness version v0.1.1-rc.2. It might not work on the latest version. Ask agent to modify the plugin if it is incompatible with your version.

Converts documents (Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, PDF) into clean GitHub-Flavored Markdown using the Rust firecrawl-anydoc crate, and describes every embedded image with a vision model so a text-only model can reason about the document's imagery.

Design target. anydoc is a pure local, dependency-free Rust converter: its core has no network calls and no llm_client parameter. But it exposes anydoc.to_document(), which returns the full document model plus the embedded binary assets (images, MIME types, and alt-text markers) on Document.assets. This plugin replicates the MarkItDown llm_client + llm_model pattern for anydoc: intercept the image assets, pass them to a VLM (OpenAI's gpt-4o or a local OpenAI-compatible model), and substitute the image placeholders in the output Markdown.

Features

• One shared Markdown output for every format — anydoc funnels every format through one document model and GFM serializer, so headings, tables, lists, equations, links, footnotes, and speaker notes come out consistently.
• Embedded-image description (vision pipeline) — images are described by a VLM and substituted into the Markdown as ![<description>](asset://<id>). Images the converter drops (empty alt text) are appended as a described list.
• Batched vision requests (the important part) — images are never all sent in one API request. They are split into batches of maxImagesPerRequest (default 10) and each batch is a separate request, so vision performance is not degraded by a giant multi-image prompt. Configurable.
• convert_document tool for agents — the agent converts any document to Markdown on demand (offset/limit-free; returns the full Markdown, capped).
• Composer document attach (native drop + dock strip) — documents attach through the same whole-page drag the harness natively uses for images (there is no composer tool-row paperclip button and no competing drag overlay, which previously froze the harness's own drop overlay). Dropped documents show as attachment cards in a slim strip in the composer's input dock, above the composer card. Documents are stored and the agent is handed the source file path; images keep the harness's built-in image path.
• Extracted images saved as PNG — when markdownOutputDir is set, convert_document creates a subfolder there named after the Markdown file (e.g. report/) and saves both the .md file and every extracted embedded image as a PNG inside it. The Markdown references those on-disk PNGs by relative filename, so the .md and its images are portable together.
• Agent-driven conversion — the plugin never injects the converted Markdown. It hands the model the saved source file path(s) and lets the agent call convert_document on each. The converter saves the Markdown (and its images) to markdownOutputDir and returns the .md file path.
• Offline-first — when no VLM endpoint/model is configured, the converter still produces Markdown and still batches, using an honest offline metadata description for each image. No network call, no API key required.
• Self-contained — the host half has no @deepseek-ai/* imports (always loads), and the client half rides the standard __ModuleLoader__ slot/RPC seams.

Install

pip install -r converter/requirements.txt   # firecrawl-anydoc + pdfplumber + Pillow
dsh plugin --profile web add dsh-anydoc-markdown
# restart dsh web

pdfplumber + Pillow are only needed to describe images embedded in PDFs. If they're unavailable, PDFs still convert (text only) and the rest of the vision pipeline is unaffected.

Usage

1. The agent calls the convert_document tool with the path to a document: convert_document(file_path: "report.pptx").
2. The plugin runs the bundled Python wrapper (converter/anydoc_vlm.py) against anydoc, then describes every embedded image via the VLM in batches.
3. The tool returns the Markdown (with image descriptions substituted) to the model.

Composer document attach (native drop / multi-file)

Documents attach through the same whole-page drag the harness uses for images: drag any number of documents (.docx .pdf .pptx .xlsx .odt .rtf .epub .csv .txt .md .html) onto the page and release. The harness's own drop overlay (the one that says "Drag images and documents here to add them") invites the drop, and the plugin shows the attached documents as cards in the composer's input dock strip (a line above the composer card). Images dropped at the same time keep using the harness's built-in image attach.

1. Drag documents onto the page (or drop them) → they stream to the host's attach.upload RPC channel (/dsh-anydoc by default). Each uploaded document is written to attachmentsDir and shown as a card (name + size + remove) in the dock strip. The cards clear automatically once you submit the message — the documents are already saved to disk, so the agent still gets their paths on the next step.
2. On the next agent step, the plugin hands the model the saved source file path(s) as a model-visible pre-step message ("Use convert_document on each path below"). It never injects the converted Markdown.
3. The agent calls convert_document(file_path: "<source path>") on each. The tool converts the document, saves the Markdown to markdownOutputDir, and returns the .md file path (savedPath) alongside the Markdown.
4. Failed/oversized/unsupported files are reported individually and never break the rest of a batch.

Configuration

All fields have sensible defaults — install and use without touching any. Tune only what you need. Every knob is a validated cordis.patch.yml field; none are hardcoded.

- id: dsh-anydoc-markdown
  config:
    maxImagesPerRequest: 10
    visionEndpoint: https://api.openai.com/v1
    visionModel: gpt-4o

How images are handled

The VLM call is OpenAI-compatible; point visionEndpoint/visionModel at any endpoint speaking the chat-completions wire format (OpenAI, a local Ollama VL model, an OpenAI-compatible proxy, etc.). With visionEndpoint: '' (default) the wrapper uses offline metadata descriptions.

Vision pipeline (converter details)

The Python wrapper converter/anydoc_vlm.py is where the conversion + image description happens (see converter/README.md for the full write-up):

• Batched requests (the hard requirement). Images are never all sent in one API request — that degrades vision performance. describe_many() slices them into groups of maxImagesPerRequest (default 10) and issues one chat-completions request per group, so 23 images yield [10, 10, 3] across 3 requests rather than one giant multi-image prompt.
• Retries + tolerant parsing (fail-closed). Each batch is retried up to 3 times (with backoff) on an HTTP error, a timeout, or a response that isn't a JSON array of exactly N descriptions. Array responses wrapped in prose or a fenced JSON block are still parsed. Only if every retry fails does that batch degrade to an offline metadata description — and the note then states why (vision request failed after retries / vision unavailable after retries) instead of claiming the model is unconfigured.
• PDF support. PDFs carry no anydoc document model, so their embedded raster images are recovered from the PDF bytes with pdfplumber and described the same way. Needs pdfplumber + Pillow; fails closed to text-only Markdown if they're missing. Scanned pages need --ocr hosted (Firecrawl Parse).
• Offline-first, honest metadata. With no endpoint/model configured, the wrapper still produces Markdown and still batches, using an honest offline metadata description per image ([Offline metadata description - no vision model was configured.]). No network call, no key required.

Converter CLI

python converter/anydoc_vlm.py <path> \
  [--format NAME] [--ocr reject|hosted] [--max-images N] \
  [--vision-endpoint URL] [--vision-model MODEL] [--api-key KEY] \
  [--save-images-dir DIR] [--mock-vision] [--json] [--out FILE]

Verify the batching with the bundled test document (23 images, max 10):

python converter/anydoc_vlm.py tests/multi.docx --mock-vision --max-images 10 --json
# vision_batch_sizes: [10, 10, 3], vision_requests: 3

Architecture

dsh-anydoc-markdown/
├── index.js                 # host plugin: `convert_document` tool + `attach.upload` RPC channel
├── lib/client.js            # web client: document drop + dock-strip cards (no tool-row button/overlay)
├── cordis.patch.yml         # bundle config (all knobs)
├── package.json             # dsh.bundle.patch + dsh.client, files
├── converter/
│   ├── anydoc_vlm.py        # anydoc + batched-vision engine (the "llm_client" seam)
│   ├── requirements.txt     # firecrawl-anydoc + pdfplumber + Pillow (PDF images)
│   └── README.md            # wrapper docs + how the batching works
└── README.md

The Python wrapper is where the real work happens (see converter/README.md): anydoc.to_document() yields the embedded assets, VisionClient.describe_many() batches them (≤ 10 per request), and substitute() swaps the image placeholders for the descriptions.

Development

python converter/anydoc_vlm.py tests/multi.docx --mock-vision --max-images 10 --json
node --check index.js
python -m py_compile converter/anydoc_vlm.py

Security & notes

• anydoc runs conversion locally; only scanned PDF pages opted into --ocr hosted leave the machine (Firecrawl Parse). The vision call itself goes only to the configured visionEndpoint.
• The plugin fails closed: if the interpreter is missing, anydoc isn't installed, or a conversion errors, that call fails loudly while the harness keeps working the way dsh-file-upload's design does.
• Uploaded documents are written to attachmentsDir (persistent) and the converted Markdown (plus its extracted image PNGs) to a per-document subfolder inside markdownOutputDir; both are configurable and written only when you configure them. Filenames are sanitized (no path traversal), non-allowlisted extensions are rejected, and files are capped by maxFileBytes.

License

MIT