dsh-research

Local, free, keyless research layer for dsh coding agents.

Other tools give the agent a chunk of page text. dsh-research gives it a filesystem. GitHub repos shallow-clone into real source trees; web pages and PDFs become clean markdown files in your workspace. The agent navigates them with the grep/read/ls tools it already has — no API spend, no proprietary content jail, clones cached across sessions.

$ agent: "fetch_content https://github.com/liliMozi/openhanako"
→ Materialized → .cache/dsh-research/repos/liliMozi/openhanako
  (real source tree — grep it, read any file, zero context cost)

Why

Coding agents researching external material face two bad options: fetch a page and dump 100k characters into context, or paste HTML crumbs. Firecrawl's MCP solves this in the cloud, per-request, with a key. pi-web-access solves it inside a closed artifact store the agent can only page through.

Materialization solves it locally: external content becomes plain files at stable paths, the context window only receives paths plus a preview, and the agent pays tokens for exactly what it greps. Fetch the same URL tomorrow — cache hit, zero cost.

Tools

Web search is deliberately not here — dsh's packages/web family already owns web_search.

Storage layout

<workspace>/.cache/dsh-research/
├── repos/<owner>/<repo>/    # shallow clones, reused across sessions
├── pages/<url-hash>/
│   ├── content.md           # the materialized markdown
│   └── index.json           # provenance (url, kind, time)
└── tmp/                     # in-flight PDF downloads

Config

- id: dsh-research
  name: '@ajin/dsh-research'
  githubToken: <optional PAT, env fallback GITHUB_TOKEN>
  previewChars: 1500

Requirements

• Node.js >= 18, git on PATH
• PDF conversion: Python 3.9+ with pymupdf4llm (shared with dsh-doc-to-markdown)

Development

npm install
npm test        # vitest

Status

0.2.0 — materialization pipeline complete: three sources, hash-addressed cache, cache-hit short-circuit, path-scoped read_response.