dsh-nature-papers

A DSH Web plugin that scrapes nature.com in real time for the latest papers on biochemistry / bioinformatics, ranks them with journal impact factor (IF) as the primary quality criterion, and shows 3 papers per day in a floating panel at the bottom-right corner of the page — with direct links to the originals, journal IF, and content summaries. Every daily recommendation is archived by date (year-month-day) and can be browsed later. The panel footer also offers a one-click Exit Harness action.

This document is the full technical implementation guide (open-source documentation): architecture, per-feature implementation details, the Nature anti-bot automatic fallback mechanism, data formats, and API contracts.

---

Table of Contents

1. Feature Overview
2. Directory Structure
3. Architecture: The DSH Plugin Model
4. Feature Implementation Details
   • 4.1 Real-time scraping (nature.com search pages)
   • 4.2 Nature anti-bot automatic fallback (Client Challenge → PubMed)
   • 4.3 Impact-factor-first ranking
   • 4.4 Content summaries (abstract fetch chain)
   • 4.5 Three papers per day and caching
   • 4.6 History and de-duplication
   • 4.7 Real-time refresh
   • 4.8 The bottom-right panel (client side)
   • 4.9 Exit Harness
5. HTTP API Contract
6. Configuration
7. Storage Format
8. Rate Limiting, Fault Tolerance & Anti-bot Measures
9. Known Limitations
10. Operations: Install / Update / Start / Stop / Uninstall

---

Feature Overview

• 🧬 3 papers per day: updates automatically each day (re-scrapes across day boundaries); cached within a day so it never flickers.
• 🏆 Impact-factor first: ships a reference table of approximate JCR impact factors for ~90 journals; candidates are sorted by "IF descending → publication date descending", and each card shows IF x.x.
• 🔗 Direct links: the title opens the nature.com original in a new tab, plus a DOI link.
• 📄 Content summaries: prefers the article-page Abstract, falls back to the search-page excerpt; expandable/collapsible.
• 📜 History: archived per YYYY-MM-DD, expandable to review any past day.
• 🔄 Real-time refresh: the button re-scrapes immediately and re-picks (preferring papers never recommended before).
• ⏻ Exit Harness: gracefully shuts down the DSH server process after a two-step confirmation.
• 🛡 Anti-bot automatic fallback: when nature.com serves a Client Challenge, the plugin automatically switches to the PubMed E-utilities mirror (still scoped to Nature Portfolio journals; links still point to nature.com).

---

Directory Structure

nature-papers/
├── package.json      # package manifest: dsh.client metadata, exports map
├── README.md         # English documentation (this file)
├── README_zh.md      # Chinese documentation
├── LICENSE           # MIT license
├── .gitignore
├── install.ps1       # install/update/uninstall script
├── start-dsh.ps1     # start dsh web script
├── stop-dsh.ps1      # stop dsh web script
├── restart-dsh.ps1   # restart dsh web script
├── test-scraper.mjs  # standalone scraper-core test
└── lib/
    ├── index.js      # host side (Cordis plugin): routes / storage / daily scheduler / exit
    ├── scraper.js    # scraping core (pure Node, testable standalone): dual sources, parsing, ranking
    ├── if-data.js    # journal impact-factor reference table + name normalization
    └── client.js     # browser bundle: bottom-right panel (shell.overlay slot)

Companion scripts live in the repository root (run the commands below from there):

Script	Purpose
install.ps1	Syncs code into the install dir + writes the cordis.patch.yml entry (supports -Uninstall)
start-dsh.ps1	Starts dsh web in the background (hidden window, logs to disk)
stop-dsh.ps1	Stops the dsh web process precisely by the listening port
restart-dsh.ps1	Kill old process → confirm port free → start new process → self-check the plugin
test-scraper.mjs	Standalone scraper-core test (no Cordis dependency)

---

Architecture: The DSH Plugin Model

DSH Web is a Cordis plugin tree: the web profile's empty root config cordis.yml is composed from several patch layers (@deepseek-ai/dsh-base, @deepseek-ai/dsh-web-app bundle layers + the user layer cordis.patch.yml). Mounting a plugin = inserting a loader entry into the composed tree and having it imported and activated.

Mounting chain (host side)

1. Place the package: copy this directory to $DSH_HOME\profiles\node_modules\dsh-nature-papers (a real directory). The DSH server resolves bare Node module specifiers by walking up from the profile directory, so both import('dsh-nature-papers') and its dependency @deepseek-ai/schemastery resolve.

2. Register the entry: write to cordis.patch.yml (maintained automatically by install.ps1):

   - insert:
       - id: nature-papers
         name: dsh-nature-papers
         config:
           query: 'bioinformatics biochemistry'
           count: 3
           requestDelayMs: 1000
           storageFile: !!js dshHomePath('storages/nature-papers.json')

   The insert patch appends the entry to the composed tree; the !!js expression is evaluated at entry activation (the eval scope is injected with dshHomePath by boot()).

3. Activation: the loader import()s the module by name → unwrapExports takes the default export { name, inject, Config, apply } → Cordis resolves inject: ['webServer'] (waits for the service) → validates config via the schemastery Config → runs apply(ctx, config): registers 5 HTTP routes, loads/persists the history store, and starts the daily-rollover timer. When the entry is stopped or updated, the disposers inside ctx.effect unregister the routes and clean up timers.

Mounting chain (client side)

lib/client.js is a browser bundle, recognized through the package's dsh.client declaration (platform: web) and exports["./client"]:

1. dsh-client-modules (host side) scans loader entries for packages declaring dsh.client → hashes the bundle content into the window.__DSH_BOOT__ manifest → serves it at /plugins/dsh-nature-papers/client.js.
2. On page load the shell reads __DSH_BOOT__ → loads the bundle script → the bundle calls window.__ModuleLoader__.load({ id, factory }) to register the factory (lazy CJS: registering ≠ executing).
3. The Cordis client loader materializes the factory (factory(require), where require('react') is available) → plugin object { name, inject: ['slots'], apply }.
4. apply() registers the panel component into the shell.overlay slot via ctx.slots.inject('shell.overlay', ...) — that slot is rendered by ui-layout's AppFrame overlay layer (position:absolute; inset:0), and the panel is positioned with position:absolute; right:16px; bottom:16px — i.e. the bottom-right corner of the page.

Data flow

Browser panel ──fetch──▶ /plugins/dsh-nature-papers/* ──▶ host routes
                                                          │
                                            ┌─────────────┴──────────────┐
                                      Source A: nature.com search      Source B: PubMed E-utilities
                                            └─────────────┬──────────────┘
                                                          ▼
                                          IF ranking → de-dup → abstract enrichment
                                                          ▼
                                           $DSH_HOME/storages/nature-papers.json

---

Feature Implementation Details

4.1 Real-time scraping (nature.com search pages)

Request construction (scrapeNatureSearch):

• URL: https://www.nature.com/search?q=<query>&order=date_desc, where query is space-separated keywords (AND semantics), default bioinformatics biochemistry.
• Pagination: at most 3 pages (&page=2/3); stops early when a page yields < 10 rows or the candidate cap (maxCandidates, default 90) is reached; requestDelayMs between pages.
• Headers: browser UA + accept: text/html,...; timeout requestTimeoutMs; one retry on failure (1.2s backoff).

Row parsing: split each result by <li class="app-article-list-row__item"> and extract fields with regexes:

Field	Anchor	Notes
Title	<h3 class="c-card__title">…<a href="/articles/<id>">	tags stripped, entities decoded
Link	href="(/articles/[a-zA-Z0-9-]+)"	assembled as https://www.nature.com/articles/<id>
Type	data-test="article.type">…<span class="c-meta__type">	dropped when on the blocklist (news/comment/editorial/news & views/…)
Journal	data-test="journal-title-and-link">	plain text
Date	<time … datetime="YYYY-MM-DD">	ISO date, used as the secondary sort key
Excerpt	data-test="article-description">…<p>…</p>	1–2 sentence summary shipped with the search page (fallback)
Open access	row contains u-color-open-access	boolean flag
DOI	derived from the article id: /^s\d+-\d+/ → 10.1038/<id>	null for other formats (e.g. BMC journals)

4.2 Nature anti-bot automatic fallback (Client Challenge → PubMed)

Background: for scriptless crawlers nature.com serves a JavaScript challenge page ("Client Challenge", ~3 KB of HTML containing a loadScript routine); a plain HTTP client cannot pass it (a real browser must execute JS to obtain a cookie).

Detection (two places):

• Search page: html.includes('Client Challenge') || !html.includes('app-article-list-row') → throws nature.com 触发了反爬校验（Client Challenge），已切换备用源 (nature.com served a Client Challenge; switched to the fallback source).
• Article page: when Client Challenge is hit, that paper's abstract is treated as unavailable, returns null, and the excerpt fallback is used.

Fallback flow (generateEntry):

1. Try source A first (live nature.com scraping);

2. On error, automatically switch to source B (PubMed E-utilities) without interrupting the user's request;

3. Source B query construction:

   • esearch: term = ("Nature"[ta] OR "Nature Communications"[ta] OR …) AND (bioinformatics[tiab] OR biochemistry[tiab] OR "computational biology"[tiab] OR …), with sort=date&retmax=90 — the [ta] journal field pins the search to ~55 Nature Portfolio journals, and the [tiab] topic terms keep topical relevance;
   • esummary: fetches titles, full journal names, publication dates and DOIs for all hits (up to 90) in one call;
   • News filtering: Nature news items have DOIs like 10.1038/d41586-…; rows where doi.startsWith('10.1038/d4') are dropped — only research-type papers remain;
   • Link restoration: DOIs with the 10.1038/ prefix map back to https://www.nature.com/articles/<suffix> (links still point to nature.com); other prefixes go to https://doi.org/<doi>;
   • Abstracts: one batched efetch (retmode=xml&rettype=abstract) for the top count*2 PMIDs; the XML is split on <PubmedArticle> blocks to extract <ArticleTitle> / <AbstractText> (multiple sections joined) / <Journal><Title> / <PubDate> / <ELocationID EIdType="doi">.

4. The panel labels the source of the batch ("来源：Nature 实时" / "来源：PubMed 镜像" — Source: Nature live / Source: PubMed mirror) and surfaces the switch reason in the response (sourceError), displayed in a notice bar at the top of the panel.

Proactive anti-bot measures: browser UA, request delay (default 1.5 s), per-request timeout, retry with backoff, page cap, fetching article pages only when the excerpt is insufficient (fewer requests), and same-day caching (no repeated scraping within a day).

4.3 Impact-factor-first ranking

Data (if-data.js): approximate JCR impact factors (mostly the 2023 release) for ~90 journals — covering Nature (flagship), Nature research journals, Nature Reviews journals, the npj series, and high-IF Springer Nature/BMC journals hosted on nature.com (Signal Transduction and Targeted Therapy 40.8, Cell Research 28.1, Molecular Cancer 27.7, Genome Biology 10, etc.).

Name normalization: lowercase → strip non-alphanumerics (including "&" and spaces) → strip a leading "the". This way nature.com's Communications Biology and PubMed's Nature reviews. Molecular cell biology both hit the same table entry.

Sort rule (rankRows):

sort((a, b) =>
  (b.journalIf - a.journalIf) ||              // ① IF descending ("impact factor first")
  b.pubDate.localeCompare(a.pubDate) ||       // ② same IF: newest publication first
  a.title.localeCompare(b.title))             // ③ stable tiebreak

Journals not in the table get IF 0 (below every known journal); cards show IF — when unknown. The numbers are for ranking and display only, not licensed data.

4.4 Content summaries (abstract fetch chain)

• Source A: use the search-page excerpt first; only when the excerpt is missing or shorter than 60 characters, fetch the article-page Abstract — locate the content after the id="Abs1-content" opening tag, cut before </section>, then stripTags and collapse whitespace; a result under 40 characters is considered invalid. Fetches are serial with requestDelayMs spacing (polite rate limiting).
• Source B: one efetch returns full abstracts for the top candidates (PubMed abstract coverage is ~100%).
• Final fallback: abstract || snippet || '（暂无简介，请点击标题查看原文）' (no summary available; click the title to read the original).
• The client truncates summaries longer than 140 characters to three lines with an "expand/collapse" toggle.

4.5 Three papers per day and caching

• Storage: a single JSON file (config.storageFile, default $DSH_HOME/storages/nature-papers.json); see Storage Format. Writes are atomic (write .tmp then rename); a corrupt file is renamed to .bak and rebuilt.
• Generation triggers:
  1. GET /today on the first request of the day (lazy generation, includes the live scrape);
  2. the daily-rollover timer (checks every 30 minutes whether the local date changed; pre-generates in the background when a new day has no data yet);
  3. the user pressing "refresh" (forced generation).
• Idempotence: within a day, /today always returns the same day's entry (cache first) — no flicker on page reload; the set rotates automatically on the next day.
• Concurrency coalescing: state.generating holds the in-flight generation promise, so concurrent requests share one generation instead of scraping repeatedly.

4.6 History and de-duplication

• After each successful generation the entry is inserted at the head of the history (dates descending); same-day entries are replaced; capped at historyCap (400 days).
• De-dup (never recommend the same paper twice): the candidate pool filters out every URL already recommended in the history; if fewer than count papers remain, relax to excluding only the last 30 days; if still insufficient, allow repeats (take the current best).
• The client "History" view groups entries by date; clicking a date expands that day's cards (TOP number, journal, IF, title link, summary).

4.7 Real-time refresh

• POST /refresh → generate(force=true): re-scrapes immediately and counts today's existing entry as already-seen too (rotates in a fresh set).
• On failure returns 502 { ok:false, error }; the client shows the error with a retry button.

4.8 The bottom-right panel (client side)

• Mounting: the shell.overlay slot (list kind, root scope), registration id nature-papers, order: 100.
• State machine: collapsed (collapses to a pill, remembered in localStorage) / view (today ↔ history) / loading / error / entry / history / expanded (summary expansion) / exitState (exit confirmation).
• Daily self-healing: the window focus event plus a 10-minute interval check whether "local date ≠ panel date" → automatically re-fetch /today.
• Styling: uses the app theme CSS variables (--dsw-alias-*) with fallbacks, adapting to light/dark mode; styles are injected via <style data-plugin="dsh-nature-papers"> and cleaned up by the HMR machinery when the plugin is removed.

4.9 Exit Harness

• Host side: POST /shutdown → respond 200 {ok:true} first (so the panel can show its final state), then after 200 ms call the launcher-injected appExit service (ctx.get('appExit')), which runs the graceful shutdown sequence: dispose the plugin tree, close the HTTP server, exit the process; falls back to process.exit(0) when appExit is unavailable.
• Client side: the footer button "⏻ 退出 Harness" → first click enters "⚠ 确认退出？" (auto-resets after 4 seconds if not confirmed) → second click POST /shutdown and shows "正在退出…".

---

HTTP API Contract

Common prefix /plugins/dsh-nature-papers; responses are { ok: boolean, data?: any, error?: string }.

Method	Path	Description	Error codes
GET	/today	Today's picks (first request of the day triggers a live scrape)	502 upstream failure; 405 wrong method
GET	/history	Full history (dates descending)	405
POST	/refresh	Force re-scrape and re-pick today's picks	502; 405
GET	/info	Runtime status: date, history days, source, last generated at, last error	405
POST	/shutdown	Gracefully shut down the server process	405

Example /today response:

{
  "ok": true,
  "data": {
    "date": "2026-08-14",
    "papers": [
      {
        "rank": 1,
        "title": "Acquired resistance to the RAS(ON) multi-selective inhibitor …",
        "url": "https://www.nature.com/articles/s41591-026-04537-w",
        "journal": "Nature Medicine",
        "journalIf": 58.7,
        "journalIfKnown": true,
        "pubDate": "2026-08-11",
        "doi": "10.1038/s41591-026-04537-w",
        "summary": "Circulating tumor DNA analyses in 44 patients …",
        "openAccess": true,
        "source": "nature"
      }
    ],
    "sourceError": null
  }
}

---

Configuration

Field	Default	Description
query	bioinformatics biochemistry	Nature search keywords (space = AND)
count	3	Papers per day (1–10)
storageFile	$DSH_HOME/storages/nature-papers.json	History storage path
requestDelayMs	1500	Scrape interval (rate limiting)
requestTimeoutMs	25000	Per-request timeout
maxCandidates	90	Candidate pool size (best N recent papers by IF)
historyCap	400	History retention in days

---

Storage Format

{
  "history": [
    {
      "date": "2026-08-14",
      "papers": [
        {
          "rank": 1,
          "title": "…",
          "url": "https://www.nature.com/articles/…",
          "journal": "Nature Medicine",
          "journalIf": 58.7,
          "journalIfKnown": true,
          "pubDate": "2026-08-11",
          "doi": "10.1038/…",
          "summary": "…",
          "openAccess": true,
          "source": "nature"
        }
      ]
    }
  ]
}

---

Rate Limiting, Fault Tolerance & Anti-bot Measures

Mechanism	Implementation
Request delay	requestDelayMs between pagination and article-page fetches
Timeout	AbortSignal.timeout(requestTimeoutMs) per request
Retry	one retry on network failure with 1.2 s backoff
Anti-bot detection	Client Challenge marker string + missing result rows (double check)
Dual-source failover	nature.com failure → PubMed; both fail → 502 with a clear error message
Request-volume control	same-day caching; article pages only when the excerpt is insufficient; PubMed abstracts fetched in one batched efetch
Storage safety	atomic writes (tmp + rename); corrupt files auto-rebuilt via .bak
Concurrency	generation promises coalesced (shared in-flight)
Lifecycle	all routes/timers registered inside ctx.effect; cleaned up automatically on unload

---

Known Limitations

• Impact factors are approximate JCR reference values (mostly the 2023 release), used only for ranking and display, not licensed data; the numbers go stale over time.
• nature.com may serve a Client Challenge to high-frequency access; the plugin then uses the PubMed mirror automatically (see 4.2) and labels the source on the panel.
• The endpoints are exposed on loopback only, with no authentication (consistent with DSH's other /plugins routes).
• Hot reload of code is limited: cordis.patch.yml is watched by watchUserPatches, but that watcher proved unreliable in practice (BOM encoding, file deletion, and chokidar exact-file watch breakage all made it stop responding), so *plugin code changes (`lib/.js) require restartingdsh web`** (see Operations below); entry-level config changes should also be followed by a restart to be safe.
• The panel is a fixed-position overlay (not draggable); it stacks with other shell.overlay registrations in registration order.

---

Operations: Install / Update / Start / Stop / Uninstall

Run the commands below from the repository root (powershell -ExecutionPolicy Bypass -File .\xxx.ps1). The scripts resolve $DSH_HOME automatically (env var, falling back to ~/.dsh); the dsh launcher is auto-detected (PATH or npx caches, pinnable via -DshBin); the port defaults to 3080 and is configurable via -Port. No machine-specific paths are hardcoded.

# Install / update (syncs code + writes the patch entry; falls back to in-place
# overwrite when the server is running and the directory is locked)
powershell -ExecutionPolicy Bypass -File .\install.ps1

# Uninstall
powershell -ExecutionPolicy Bypass -File .\install.ps1 -Uninstall

# Start / stop (a restart is required for code changes to take effect)
powershell -ExecutionPolicy Bypass -File .\stop-dsh.ps1
powershell -ExecutionPolicy Bypass -File .\start-dsh.ps1
powershell -ExecutionPolicy Bypass -File .\restart-dsh.ps1   # kill old → confirm port → start new → self-check

# The panel's "⏻ 退出 Harness" is equivalent to stop-dsh.ps1 (graceful exit)

Diagnostics: $DSH_HOME\restart-result.txt (restart self-check), $DSH_HOME\web-server.log / web-server.err.log (server logs), GET /plugins/dsh-nature-papers/info (runtime status).

Development tips: the scraping core scraper.js has no Cordis dependency, so you can validate it standalone with node test-scraper.mjs; after changing lib/*, run install.ps1 and restart.