dsh-data-quality

• 1024 store channel: npm i -g dsh1024 once, then dsh1024 plugin --profile web add dsh-data-quality (counts toward the deepseek1024.com install ranking).

Deterministic data profiling, cleaning, and verification for DeepSeek Harness.

All computation is plain TypeScript in the harness process — the model never does the math. A ctx.dataQuality capability seam (Service Definition / local Provider / tool Consumers) exposes three model tools plus a frozen cross-plugin citation-checking contract.

English · 简体中文 · Español · Português · हिन्दी

📖 Ecosystem knowledge base — measured data, not marketing: plugin development guide · plugin-selection data · maintenance criteria.

⭐ 如果它帮到了你

这个插件是 DSH 插件家族的一员（40+ 个，全部 Apache-2.0）。如果你在用，给个 star —— 它不会解锁任何功能，但会让下一个人在搜索里更容易找到它。

English: part of a 40+ plugin family for DeepSeek Harness. If it is useful, a star helps the next person find it — nothing is gated behind it.

Compatibility

What you get

• ctx.dataQuality service — a Cordis service other plugins may optionally consume (inject = ['dataQuality']). Besides the three dataset operations behind the tools, it implements the frozen verifyCitations(request) contract: verify that numbers/strings cited in a document match a dataset snapshot, with relative-tolerance numeric comparison and verified / mismatch / not-found / unverifiable statuses.
• data_profile tool — dataset profiling: row/column counts, inferred column types (number/date/boolean/string/empty/mixed), missing rates, unique counts, numeric distributions (min/max/mean/median/p25/p75), IQR outlier counts, mixed-type suspicion notes, and full-table sha256 content-hash duplicate detection with the duplicate rate and a bounded sample of duplicate row indexes. Adds a deterministic DAMA six-dimension scorecard (completeness, uniqueness, validity, consistency, timeliness, accuracy — accuracy is reported undetermined without a declared schema, never fabricated). Optional deterministic systematic sampling for large files.
• data_clean tool — ordered declarative cleaning rules: dedupe (by column group), fill-missing (constant/mean/median/forward), coerce-type (number/date/boolean; failures counted and set to missing), normalize-unit (e.g. 万/亿 suffixes to base units), trim, map-values (enum mapping). Returns a per-rule audit log, a pre-delivery contract validation summary (dedupe before/after, uniqueness, non-null and type regressions), and a bounded preview; writes the cleaned dataset only when outputPath is given, and never overwrites the source.
• data_verify tool — declarative verification rules: not-null, unique, range, regex, enum, cross-column (e.g. startDate < endDate), freshness (date column within N days of a reference date). Per-rule pass/fail with capped failing-row evidence; an overall failure is a normal passed: false result, not a tool error.
• data_report tool — read persisted reports back from the data_quality storage domain: by exact reportKey (path-safe validation, missing keys fail loud) or by kind (chronological listing). Returns the report envelope(s) — kind, dataset, timestamp, and the full stored report. format: html renders one profile/clean report as a self-contained offline HTML document (inline CSS/JS, no external requests) with the DAMA six-dimension scorecard and the profile/cleaning summary tables.
• Durable reports — every profile/clean/verify/citation run persists to the data_quality storage domain (JSON backend), keyed by run timestamp plus a dataset-path fingerprint; the key is returned as reportKey in tool results. Clean reports also persist the bounded preview and the contract summary, so every model-visible result is reconstructable from its reportKey; each clean run additionally persists a clean-diff before/after profile report.
• Session events — on hosts that can carry them safely, runs append data-quality/profile / data-quality/clean / data-quality/verify events (with the ignorable marker where supported). On the published 0.1.7-rc.2 line (as on earlier rc lines) the append is skipped by design — the storage-domain report is always the durable copy (see "Known limitations").

Quick start

npm channel

dsh plugin --profile web add dsh-data-quality

Tarball channel (no build permission needed)

pnpm pack                                  # produces dsh-data-quality-<version>.tgz
dsh plugin --profile web add ./dsh-data-quality-<version>.tgz

Git channel

dsh plugin --profile web add github:YOUR_ORG/dsh-data-quality#<commit-sha>

The first add fails because pnpm blocks the package's prepare build; copy the exact key pnpm printed into the profile's pnpm-workspace.yaml and re-run:

allowBuilds:
  'dsh-data-quality': true

Restart the profile after installing (bundles activate on restart). Then ask the agent, in a workspace containing a CSV:

Profile holdings.csv, then clean it by trimming whitespace, deduplicating on fund_code, and normalizing the holding_value column's 万/亿 units; finally verify fund_code is unique and not null.

Install & uninstall

dsh plugin --profile web add dsh-data-quality      # install (npm) — or the forms above
dsh plugin --profile web remove dsh-data-quality   # uninstall

Configuration

All keys are optional (defaults shown); invalid values fail loudly at load. Every key is settable from cordis.yml (the bundle ships cordis.patch.yml with the same defaults).

Tools & surfaces

data_profile({ path, sample?, industryPreset? })

Profiles a workspace dataset. path is workspace-relative (.csv/.tsv/.json/.jsonl; JSON must be an array of flat objects). sample takes every ceil(N/sample)-th row for the column cards (deterministic; row counts stay exact). industryPreset (retail/saas/fund/real-estate/e-commerce/healthcare/logistics/manufacturing/energy) injects that industry's expected columns so the scorecard accuracy dimension becomes determinable; unknown ids fail loud. Returns the structured report — the duplicate rate, bounded duplicate-row indexes, numeric count/distinct distributions, file encoding (UTF-8 BOM + validity), and the weighted six-dimension DAMA scorecard — and renders a human-readable per-column summary plus scorecard.

data_clean({ path, rules, outputPath?, dryRun? })

Applies rules in array order, each seeing the previous rule's output. Rule reference:

The source file is never overwritten. With outputPath the cleaned dataset is written there (workspace-confined, format by extension); without it the run is preview-only. With dryRun: true no file is written and nothing is persisted — the result returns the per-column cleaning plan and the expected contract/diffPreview. The result also carries a pre-delivery contract summary (dedupe before/after rows, uniqueness over the dedupe key or full rows, non-null regression for fill-missing columns, type regression for coerce-type columns, and the per-column decision trace), and a clean-diff before/after profile report is persisted to the storage domain.

data_report({ key?, kind?, format? })

Reads persisted reports back from the storage domain. Pass key (the exact reportKey a prior run returned) to fetch one report, or kind (profile/clean/clean-diff/verify/citations) to list every report of that kind chronologically; exactly one of key/kind is required. Malformed or missing keys fail loud. format: html (with key) renders the report as a self-contained offline HTML document — inline CSS/JS, no external requests, the DAMA six-dimension scorecard, and the profile/cleaning summary tables (profile/clean reports only).

data_verify({ path, rules, expectations? })

Evaluates verification rules. Rule reference:

expectations reconciles deterministic metrics against expected values: rowCount, columnSum, columnMean, uniqueCount, nullCount (each with column except rowCount, plus expected and an optional relative tolerance in [0, 1]). Each expectation yields passed plus actual/expected/tolerance; a mismatch is a normal passed: false verdict, never a tool error. Invalid metrics, missing columns, and out-of-range tolerances fail loud.

A missing cell fails every rule that reads it. Evidence is capped at evidenceRowLimit failing rows per rule.

ctx.dataQuality (for other plugins)

const result = await ctx.dataQuality.verifyCitations({
  dataset: 'holdings.csv',          // resolved against workspaceRoot
  citations: [
    { id: 'c1', path: 'rows[3].nav', value: 1.234, tolerance: 0.01 },
    { id: 'c2', path: 'summary.annualReturn', value: '12.34%' },
  ],
})
// result.results[i] = { id, status: 'verified' | 'mismatch' | 'not-found' | 'unverifiable', actual?, note? }

Locators walk the dataset document: CSV/TSV load as { columns, rows } (so rows[3].nav resolves), JSON is the parsed value, JSONL the array of parsed lines. Numbers compare with relative tolerance (|a-b| <= tolerance * max(|a|, |b|)); a CSV string cell that parses numerically compares as a number; strings compare exactly; incomparable type pairs are unverifiable. The service also exposes profileDataset / cleanDataset / verifyDataset (the same operations the tools call).

Permissions & data

• Reads workspace dataset files (allowlisted extensions only).
• Writes only: the data_clean output file (explicit outputPath, workspace-confined, never the input) and reports in the data_quality storage domain under the harness data directory.
• No network, no credentials, no external processes — all parsing and statistics are in-process TypeScript.
• Reports may contain sample cell values from your datasets (bounded by evidenceRowLimit and display truncation); the session log records tool arguments and results as usual.

Security boundaries

• Path confinement — dataset and output paths must resolve inside the session workspace (verifyCitations uses workspaceRoot); .. escapes and outside absolute paths reject, and both sides are normalized before comparison (Windows slash-safe).
• Bounded work — maxRows / maxFileSizeMB guards reject oversized inputs loudly; abort signals cancel long loads mid-stream.
• No overwrite — data_clean refuses an outputPath equal to the input path.
• Deterministic computation — same input, same output; the only clock is the one injected for freshness defaults and report timestamps.

Known limitations

• Session events are adaptive. The published 0.1.7-rc.2 line (like the earlier rc lines) has no plugin session-event registration surface and its Session.append cannot stamp the ignorable marker, so appending an unknown data-quality/* type would make the session log unreadable on restore. The plugin therefore appends only when the host knows the vocabulary or supports the ignorable append flag; on the published line the storage-domain report is the durable record.
• CSV dialect — comma/tab with RFC-4180 quoting, header row required, blank lines skipped, no delimiter auto-detection or comment lines.
• Type parsing is strict — numbers have no thousands separators; dates are YYYY-MM-DD / YYYY/MM/DD / ISO-like datetimes (UTC); booleans are true/false/yes/no/1/0. Everything else profiles as string/mixed — clean it with coerce-type when intended.
• JSON must be tabular for the tools (array of flat objects); verifyCitations walks arbitrary JSON documents.
• No ML anomaly detection, no PII masking, no databases, no SQL — rule-based suspicion notes only.

Development

pnpm install
pnpm run typecheck && pnpm run typecheck:ci && pnpm run typecheck:checkout && pnpm test && pnpm run build
pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:readme-sync && pnpm pack

• Tests run vitest against the REAL Context/Session/ToolRuntime/storage domain from the 0.1.7-rc.2 peers (no hand-written service mocks) plus pure engine specs; every clean/verify rule has positive and negative cases, and verifyCitations covers all four statuses.
• scripts/loader-runner.mjs boots the real Loader composition and executes the profile → clean → verify chain against fixtures/ without an API key.
• Release: node scripts/release.mjs <x.y.z> (never pushes; the tag triggers release.yml).

Topics

dsh · dsh-plugin · deepseek-harness · cordis · data-quality · data-cleaning · data-profiling · data-verification

Contributors

Thanks to everyone who has shaped this plugin.

• PerryLink — maintenance and releases (0.1.2/0.1.3), peer-dependency upgrades, the npm version/downloads/CI badges, and recent fixes.
• dsh-data-quality contributors — the initial scaffold, the ctx.dataQuality seam and frozen verifyCitations contract, the deterministic dataset layer and pure engines, the data_quality storage-domain reports, the real-service vitest suite, the CI/compat/release workflows, and the five-language READMEs.

This repository has no public issue or pull request history yet; individual PR/issue numbers will be credited here as they arrive.

PerryLink DSH Plugin Family

This project is one of the 44 DeepSeek Harness plugins maintained by PerryLink. If this one helps you, the others likely will too: