nirs4all-datasets¶

A citable, reproducible bank of raw NIRS (Near-Infrared Spectroscopy) reference datasets — for benchmarking, exploring, and comparing models on a common, version-pinned, provenance-rich footing.

A dataset here is raw measured reality, not a benchmark task: one or more spectral sources (instruments), any number of variables (every target and metadata column), the native splits if the origin defined them, and full provenance back to the origin that published the data. The task — which Y, which split, which metric — is a choice the consumer makes; it is never baked into the dataset.

The heavy bytes never enter git and are never re-hosted. The catalog links to each dataset’s origin (Zenodo / a data Dataverse / a vendor archive) and downloads on demand: get("name") resolves a dataset local-first (from the registry checkout), otherwise fetches it by its Dataverse / open-canonical DOI, SHA-256-verifies it, caches it through the native acquisition core, and returns a NirsDataset.

Python high-level API uses the catalog checkout

get() / list() read the git-tracked catalog under a registry root (root=, default .): list() reads catalog/datasets.yaml, and get() needs the dataset’s local descriptor + manifest before it can fetch. Non-Python clients use the bundled or committed catalog/index.json instead: n4ds_resolve returns the byte contract and tier-sanitized descriptor, then R/WASM/Rust read the verified Parquet with their host-native readers. See Getting started.

Boundary rule

For NIRS modelling objects and instrument-file reads, nirs4all-datasets delegates and re-implements nothing: it reuses nirs4all for PCA projection, outlier filtering, and the SpectroDataset handed back by to_nirs4all(), and nirs4all-io / nirs4all-formats for reading instrument files. It owns only catalog-level concerns — descriptors, the canonical Parquet writer, the descriptive card statistics nirs4all does not expose, Croissant/Datasheet rendering, the index, origin health, and the site.

Status

0.3.x, pre-1.0 — the on-disk and API contracts may still change.

What you get¶

Three deliverables sit behind one catalog:

a git-tracked catalog — one hand-checkable descriptor plus a machine-generated identity card (stats, per-source / per-variable dataviz, an MLCommons Croissant record, and a Datasheet) per dataset. The heavy bytes never enter git.
a native acquisition contract — catalog/index.json plus the Rust/C/WASM/R bindings that resolve a dataset id to descriptor + SHA-256-pinned files without importing Python providers.
an optional Python package/binding — nirs4all_datasets.get("name") wraps the native acquisition core and returns a NirsDataset from a registry checkout;
a static site — a browsable, qualified catalog with whole-bank dataviz and per-dataset identity cards.