Dataset catalog

The catalog is the heart of nirs4all-datasets: a git-tracked set of descriptors and machine-generated identity cards, one per curated dataset. Every entry is version-pinned and provenance-rich, and its heavy bytes are downloaded on demand from the dataset’s origin — never re-hosted here.

How a dataset is described

Each dataset carries:

  • Sources (X), 1..n — one spectral block per instrument, kept separate, with its own axis unit and range. Multi-instrument datasets keep each block as its own source; sources are aligned by sample identity, never by row position.

  • Variables (Y + metadata), 0..n — every column is a potential target. A target is flagged only when the origin declares one; X-only / metadata-only datasets are valid, and no target is ever invented.

  • Splits — native train/test/fold partitions are recorded so you can reproduce a paper’s split, but they are documented, never auto-applied.

  • Tier — public (shown and openly fetchable from the origin), private (shown; export needs a token), or anonymized (variable names masked + numeric targets normalized; export needs a token).

  • Versions — a content version (bumps when the bytes change) and a metric-protocol version (lets cards be re-qualified under a new protocol without rebuilding the data).

Each generated card bundles descriptive statistics, per-source / per-variable dataviz, an MLCommons Croissant JSON-LD record, and a Datasheet (card.md).

Browse what is available

The current, generated overview of every catalogued dataset — its state, whether it is materialized, its origin, distribution, and human-validation status — lives in the Dataset status page. The Datasets pending a private Dataverse upload page lists the small subset that cannot be retrieved from an open origin and must be uploaded to the private Dataverse before users can fetch bytes with a token.

From your own checkout you can list and inspect the catalog directly:

n4a-datasets list                 # every catalogued dataset
n4a-datasets list --tier public   # filter by tier / domain / spectro-family
n4a-datasets card <id>            # one dataset's identity card

or from Python:

import nirs4all_datasets as n4ad

n4ad.list(tier="public")
n4ad.card("corn_eigenvector_nir")

See Getting started for downloading and loading a dataset.