Design — reference NIRS dataset bank¶

v1 — rephrased from your annotations. Reference document for rethinking the project before the full analysis and implementation. Annotate / correct freely.

1. Vision¶

A versioned and qualified database of raw NIRS datasets, cataloged and linked to its sources (we never republish other people’s open data; we point to the origin). Three deliverables:

a catalog (descriptors + identity cards + index) — lightweight and tracked in git;
a website — navigation, identity cards, dataviz;
a native acquisition core — Rust crates + C ABI + bindings that retrieve verified dataset bytes without requiring Python;
an optional Python package/binding — import … / ….get("name") to retrieve a dataset (or its metadata) locally, plus bridges into the rest of the ecosystem.

The bytes live at their source; git only carries metadata + cards + index. Everything is hashed and versioned. An add/qualification pipeline grows and evolves the bank.

Ultimate goal: make it possible to cite, benchmark, compare, and explore on a serious, reproducible base, with datasets ranging from X only to X + Y to X + Y + metadata.

2. A dataset = measured reality (raw, first-class)¶

A dataset is not a benchmark task (a Y+split choice). It is raw data, kept as raw as possible. It contains:

Element	Rule
X — 1..n sources	One source = one instrument / acquisition, kept separate (never merged or resampled), with its own native axis (nm / cm^-1 / um / …). A dataset can be multi-source (several X blocks), and sources can have different sizes — see the note below. (`nirs4all-formats` / `nirs4all-io` added as needed to read vendor formats and run metrics.)
Variables (Y + metadata)	No intrinsic difference. If the source declares targets, mark them. Otherwise any column is a potential Y. Multi-target, mixed types, never split apart. If no Y is declared, do not invent one; document metadata and metrics.
Sample ID	`observation_id`, etc. — index, never a Y.
Metadata	All preserved, with no prior filtering or processing for now.
Splits / folds	None by default. Kept only if the source defines them (train/test, folds, even several versions), and then documented.

Coverage: X only · X + Y · X + Y + metadata. The Python package can return data with or without a split (concatenated by default).

Multi-sources & repetitions (structural). Like Y, X can be multiple (multi-source dataset: several instruments / acquisitions). Each source is managed independently: no fusion, no resampling, no imposed common grid. Most importantly, because of asymmetric spectral repetitions (a sample scanned a variable number of times depending on the source), sources can have different numbers of spectra; they are not row-aligned. Implications:

each source carries its own (n_spectra x n_wavelengths) dimension and its own repetition indexing;
alignment between sources, and with Y / metadata, is done by sample identity (ID), never by row position;
the pipeline, card (stats per source), and Python package preserve this structure (never flatten or force alignment); the package returns sources separately and can, on request, concatenate repetitions or return only one source.

3. Governance & visibility — 3 tiers¶

We catalog everything. What varies is what we show and what we export. The right token (in the consumer binding/package) unlocks full access.

Tier	Website — metadata & metrics	Byte export (consumer binding/package)
public	everything, named	yes, for everyone (from the origin)
private	everything, named	token required (private Dataverse)
anonymized	unnamed metadata + normalized Y (metrics on anonymized data)	token required

Datasets whose source cannot be automated go into a private Dataverse, retrievable with a token (no open publication from us).
Anonymization (the most protected tier) masks variable names and normalizes Y: metrics are published without revealing real values/identities.

4. Architecture¶

  origin sources (DOI / URL / private Dataverse)
                    |  (tested regularly; if an origin goes down -> switch to private Dataverse)
                    v
  +-------------------------------------------------------------+
  |  add / qualification PIPELINE (re-runnable script)           |
  |  raw ingest (nirs4all-formats/io)  ->  metrics  ->  card     |
  +-------------------------------------------------------------+
                    | writes / updates
                    v
        git: descriptors + identity cards + index        (bytes never in git)
                    |
        +-----------+---------------------------+
        v                                       v
   website (cards + dataviz, by tier)      optional Python package  ....get("name")
                                           (local download; token = full access; with/without split)

Catalog (git) — one descriptor + one card per dataset, plus the index. Lightweight source of truth.
Add/qualification pipeline (a script) — (1) records a new raw dataset, (2) computes metrics -> identity card, (3) updates the website + acquisition/catalog files. Re-runnable: when the metric protocol evolves, run it again to update old cards without touching the data.
Website — navigable catalog, cards, dataviz on X, Y, and metadata, respecting tiers.
Acquisition core + bindings (this repo) — the Rust core resolves/fetches/verifies bytes; the optional Python package adds get()/NirsDataset, metadata/card access, and the bridges to nirs4all / nirs4all-io.

5. Provenance, integrity, versions, evolution¶

Provenance — origin source(s) (DOI/URL + mode: open / token / manual / script); related publications (papers) referenced but kept distinct from data sources.
Integrity — hash chain origin -> raw -> canonical -> card.
Versions (2 proposed axes) — (a) dataset content version: bump when bytes change; (b) metric protocol version: bump when metrics are enriched -> re-qualification without content changes. (to validate)
Origin resilience — origins tested regularly; if an origin goes down, the dataset switches to the private Dataverse.
Evolution — datasets come from a OneDrive; pulls will evolve, chantiers/ + unusable/ will migrate into the existing structure later, and version iterations are expected. Therefore the pipeline (§4) must provide a robust add mechanism + hashes/versions designed for that evolution.

6. Identity card (diagnostics)¶

Per dataset, as complete as possible, and extensible (a final protocol + metric list will arrive; we must be able to add them and re-qualify existing datasets):

spectral + PCA / dimensionality + quality by X block;
stats by Y variable (distribution / class balance) — multi-target;
dataviz on X, Y, and metadata;
train<->test shift if a split exists;
hashes + citation + provenance.

7. Migration from the current state (`NIRS DB/`, 19 GB)¶

Source	Decision
`v2.0/` (164) — machine-readable `dataset_card.json` cards	canonical: the bank becomes these datasets.
`regression/ classification/ multimachines/` (484 v1 sheets)	deleted (they were tasks, not datasets).
`Publications/`	keep + organize those referenced by a dataset; ignore the rest.
`chantiers/`, `unusableDB/`	ignored for now — they will migrate into the existing structure later (anticipate this).

Code consequences: remove the v1 path (discover.find_leaves / build_descriptor) because it becomes dead; the card becomes multi-Y; add the 3 tiers + get() package/binding + add pipeline + origin health-check. Guardrail: list any v1 datasets with no v2.0 equivalent before deletion (no silent loss).

8. Still-open points (next iteration)¶

Consumer API — should get() default to concatenated (no split) or partitioned data? How should multiple splits be handled? What should metadata / card accessors be named? Maintainer note: if splits are available, users must be able to download split data, so these should be options; choose natural names for metadata and metric functions.
Anonymization — define “normalized Y” (z-score? min-max? rank?) and “unnamed metadata” (drop names -> col_0...?). Maintainer note: choose what feels natural in ML or spectroscopy.
Token — one token for all private data, or per dataset? Reuse the Dataverse token. Maintainer note: start from the existing Dataverse behavior once Dataverse is up; use a Dataverse token.
Versions — validate the 2 axes content / metric protocol (§5). Maintainer note: the protocol and metric/metadata schema will evolve, including repetitions, aggregations, etc.; start with these axes.
Default Y designation — when no Y is declared, what rule chooses the metadata column (and how is it documented)? Maintainer note: no rule; do not set Y. Show metadata and provide metadata dataviz like Y dataviz (histograms, etc.).