Architecture¶

The detailed reference. For the high‑level picture read ONBOARDING.md; for the rationale, see Design — reference NIRS dataset bank.

1. The lifecycle pipeline¶

A dataset flows through eight stages; each module owns one. The CLI (n4a-datasets) is a thin Typer layer that lazy‑imports the heavy deps.

NIRS DB/v2.0/<leaf>/dataset_card.json + X*/Y/M.csv
        │  bootstrap.py            DESCRIBE   card.json -> catalog/datasets/<id>.yaml (schema-2.0 descriptor)
        ▼
organize.py ─► canonical.py        CANONICAL  id-keyed CSV -> datasets/<id>/canonical/ (per-source Parquet,
        │                                     joined by sample_id), + dataset.json
        ▼
manifest.py                        MANIFEST   content-addressed manifest.json (drives incrementality)
        │
        ▼
qualify/profile.py                 QUALIFY    card.json + card.md + croissant.json + assets/*.png
        │  (registry.py metrics, plots.py, anonymize.py, croissant.py, datasheet.py)
        ▼
catalog.py                         INDEX      catalog/datasets.yaml + a whole-bank `summary`
        │
        ▼
health.py                          HEALTH     probe origins -> catalog/health.json (degrade dead public)
        │
        ▼
status.py                          STATUS     docs/DATASET_STATUS.md + docs/PRIVATE_DATASETS.md
        │                                     + catalog/validation.yaml (human review)
        ├─► site/                  SITE        the static catalog site (pure-render)
        ▼
access.py ◄── get("<id>")          USE         local-first, else fetch by DOI/origin, verify, cache -> NirsDataset
        ▲
publish.py + dataverse.py          PUBLISH     (future) personal-Dataverse governance for protected data

Bulk vs one dataset. bulk.py runs organize+qualify across a process pool (spawn, with a serial fallback if a worker dies) with per‑dataset failure isolation; discover‑style bulk authoring is bootstrap.py over the whole v2.0/ tree (idempotent, managed‑orphan prune, catalog/reconciliation.json).

2. Module map¶

Module	Role
`schema.py`	The contract: `DatasetDescriptor` + `Manifest` pydantic models, enums, validators, `publication_blockers()`.
`bootstrap.py`	v2.0 `dataset_card.json` → schema‑2.0 descriptor (modality/axis/variable‑type inference, honest governance, origin/publication routing).
`canonical.py`	id‑keyed CSV → per‑source Parquet + per‑sample `variables.parquet` + `splits/` + `dataset.json`; sample‑identity join; comma‑decimal coercion.
`manifest.py`	`processing_hash` (byte‑determining) + `metadata_hash` (displayed) + `needs_rebuild` incrementality.
`organize.py`	Place raw → call `build_canonical` → write the manifest, incrementally.
`qualify/profile.py`	Build `card.json` from the canonical Parquet (per‑source + per‑variable stats); orchestrate `qualify()` (card.json + card.md + croissant.json + assets).
`qualify/registry.py`	Extensible, protocol‑versioned metric registry (scopes: source/variable/dataset).
`qualify/metrics.py` `plots.py`	Pure numerics; per‑source + per‑variable plots (Agg).
`qualify/anonymize.py`	The anonymized‑tier transform + the `public_card` / `public_descriptor` chokepoints.
`qualify/croissant.py` `datasheet.py`	MLCommons Croissant JSON‑LD + Datasheets‑for‑Datasets `card.md`.
`dataset.py`	`NirsDataset` — the consumer reader (`x`/`y`/`metadata`/`split`/`to_nirs4all`), tier‑masking on read.
`access.py`	`get()` — local‑first, else fetch by Dataverse DOI (token) / open origin; SHA‑256 verify; cache.
`catalog.py`	The index entry + the whole‑bank `bank_summary`; staleness, not lies.
`health.py`	Origin liveness probe (injectable session) → `catalog/health.json`.
`status.py`	Per‑dataset status (state/origin/validation/distribution) + the reports + the validation registry.
`site/`	The static‑site generator (pure‑render: pyyaml + stdlib; tier‑gating in `model.py`).
`bulk.py`	Parallel organize+qualify with failure isolation.
`cli.py`	The `n4a-datasets` Typer CLI.
`config.py` `dataverse.py` `publish.py`	Token hygiene; the Dataverse REST client; the publish/governance flow.

3. Schema 2.0 (the descriptor)¶

catalog/datasets/<id>.yaml → DatasetDescriptor:

sources: list[Source] (≥1) — source_id, instrument, modality, axis unit/range, n_observations, n_variables.
variables: list[Variable] (may be empty) — name, role (target|metadata), type (numeric|categorical|text|identifier|datetime), unit, classes. .targets / .metadata_variables are properties. No task_type — the task is a consumer concern.
ids: IdentitySpec — observation_id (per spectrum), sample_id (physical sample), sample_id_available.
alignment_level — observation | sample. splits: list[SplitRef] (applied=False).
tier — public | private | anonymized. versions: Versions — content + schema_protocol.
governance (license + open‑data fields; no visibility/confidentiality), provenance, origin_sources: list[OriginSource] (where the bytes live — never checksums), publications, datacite, dataverse, reproducibility, generation (managed‑descriptor provenance).

Two‑level validation (schema.py + catalog/scripts/validate.py): schema validity (every field well‑formed) is separate from publishability (publication_blockers(), which gates only the public tier: open license + open non‑SCRIPT origins + the responsible‑release fields).

4. Canonical on‑disk layout¶

datasets/<id>/canonical/
  dataset.json                 {format_version, id, join_key:"sample_id", alignment_level,
                                sources:[{source_id,path,n_observations,n_variables,axis_*}], variables, splits}
  sources/<source_id>.parquet  observation_id (str), sample_id (str), <wavelength cols> (float32)
  variables.parquet            sample_id (str), <all non-spectral Y/M cols> (native dtype)   [optional]
  splits/<name>.parquet        sample_id (str), partition (str)                              [optional]

Sources may have different row counts (asymmetric repetitions). Everything is joined by sample_id, never by row position. variables.parquet is per‑sample (one row per sample_id); the standardization script’s source_* provenance‑plumbing columns are excluded. Parquet is chosen because Dataverse does not auto‑ingest it, so uploaded bytes stay byte‑identical to local ones (the SHA‑256 verify depends on it).

5. Tiers and anonymization (load‑bearing)¶

public shows everything and is openly fetchable; private shows metadata + metrics but export needs a token; anonymized masks variable names (var_NNN), z‑scores numeric targets, and removes identifying free text. The anonymized tier is enforced automatically by tier through one chokepoint — qualify.anonymize.public_card / public_descriptor:

qualify() writes card.json itself already‑anonymized for the anonymized tier (and renders card.md/croissant.json from the masked view);
the catalog entry, NirsDataset.descriptor/variables()/card(), the site, and Dataverse publish metadata all derive their displayed fields from the public card/descriptor.

So no tracked artifact, index entry, public API, or published metadata can leak an anonymized identity. (Reviewed by Codex: GO. Regression test: tests/test_anon_enforcement.py.)

6. Incrementality + the two axes¶

processing_hash covers only byte‑determining descriptor fields (sources, ids, alignment, splits, versions.content); editing a name/tier/variable‑role/origin or bumping versions.schema_protocol never rebuilds canonical bytes.
metadata_hash covers the displayed content and drives card re‑render (card_metadata_fresh).
needs_rebuild compares raw‑file hashes + the converter identity + processing_hash against the previous manifest. A content version bumps on a byte change; a metric‑protocol version bumps to re‑qualify cards (build-all --protocol-refresh) without rebuilding data.

7. Status + validation¶

status.py derives, per dataset: state (described→canonical→qualified=metrics computed), materialized (canonical + SHA‑256), origin (reachability from the health probe), distribution (open / on_dataverse / upload_pending). Validation is the human axis, recorded in catalog/validation.yaml (pending→reviewed→approved), which bootstrap never touches. n4a-datasets status refreshes the registry and writes docs/DATASET_STATUS.md + docs/PRIVATE_DATASETS.md.

8. Access model (and the PyPI caveat)¶

get(name, *, root=".", source, split, token, …) resolves local‑first (<root>/datasets/<id>/ canonical + <root>/catalog/datasets/<id>.yaml), else fetches by the descriptor’s Dataverse DOI (token for private/anonymized) or an open origin, verifies SHA‑256, and caches. The token travels only in the X-Dataverse-key header and is never sent on an S3 redirect.

PyPI note. The wheel ships the code and the bundled cross-language catalog/index.json. A pip‑installed Python consumer still points get(root=<checkout>) at a clone of this repo for the high-level get()/list()/card() surface, because that layer reads descriptors/cards and returns NirsDataset. Non-Python bindings consume catalog/index.json directly: n4ds_resolve returns the byte contract plus the tier-sanitized descriptor, so R/WASM/Rust can inspect sources/variables and read verified Parquet without the Python provider package. See RELEASING.md.

9. Conventions¶

Python 3.11+, Google docstrings, ruff (line length 220) + mypy, type hints on public APIs, py.typed. nirs4all is an optional extra ([nirs4all]) — imported lazily, degrading gracefully when absent. Each green gate: ruff check . + mypy --config-file pyproject.toml src + validate.py (+ --check-publish)

pytest. The enum mirror (AxisUnit/SignalType/Modality mirror nirs4all by value) is guarded by tests/test_schema.py.