Releasing

Three independent release flows: the package (PyPI), the datasets (Dataverse, for protected data), and the human validation sign‑off.

1. The package → PyPI

The package builds cleanly and passes twine check (sdist + platform-specific wheels — maturin bundles the native nirs4all_datasets._n4ds pyo3 extension, so wheels are per-platform, not py3-none-any). It depends only on published packages — nirs4all is an optional [nirs4all] extra, so pip install nirs4all-datasets resolves without the unpublished siblings.

One‑time setup (no API token — Trusted Publishing / OIDC)

  1. Reserve the name / first upload. Either create the project on PyPI by doing one manual upload from a maintainer machine:

    python -m build && python -m twine check dist/*
python -m twine upload dist/*          # asks for your PyPI token once

    …or create an empty project + a pending Trusted Publisher (PyPI now supports project‑less pending publishers).

  2. Add the Trusted Publisher on PyPI → Your project → Publishing → Add a GitHub publisher:

    • Owner GBeurier, repository nirs4all-datasets, workflow release.yml, environment pypi.

  3. Create the GitHub environment pypi (repo → Settings → Environments) — optionally require manual approval so a tag does not publish unattended.

Cut a release

# 1. bump the version
#    pyproject.toml:  version = "0.2.0"        (drop the .devN suffix for a real release)
# 2. commit + tag
git commit -am "release: v0.2.0"
git tag v0.2.0
git push origin main --tags

The tag triggers .github/workflows/release.yml: it builds, runs twine check, and publishes to PyPI via OIDC (gated on the pypi environment). Verify with pip install nirs4all-datasets in a clean venv.

Caveat — runtime catalog. The wheel ships the code and the bundled cross-language catalog/index.json, but the assembled catalog (catalog/datasets.yaml), descriptors, cards, and manifests still live in the registry checkout. A pip-installed Python consumer therefore still points get(root=<checkout>) at a clone of this repo for the high-level get()/list()/card() surface. Non-Python bindings can consume the bundled or committed catalog/index.json directly.

2. The datasets → Dataverse (protected data)

Public datasets are not published here — they are linked to their origin. Private / anonymized datasets can later be uploaded to a personal Dataverse so consumers can get(token=…) them. The list of datasets awaiting upload is generated locally by n4a-datasets status (a maintainer-only worklist, not published here).

  • Token: NIRS4ALL_DATAVERSE_TOKEN env var, or ~/.config/nirs4all-datasets/config.toml (chmod 600), or a project .env. The token travels only in X-Dataverse-key, is never logged, and is never sent on an S3 redirect. Prefer the sandbox (demo.recherche.data.gouv.fr) before production.

  • Locally:

    n4a-datasets publish <id> --collection <alias> --contact-email you@example.org   # mints a DOI (first time)
n4a-datasets restrict <id>            # access-gate all files, publish a minor version
n4a-datasets grant <id> --to @user    # let a user download restricted files

  • In CI: the manual publish.yml workflow (workflow_dispatch, gated on the protected dataverse-publish environment; token = the DATAVERSE_TOKEN secret).

  • The governance gate (validate.py --check-publish) refuses a public dataset that is not openly licensed/sourced. Full walkthrough: PUBLISHING.md.

On success the minted DOI is written back into catalog/datasets/<id>.yaml, so the dataset moves from upload_pending to on_dataverse in the status reports automatically.

3. Human validation sign‑off

Validation is tracked in catalog/validation.yaml (never touched by bootstrap). To review a dataset, edit its record and bump validation pending → reviewed → approved (add reviewed_by / reviewed_at / notes), then refresh the reports:

n4a-datasets status --root .     # rewrites docs/DATASET_STATUS.md + docs/PRIVATE_DATASETS.md
git add catalog/validation.yaml docs/DATASET_STATUS.md docs/PRIVATE_DATASETS.md && git commit

Release checklist

  • Green gate: ruff check . · mypy --config-file pyproject.toml src · validate.py (+ --check-publish) · pytest -q

  • catalog, health-check, status re‑run and committed (the index/health/status reports are up to date)

  • version bumped in pyproject.toml; python -m build && twine check dist/* clean

  • tag vX.Y.Z pushed (PyPI) / publish.yml dispatched per dataset (Dataverse)