Publishing protected data to a personal Dataverse (FUTURE)

Public datasets are NOT published here. This catalog never re-hosts open data — a public dataset is only ever linked to its origin (the Zenodo / data-Dataverse / vendor archive that published it), and get() fetches it from there. Publishing is a future capability reserved for protected data — the private and anonymized tiers — on a personal Dataverse, used as a token-gated fallback for data whose origin cannot be linked (no automatable source) or whose origin has rotted. The orchestration and CLI below exist (covered by unit tests with a mocked client) but the live end-to-end path is not exercised yet; rehearse on the sandbox first when it is enabled.

This is the end-to-end walkthrough for putting protected datasets on a personal Dataverse (Recherche Data Gouv / CIRAD) with nirs4all-datasets, and for reading them back with a token. The production steps are identical to the sandbox; only the instance + collection differ.

Sandbox (rehearse here)

Production

Instance

https://demo.recherche.data.gouv.fr

https://entrepot.recherche.data.gouv.fr

Use for

testing the full flow, throw-away datasets

the real, citable datasets

The CLI does the upload for you (create dataset → upload files tabIngest=false → publish → mint DOI). You never touch the web upload form.

0. One-time: get a token and point the tools at the sandbox

  1. Create an account on the sandbox, then copy your API token from your account page (it is per-user — it authenticates “as you”, so treat it like a password).

  2. Make it available to the CLI (resolution order: explicit arg → env → ~/.config/nirs4all-datasets/config.toml → project .env):

export NIRS4ALL_DATAVERSE_INSTANCE="https://demo.recherche.data.gouv.fr"
export NIRS4ALL_DATAVERSE_TOKEN="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
# (or:  ~/.config/nirs4all-datasets/config.toml  with  [dataverse] instance=... token=...  ; chmod 600)

  1. Have a collection alias to publish into (create one in the sandbox UI, or ask for one). The alias is the short name in the collection’s URL, e.g. nirs4all.

You can also pass --instance https://demo.recherche.data.gouv.fr on each command instead of the env var.

The token travels only in the X-Dataverse-key header, is never logged, and is never sent on a redirect to signed object storage.

1. Pick a protected dataset and set its tier

The descriptor (catalog/datasets/<id>.yaml) decides access via its tier. Publishing here is for the protected tiers only:

tier: private        # public | private | anonymized
governance:
  license: LicenseRef-not-cleared   # protected data need not carry an open license
  owner_steward: "..."
  redistribution_rights: "Private use only; redistribution not cleared."
  consent_ethics_status: "..."
  anonymization_status: "..."

  • private → everything is shown in the catalog/site, but files are uploaded access-restricted; only Dataverse accounts you grant access to can download them (with their own token). Metadata stays visible.

  • anonymized → like private, plus the served data masks variable names and z-scores numeric targets (so you can publish metrics without revealing real values/identities).

  • public is not published here — leave it linked to its origin. The governance gate (publication_blockers()) only checks the public tier, so a public descriptor must carry an open license + open origins; a private/anonymized descriptor is publishable to your personal Dataverse without those open-data requirements.

2. Validate before any network call

python catalog/scripts/validate.py --check-publish     # schema + publishability of public-tier datasets

3. First publication

n4a-datasets publish <id> --collection nirs4all --contact-email you@cirad.fr

This creates the dataset, uploads datasets/<id>/canonical/ (+ raw/) with tabIngest=false (so bytes stay pristine and SHA-256-verifiable), restricts the files because tier != public, publishes the version, and mints a DOI. On success it automatically:

  • writes the DOI + version back into catalog/datasets/<id>.yaml (dataverse.doi, dataverse.dataset_version),

  • records the Dataverse file ids + native checksums in datasets/<id>/manifest.json (this is what makes token-gated download work), and

  • rebuilds the card / Croissant / catalog.

Commit those updated files afterwards (catalog/datasets/<id>.yaml, datasets/<id>/{card.json,card.md,croissant.json,manifest.json}).

4. Verify on the sandbox

Open the dataset’s DOI landing page in the sandbox UI. Check the files, the metadata, and that the files show as restricted.

5. Read it back (this is the payoff)

import nirs4all_datasets as n4ad

ds = n4ad.get("<id>", token="...")    # private/anonymized: Dataverse access API with a permitted token
# the token is also auto-resolved from settings if you omit it for a non-public dataset

or from the shell: n4a-datasets get <id> [--token ...]. If the canonical data is still present locally, get uses it directly (no download). Downloading only succeeds if the token belongs to an account you granted access to (next section); without access, Dataverse refuses the restricted files.

A public dataset never reaches this path — get("<public-id>") fetches its bytes from the origin DOI/URL through the native acquisition core, with no token and no personal Dataverse.

6. Manage access (protected datasets)

n4a-datasets grant  <id> --to @collaborator --role fileDownloader   # let a user download
n4a-datasets grant  <id> --to "&my-group"                           # or a Dataverse group
n4a-datasets revoke <id> --to @collaborator                         # remove access
n4a-datasets restrict <id>          # make all files restricted (then republishes a minor version)
n4a-datasets restrict <id> --off    # make them downloadable again

Roles and restriction can be changed at any time after publication.

7. Update the data (new version)

Re-build the changed dataset (so datasets/<id>/canonical reflects the new bytes), then run the same publish command — it auto-detects the existing DOI and publishes a new version (the DOI stays stable; the version increments). The manifest’s checksums + dataset_version are refreshed so get keeps verifying the right bytes.

n4a-datasets build-all --source-tree <src> --only <id> --force   # refresh canonical from source
n4a-datasets publish <id>                                        # -> new version, same DOI

8. Clean up a botched sandbox attempt

If a publish failed mid-way and left a draft, delete the draft from the sandbox UI (or via the API) and retry. Sandbox datasets are disposable.

Going to production

Repeat with the production instance + a real collection:

export NIRS4ALL_DATAVERSE_INSTANCE="https://entrepot.recherche.data.gouv.fr"
n4a-datasets publish <id> --collection <prod-alias> --contact-email you@cirad.fr

Then commit the descriptor/card/manifest changes and push — the catalog site shows the DOI + version.

Notes & current limits

  • This is a future path: the Dataverse write methods (publish/update/grant/restrict) are covered by unit tests with a mocked client; the sandbox run is the first real end-to-end test — do it before production once the personal Dataverse is enabled.

  • Only protected (private / anonymized) datasets are published here. Public data is linked to its origin and never re-hosted.

  • One collection can hold several restricted datasets (access is per dataset/file).

  • A token authenticates a user; “private” = restricted files + access granted to specific accounts.