DataLad - Neuroanalytics Datasets

Git versions your code. It does not version your data — a 40 GB derivatives directory will not go in a git repository, and the usual workaround is a folder called analysis_v3_FINAL_actually_final.

DataLad closes that gap. It layers git-annex underneath git so that large files are tracked by content hash while their actual bytes live elsewhere, and it records how every output file was produced — the command, its inputs, and the software environment. A DataLad dataset is a git repository, so everything you already know about branches, commits and history still applies.

Why it matters here¶

Most projects in this program are secondary analyses: take an open dataset, run a pipeline over it, analyze the derivatives. Six months later a reviewer asks which fMRIPrep version produced a figure. Without provenance that is archaeology. With datalad run, it is one command.

It also solves a problem specific to this field: you can clone a dataset and get the full directory structure and history without downloading the data, then fetch only the subjects you need. On a cluster with a storage quota, this matters a great deal.

Core ideas¶

Datasets are nestable. A study dataset can contain your raw BIDS data as one subdataset and each pipeline’s derivatives as another, each with independent history. This is how Nipoppy organizes things.

Files can be present or absent. datalad get fetches content on demand; datalad drop removes local content while keeping the file’s identity and history. The file never stops existing as far as git is concerned.

Provenance is captured at execution time, not documented afterwards. This is the part that changes day-to-day habits, and the part worth actually adopting.

The commands that matter¶

Roughly in the order you will meet them:

# Create a dataset using the YODA layout (see below)
datalad create -c yoda my-study

# Clone a dataset - fast, because file content is not downloaded
datalad clone https://github.com/OpenNeuroDatasets/ds000114.git
datalad get sub-01/            # fetch only what you need
datalad drop sub-01/           # free the space again, keep the history

# Record changes to code and configuration.
# Inside a DataLad dataset this replaces `git add` + `git commit`,
# because it handles annexed and git-tracked files correctly.
datalad save -m "Add motion regression script"

# Run a command with full provenance
datalad run -m "Extract confounds" \
  --input "derivatives/fmriprep/sub-01/**" \
  --output "results/sub-01_confounds.tsv" \
  "python scripts/extract_confounds.py sub-01"

# Same, but records the container image too, so the whole
# software environment is reproducible
datalad containers-run -n fmriprep-2440 --input ... --output ... "..."

# Re-execute a recorded command from its provenance record
datalad rerun <commit>

datalad run commits outputs only if the command succeeds. On failure nothing is recorded, so your history does not fill with half-finished states.

YODA¶

-c yoda applies the YODA principles — a convention for laying out an analysis dataset so that it is self-contained and reusable:

Code lives in code/, and is version-controlled in git rather than annexed
Input data is a subdataset, never copied in — so its own provenance is preserved
Outputs are generated by recorded runs, not by hand
The dataset is structured so someone can clone it and re-execute it

The practical consequence is that “the analysis” becomes a thing you can hand to someone whole, rather than a directory plus a set of verbal instructions.

Fitting it into the workflow¶

Organize your data — a DataLad dataset is a good container for a BIDS study
Preprocess — run pipelines through datalad containers-run so the container version is recorded alongside the command
Share your work — datalad push to a sibling on a cluster, GitHub, or an archive

Where it is already in use¶

CONP distributes datasets as DataLad datasets — cloning one is how you get CONP data
brainlife.io uses it for provenance
Nipoppy builds on DataLad conventions for dataset organization

Getting started¶

The DataLad Handbook is genuinely good and is the right place to start — it is written as a tutorial rather than a reference.

Two pieces of advice from people who have adopted it:

Start a new project with it rather than converting an old one. Retrofitting provenance onto an analysis that already exists gives you the file layout but not the history, which is most of the value.
Learn save, run, and get first. Subdatasets, siblings and special remotes are powerful and can wait until you have a reason to need them.