Leakage is when information from your test set reaches your model during training. The result is an accuracy figure that is real in the sense that you computed it correctly, and worthless in the sense that it will not survive contact with new data. It is the single most common reason a promising neuroimaging classifier fails to replicate, and it is usually invisible — the code runs, the numbers look good, nothing errors.
Three forms account for most cases in this field:
Subject leakage. The same participant contributes scans to both training and test sets. With longitudinal cohorts this happens by default if you split on scans rather than on people. The model recognizes the individual, not the condition.
Site leakage. Splitting randomly across a multi-site cohort puts every site on both sides, so the model never has to generalize to an unseen scanner — the thing you actually need it to do. See harmonization.
Preprocessing leakage. Any step fitted on the full dataset before splitting — feature selection, normalization, dimensionality reduction, imputation — carries test-set information into training. Selecting the top-k features by their correlation with the outcome across all subjects is the classic version, and it can manufacture strong accuracy from pure noise.
The general rule: the split comes first, and every fitted transformation is fitted inside the training fold only.
Related¶
Working with imbalanced data — its “Combination and Proper Cross-Validation” section covers resampling inside the fold