3. Stats

Functions for non-parametric significance testing of decoding results.

Functions

• randomize_accuracy Randomizes a vector of subject-level statistics (e.g. accuracy) by performing a specified number of sign randomization iterations to get an empirical null distribution of the mean or t-statistic across subjects. A one-tailed p-value is calculated taking the observed statistic into account.

• randomize_labels Randomizes the true labels and compares the observed classifier scores with each shuffled version of the labels. Version to use when we don't have subject-level statistics. Can deal with iterated cross-validation and tests either the accuracy or the sensitivity and specificity.

• run_stats Runs sign-shuffling permutation testing by calling randomize_accuracy and corrects for multiple comparisons across time, space, or both. Observed statistic provided can be: subjects x time or subjects x space x time or subjects x time x time.

  Can perform one of the following corrections for multiple comparisons:

  • omnibus (thresholding across the maximal distribution of the mean across subjects)
  • cluster (based on temporal or spatial cluster extent; for spatial clustering, a spatial_def argument needs to be provided containing the sensor/source indices) *fdr (false-discovery rate correction)
  • a combination of omnibus and FDR corrections along different dimensions in the data (see function help).

RandSVM

The functions in this directory implement randomization of the entire classification procedure, i.e. rerunning the decoding analysis with shuffled labels to get a null distribution (yes, I've done this and it takes forever). Not recommended - sign shuffling (as in randomize_accuracy) leads to similar results.