Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FocusCounter diagnostic tool #649

Merged
merged 13 commits into from
Feb 27, 2022
Merged

Conversation

tsalo
Copy link
Member

@tsalo tsalo commented Feb 22, 2022

Closes #595.

Changes proposed in this pull request:

@tsalo tsalo added enhancement New feature or request cbma Issues/PRs pertaining to coordinate-based meta-analysis diagnostics Issues/PRs related to the diagnostics module. labels Feb 22, 2022
@codecov
Copy link

codecov bot commented Feb 22, 2022

Codecov Report

Merging #649 (c238670) into main (8f166ab) will increase coverage by 0.12%.
The diff coverage is 93.42%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #649      +/-   ##
==========================================
+ Coverage   84.88%   85.01%   +0.12%     
==========================================
  Files          40       40              
  Lines        4386     4457      +71     
==========================================
+ Hits         3723     3789      +66     
- Misses        663      668       +5     
Impacted Files Coverage Δ
nimare/diagnostics.py 93.87% <92.18%> (-1.42%) ⬇️
nimare/workflows/ale.py 93.33% <100.00%> (+0.90%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f166ab...c238670. Read the comment docs.

@tsalo
Copy link
Member Author

tsalo commented Feb 23, 2022

Currently failing because Jackknife has issues with the IBMA estimator, which I'm guessing is due to changes in the aggressive mask based on the subset of experiments in the meta-analysis.

@tsalo
Copy link
Member Author

tsalo commented Feb 26, 2022

The problem is that, when you fit the same IBMA estimator multiple times, each time the aggressive mask is updated based on the new data and the existing aggressive mask using a logical or, meaning that good voxels in an older dataset will be retained when fitting to the new dataset. This causes problems when some of those older voxels are not good for the new dataset.

This problem wasn't already apparent because all of our Jackknife tests on IBMAs had good data in all of the voxels, I believe. Or at least the aggressive mask didn't change from one subset of studies to another.

I'm not sure what the best solution is. Here are some ideas:

  • Automatically replace the aggressive mask when fitting an estimator to a new dataset. I think this would just be a step in _preprocess_inputs to delete that key from the inputs_ dictionary if it exists at the beginning of the preprocessing.
  • Add an argument to amend or replace the mask to the IBMA estimator initialization, so users can decide.
  • Just make a copy of the estimator within the Jackknife _transform method so that it amends the original aggressive mask, but doesn't iteratively update masks from different subsets of the dataset.

@tsalo
Copy link
Member Author

tsalo commented Feb 26, 2022

For the sake of this PR, I will probably just make a copy, since that's probably a good idea in general. I will also open a separate issue about the aggressive mask bug.

@tsalo tsalo merged commit d427f51 into neurostuff:main Feb 27, 2022
@tsalo tsalo deleted the focus-count-diagnostic branch February 27, 2022 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cbma Issues/PRs pertaining to coordinate-based meta-analysis diagnostics Issues/PRs related to the diagnostics module. enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Focus-count post-meta-analysis method
1 participant