Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow LocalAccessPoint to load EModels from Nexus staged data #111

Conversation

GianlucaFicarelli
Copy link
Contributor

@GianlucaFicarelli GianlucaFicarelli commented Feb 26, 2024

This PR should allow to use LocalAccessPoint to load an EModel from data retrieved from Nexus.

It needs a recipes.json file containing the path to the files that have been previously retrieved:

{
  "cSTUT": {
    "morph_path": "morphologies",
    "morphology": [
      [
        "cylindrical_morphology_14.7805",
        "cylindrical_morphology_14.7805.swc"
      ]
    ],
    "params": "EMC__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json",
    "pipeline_settings": "EMPS__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json",
    "features": "FCC__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json",
    "final": "EM__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0__11.json"
  }
}                                                                                                                                                                                                   

This file is similar to the existing recipes.json already used by LocalAccessPoint, with these changes:

  • morph_path: unchanged
  • morphology: unchanged
  • params: unchanged (the EMC file contains lists of dicts instead of dicts, but it can be opened in the same way)
  • pipeline_settings: it can be now a string instead of a dict, and in that case the settings are loaded from file.
  • features contains the path to FCC (seems backward compatible)
  • protocol removed (it looks like it's present only in the legacy format)
  • final is a new key that contains the path to the EM file downloaded from Nexus. It will be used to generate on the fly a content compatible with final.json (if the name of the key final is not clear enough, it could be renamed to something different). If the key is present in recipes.json, the final parameter passed to LocalAccessPoint is used only when saving a new final.json. If the key isn't present, the behaviour is the same as before.

There are some files that can be retrieved from Nexus but not present in recipes.json. They can be added if needed, and they are:

  • EMS_*.json (EModelScript)
  • EMW_*.json (EModelWorkflow)
  • ETC_*.json (ExtractionTargetsConfiguration)

Other changes:

  • the morphologies are retrieved using the parameters in recipes.json, instead of using the parameters from the EMC file (because format and path are null in the EMC file from Nexus), or instead of using the default morphology dir.
  • when distributions are defined, remove the default uniform distribution if added at
    configuration = NeuronModelConfiguration(
    (or it can end up with a duplicate uniform distribution)

Assuming that the emodel dir contains for example:

.
├── EMC__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json
├── EMPS__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json
├── EMS__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0__11.json
├── EMW__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json
├── EM__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0__11.json
├── ETC__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json
├── FCC__emodel=cSTUT__etype=cSTUT__species=mouse__brain_region=grey__iteration=2f92aa0.json
├── figures
│   ├── emodel=cSTUT__etype=cSTUT__brain_region=grey__iteration=2f92aa0__parameters_distribution.pdf
│   ├── emodel=cSTUT__etype=cSTUT__brain_region=grey__iteration=2f92aa0__seed=11__scores.pdf
│   └── emodel=cSTUT__etype=cSTUT__brain_region=grey__iteration=2f92aa0__seed=11__traces.pdf
├── mechanisms
│   └── *.mod
├── model.hoc
├── morphologies
│   └── cylindrical_morphology_14.7805.swc
├── recipes.json
└── x86_64
    └── <files compiled with nrnivmodl>

then the LocalAccessPoint can be initialized with something like the following:

access_point = LocalAccessPoint(
    emodel="cSTUT",
    emodel_dir=".",
    etype="cSTUT",
    iteration_tag="2f92aa0",
    recipes_path="recipes.json",
)

The parameters emodel, etype, iteration_tag are still needed because they cannot be determined from recipes.json. We could add them to recipes.json, or to another config file, but it would require other changes to the initialization of LocalAccessPoint.
If they are added to EM_*.json (that already contains seed) then they could be automatically determined from this file.

@GianlucaFicarelli GianlucaFicarelli force-pushed the local_access_point_from_nexus_data branch 3 times, most recently from ab27d6d to dda9bf0 Compare February 26, 2024 14:06
"""Convert the configuration stored in EM_*.json to the format used for final.json."""
metadata = self.emodel_metadata
return {
metadata.emodel: {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can use vars(self.emodel_metadata) instead of listing all of emodel_metadata attributes individually here. Also to stay consistent with what is done in store_emodel method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Alternatively, would it make sense to add these keys to the EM_*.json file, so that all the keys can be retrieved from this file?
It would be a step forward to being able to initialize LocalAccessPoint with just recipes.json (without manually specifying etype, emodel, iteration)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really...
BluePyEModel has been designed to separate the resource data in the EM_*.json from the metadata (etype, emodel, iteration, etc.). We expect the user to always give the metadata when creating the access point, which will then fetch the appropriate files form the file system for the local access point or from nexus for the nexus access point.

Copy link
Contributor Author

@GianlucaFicarelli GianlucaFicarelli Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the use case I'm considering, I'm retrieving all the information of the EModel from Nexus and storing them locally.
Then, in a separate step, I want to load the EModel from the local data without querying Nexus anymore.
If the metadata aren't stored in any local file, it won't be possible to load the EModel with just that information, so it seems that the information retrieved from Nexus and stored locally would be incomplete without the metadata.

Alternatively, I can save the metadata from Nexus to an additional file, and I could add for example a function or a class method LocalAccessPoint.from_metadata() to load this file and initialize the LocalAccessPoint, without modifying the logic of the existing constructor.
This file would act as the entry point for my use case, as it is circuit_config.json for circuits, or simulation_config.json for simulations, or config.json for simulation campaigns.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Saving the metadata from nexus in an additional file and using it as an entry point sounds fine to me

Copy link
Collaborator

@AurelienJaquier AurelienJaquier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!
Is it ok to merge it, or do you have additional changes to do @GianlucaFicarelli ?

@GianlucaFicarelli
Copy link
Contributor Author

looks good to me! Is it ok to merge it, or do you have additional changes to do @GianlucaFicarelli ?

Thanks @AurelienJaquier, I'll add some tests with data in the format retrieved from nexus then it's ok to merge

Jaquier Aurélien Tristan added 2 commits March 1, 2024 14:42
Change-Id: Id74a0caa20c9cd6d0b21e893034e66123bc5887f
…thers

Change-Id: I2c203148cf77465731e28fd9b645906897a047b2
@codecov-commenter
Copy link

codecov-commenter commented Mar 1, 2024

Codecov Report

Attention: Patch coverage is 91.01796% with 15 lines in your changes are missing coverage. Please review.

Project coverage is 59.11%. Comparing base (2fad623) to head (ad6594a).

Files Patch % Lines
bluepyemodel/access_point/local.py 81.63% 9 Missing ⚠️
bluepyemodel/evaluation/protocols.py 16.66% 5 Missing ⚠️
...s/unit_tests/test_local_access_point_from_nexus.py 97.82% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #111      +/-   ##
==========================================
+ Coverage   58.15%   59.11%   +0.96%     
==========================================
  Files         103      106       +3     
  Lines        7380     7512     +132     
==========================================
+ Hits         4292     4441     +149     
+ Misses       3088     3071      -17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@darshanmandge darshanmandge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @GianlucaFicarelli for adding this functionality!

@AurelienJaquier AurelienJaquier merged commit 8eb60d5 into BlueBrain:main Mar 4, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants