Skip to content

Commit

Permalink
Add Nexus example (#164)
Browse files Browse the repository at this point in the history
* add nexus example

* reorganise example folders

* update readme

* add memodel readme
  • Loading branch information
ilkilic authored Aug 13, 2024
1 parent 8387c4d commit f3c4dae
Show file tree
Hide file tree
Showing 35 changed files with 2,087 additions and 23 deletions.
4 changes: 3 additions & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ To get started with the E-Model building pipeline
.. image:: https://raw.githubusercontent.com/BlueBrain/BluePyEModel/main/doc/images/pipeline.png
:alt: E-Model building pipeline

This section presents a general picture of the pipeline. For a detailed picture and how to use it, please refer to the `example directory <https://github.com/BlueBrain/BluePyEModel/tree/main/examples/L5PC/>`_ and its `README <https://github.com/BlueBrain/BluePyEModel/tree/main/examples/L5PC/README.rst>`_.
This section presents a general picture of the pipeline. For a detailed picture and how to use it, please refer to the `L5PC example <https://github.com/BlueBrain/BluePyEModel/tree/main/examples/L5PC/>`_ and its `README <https://github.com/BlueBrain/BluePyEModel/tree/main/examples/L5PC/README.rst>`_.

The pipeline is divided in 6 steps:

Expand All @@ -78,6 +78,8 @@ The pipeline is divided in 6 steps:

These six steps are to be run in order as for example validation cannot be run if no models have been stored. Steps ``validation``, ``plotting`` and ``exporting`` are optional. Step ``extraction`` can also be optional in the case where the file containing the protocols and optimisation targets is created by hand or if it is obtained from an older project.

For an example of how to run the pipeline with the Nexus access point, see the `Nexus example <https://github.com/BlueBrain/BluePyEModel/tree/main/examples/nexus/>`_

Schematics of BluePyEModel classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
13 changes: 13 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# BluePyEModel Examples

This directory includes various examples demonstrating how to use BluePyEModel:

- [**L5PC**](./L5PC/README.rst): Provides a guide for setting up and running the E-Model building pipeline locally on a personal computer or on a Slurm cluster. This is an example based on the juvenile (P14) rat somatosensory cortex layer 5 thick-tufted pyramidal cell (L5TPC) e-model optimisation of the [SSCxEModelExamples](https://github.com/BlueBrain/SSCxEModelExamples/tree/main/optimization) repository.

- [**nexus**](./nexus/README.md): Demonstrates the use of BluePyEModel through the Nexus access point.

- [**others**](./others/README.rst): Contains various examples demonstrating different functionalities and use cases.
- [**run_emodel**](./others/run_emodel/README.rst): Runs an EModel simulation on BlueCelluLab to explore the single cell behaviour.
- [**memodel**](./others/memodel/README.md): Creates, runs, analyses, and uploads a MEModel.
- [**local2nexus**](./others/local2nexus/README.md): Stores a locally built EModel (using LocalAccessPoint) to the [BlueBrain Nexus](https://github.com/BlueBrain/nexus) knowledge graph.
- [**ICSelector**](./others/icselector/icselector_example.py): Selects ion channel models using gene mappings; assigns optimisation parameter bounds and fixed values for a model.
5 changes: 0 additions & 5 deletions examples/icselector/test_icselector.sh

This file was deleted.

170 changes: 170 additions & 0 deletions examples/nexus/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# E-Model Building Pipeline with Luigi, Nexus and bbp-workflow

This example demonstrates how to build an e-model using the Nexus access, [Luigi](https://luigi.readthedocs.io/en/stable/) and [bbp-workflow](https://github.com/BlueBrain/bbp-workflow), which automates batch job pipelines. The [NexusAccessPoint](./../../bluepyemodel/access_point/nexus.py) class serves as the API that enables users to store, manage, and use e-model resources on Nexus. This class provides a set of functions to create e-model-related resources on Nexus and link them to the electrophysiological data, mechanisms, and morphology registered on Nexus.

## 1. Setup the virtual environment
To setup the virtual environment and install all the needed packages, run:

```
kinit
./create_venv.sh
```

Note: Ensure that your Python version is 3.10 or higher, as bbp-workflow is only compatible with Python >= 3.10.

Then activate the virtual environment:

```
source myvenv/bin/activate
```

## 2. Configure the Luigi pipeline:
The ``luigi.cfg`` file contains the configuration for the Luigi pipeline. Ensure you set ``virtual-env`` to the path of your virtual environment (created in the previous step) and ``chdir`` the absolute path of the current working directory. Other parameters can be adjusted based on your requirements. Below is a description of several variables you may wish to customise:

- [DEFAULT] : settings section used by bbp-workflow.
- ``account``: project name (e.g. proj72).
- ``virtual-env``: path to your virtual environment.
- ``chdir``: absolute path of the current working directory.
- ``workers``: number of workers to be used by the pipeline. Currently, only 1 worker is supported.
- ``time``: maximum time allowed for the pipeline to run.
- [Optimise] : settings section used for the optimisation.
- ``node``: number of nodes to be used on the HPC.
- ``time``: maximum time allowed to run the optimisation.
- [parallel]
- ``backend``: select the backend for processing, which can be either 'ipyparallel' or 'multiprocessing'. If left unspecified, no parallelisation will be used.
- [EmodelAPIConfig]
- ``nexus_project``: a valid Nexus project name to which the emodel should be uploaded.

For a detailed description of the configuration file, please refer to the [Luigi documentation](https://luigi.readthedocs.io/en/stable/configuration.html).

## 3. Register the resources that will be used by the pipeline on the Nexus project

Prior to executing the Luigi pipeline, it is essential to register the following resources within the Nexus project, as they will be needed by the pipeline:

- ``EModelPipelineSettings`` (EMPS): the pipeline settings of the e-model.
- ``ExtractionTargetsConfiguration`` (ETC): The extraction target configuration of the e-model and the links to the ephys data. This resource is created by parsing the ``targets.py`` using the ``configure_targets`` function in ``pipeline.py``.
- ``EModelConfiguration`` (EMC): the configuration of the e-model, which links to the morphology and mechanisms and stores a reformatted version of the parameters file of the e-model.


To create and register the ``EModelPipelineSettings`` (EMPS) and ``ExtractionTargetsConfiguration`` (ETC), run:

```
python pipeline.py --step=configure_nexus --emodel=EMODEL_NAME --iteration_tag=ITERATION_TAG --etype=ETYPE --mtype=MTYPE --ttype=TTYPE
```

The ``EMODEL_NAME`` can be any string without special characters, except for underscores, e.g., "L5_TPC_B_cAC".

The ``ETYPE``, ``MTYPE`` and ``TTYPE`` have to be valid names present in the gene map (e.g., ETYPE=="cAC", MTYPE="L3_TPC:C", TTYPE="0103 L2/3 IT CTX Glut_1").

The iteration_tag can be any string without spaces. This variable enables the user to run different tests or iterations in parallel on the same e-model. All Nexus resources related to BluePyEModel will be tagged with the iteration_tag, and the pipeline will only use resources with a matching tag. It is important to ensure that the ``iteration_tag`` specified in luigi.cfg matches the ``iteration_tag`` used when running pipeline.py. If a different ``iteration_tag`` is used in pipeline.py, the pipeline will crash because BluePyEModel will not find the expected resources.

Additionally, please note that the ephys trace files and the targets (e-features and protocols) used in this example are hardcoded in the targets.py file.

The user will be prompted to confirm their intent to deprecate the resources currently present on Nexus that have the same iteration_tag and emodel name. Following this, they will be required to enter their Nexus token, which can be obtained from https://bbp.epfl.ch/nexus/web/.
![](./img_nexus_token.png)

To set up the EModelConfiguration (EMC), which includes the model's channels, parameters and parameter distributions. You can either create the configuration based on gene data, or through a legacy json file. You can specify the legacy JSON file by adjusting the path of ``legacy_conf_json_file`` in ``pipeline.py`` and then run:

```
python pipeline.py --step=configure_model_from_json --emodel=EMODEL_NAME --iteration_tag=ITERATION_TAG --etype=ETYPE --mtype=MTYPE --ttype=TTYPE
```

If you choose to create the configuration based on gene data, you will need to provide a ttype, thus, ensure that you have also specified the ``TTYPE`` when running the ``configure_nexus`` step:

```
python pipeline.py --step=configure_model_from_gene --emodel=EMODEL_NAME --iteration_tag=ITERATION_TAG --etype=ETYPE --mtype=MTYPE --ttype=TTYPE
```

If you wish to modify the gene-based configuration before proceeding with model optimisation, you can get the configuration from Nexus and modify it before proceeding further. The jupyter notebook [edit_neuron_model_configuration.ipynb](../ncmv3//edit_neuron_model_configuration.ipynb) explains how to do so.

An example of how to run the `configure_nexus` and `configure_model_from_gene` steps is provided in the [run_pipeline.ipynb](./run_pipeline.ipynb) notebook.

## 4. Run the Luigi pipeline:
Set the variables ``emodel``, ``etype``, ``iteration``, ``mtype``, and ``ttype`` in ``launch_luigi.sh`` to match the values you chose for ``EMODEL_NAME``, ``ETYPE``, ``ITERATION_TAG``, ``MTYPE``, and ``TTYPE``. Additionally, ensure that the ``species`` and ``brain_region`` variables are consistent with the values in ``pipeline.py``.


Then, execute:

```
source myvenv/bin/activate
kinit
./launch_luigi.sh
```

The pipeline should start, and after a few minutes, you will see the message: "No more log messages received, connection timed out." To monitor the progress of the pipeline, you can use a web app by first executing the following command:

```
bbp-workflow webui -o
```

Then, open the URL returned by this command in your browser.

If an error happens during the execution of the workflow, the command ``./launch_luigi.sh`` can be run again and the workflow will restart from the latest step. If the error persists, please refer to the [Troubleshooting](#troubleshooting) section or contact the package's authors for further assistance.

## 5. Results:

Once the Luigi pipeline has run successfully, the following resources will be saved in the Nexus project specified by the ``nexus_project`` variable in pipeline.py, along with the hoc file of the e-model:

- ``FitnessCalculatorConfiguration`` (FCC): the fitness calculator configuration of the e-model, which stores the features and protocols file of the e-model.
- ``EmodelScript`` (ES): the hoc file of the e-model.
- ``EModelWorkflow`` (EMW): the resource to which all the above resources are linked to, including the workflow state.
- ``EModel`` (EM): all the information related to an optimised e-model. It contains the final parameters of the e-model from final.json, and pdfs of the e-model distribution plots, features scores and e-model response traces. It also links to EModelWorflow.

In conclusion, here is the graph structure that will be generated on Nexus upon completing the entire pipeline:

```
EModelWorkflow
|
├──> EModelPipelineSettings
|
├──> ExtractionTargetsConfiguration
| |
| ├──> Trace1
| ├──> ...
| └──> TraceN
|
├──> EModelConfiguration
| |
| ├──> Mechanism1
| ├──> ...
| └──> MechanismN
| └──> Morphology
|
├──> FitnessCalculatorConfiguration
|
├──> EModel
|
└──> EModelScript
```

You can also check the graph structure of the resources created on [Nexus](https://bbp.epfl.ch/nexus/web/).

To check that the models were optimised successfully, you can refer to the figures created in ``./figures/EMODEL_NAME/``.
The `optimisation` subfolder contains plots of the fitness versus number of generations run by the optimiser, while the `traces` and `scores` subfolders contain the voltage traces and efeatures scores for the e-models.

In addition, we provide a set of notebooks to help users understand how to edit some of the resources created by the pipeline. The first, [edit_fitness_calculator_configuration.ipynb](./edit_fitness_calculator_configuration.ipynb) explains how to edit an existing neuron model configuration on Nexus in order to add a new e-feature or protocol. The second notebook, [edit_neuron_model_configuration.ipynb](./edit_neuron_model_configuration.ipynb), explains how to load an already existing neuron model configuration from Nexus, edit it, and store it back on Nexus. A third notebook [exploit_model.ipynb](./exploit_model.ipynb) explains how to run other protocols on a model or investigate the final model yourself.


## Troubleshooting
Refer to the [Troubleshooting of BluePyEModel](https://github.com/BlueBrain/BluePyEModel/tree/main/examples/L5PC#troubleshooting) for common issues.


### When running launch_luigi.sh, if you see the error:

```
RAN: /usr/bin/ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o StreamLocalBindUnlink=yes -NT -R/tmp/2609079-sch.sock:localhost:8082 -R/tmp/2609079-agt.sock:/tmp/ssh-IlIhsVxNsOVr/agent.17 r4i3n27
STDOUT:
STDERR:
```

Solution: run the command "kinit" and provide your password.

### There is a failed task in the Luigi webapp:

Click on the red little "bug" button next to the task, it will display the log.

If the last line of the log reads something like: "failed to find resource for filter {...}", there are two possible causes:
1. You launched the pipeline with an iteration_tag different from the one you specified when configuring the pipeline. If that's the case, edit your luigi.cfg and inform the correct iteration_tag.
2. It happens from time to time that nexus forge fails to get a result even when a matching resource exists. If that's the case, launch the pipeline again, it will restart from where it stopped.
30 changes: 30 additions & 0 deletions examples/nexus/create_venv.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash

#####################################################################
# Copyright 2024 Blue Brain Project / EPFL

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#####################################################################

deactivate
module purge all

module load unstable python
python -m venv myvenv
module purge all
source myvenv/bin/activate

pip install bbp-workflow
pip install bluepyemodel[all]

deactivate
Loading

0 comments on commit f3c4dae

Please sign in to comment.