Skip to content

Home

Jump to bottom
Colley JM edited this page Mar 28, 2024 · 118 revisions

Constraints to use GRANDLIB

ROOT library is used as format for input/ouput data. GRANDLIB provides a python binding of GULL et TURTLE from C language, but proper installation is not guaranteed on all operating systems. So to prevent any problem installing these three libraries we provide an docker image where ROOT is installed and the compilation of GULL and TURTLE is correct. GRANDLIB docker images are built from Ubuntu 20.04 with ROOT given by official rootprojet.

Conda is an alternative to Docker but it is currently only available for linux, see here for more information.

Users

Use a GRANDLIB release

Note : not yet available, only with demo mode

You can use GRANDLIB release in Docker image, available on DockerHub grandlib/release. A basic tutorial to use docker for GRANDLIB is available. A specific documentation for GRANDLIB users is available here (need to update).

With Jupyter Notebook

You can use Jupyter NoteBook inside docker with your host web navigator. Just :

  • run docker with -p 8888:8888 option
  • go to directory where grandlib release is installed : /opt/grandlib/grand
  • used a specific alias grand_jupyter to start jupyter notebook
  • copy/paste in your host web navigator the last URL given by jupyter in standard output

Example:

$ docker run -p 8888:8888 -it --rm grandlib/release
root@58788b7dc04e:/home# cd /opt/grandlib/grand/  
root@58788b7dc04e:/opt/grandlib/grand# grand_jupyter 
[I 14:17:35.829 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[I 14:17:36.012 NotebookApp] Serving notebooks from local directory: /opt/grandlib/grand
[I 14:17:36.012 NotebookApp] Jupyter Notebook 6.4.11 is running at:
[I 14:17:36.012 NotebookApp] http://58788b7dc04e:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0
[I 14:17:36.012 NotebookApp]  or http://127.0.0.1:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0
[I 14:17:36.012 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:17:36.015 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-13-open.html
    Or copy and paste one of these URLs:
        http://58788b7dc04e:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0
     or http://127.0.0.1:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0

With command line in docker

If your script used some plots, you must enable correctly graphics in docker, you can review the following tutorials:

Use GRANDLIB environment in computing center

For production in computer center, most of them use Apptainer (or Singularity same CLI as Apptainer) instead Docker. So you can create from Docker image a Apptainer image with this command:

apptainer build grandlib_xxxx.sif docker://grandlib/xxx:yyy

or at CCIN2P3 used an existing image here

$THRONG_DIR/apptainer

Use Apptainer on an interactive node

apptainer shell --bind /pbs,/sps/grand /path/to/grandlib_xxxx.sif

Use Apptainer on a computing node

You can find an example of Apptainer/Singularity use with batch system in CCIN2P3 documentation.

Developers

You can use GRANDLIB development environment docker image, available on DockerHub grandlib/dev. This docker image is different of release image, it doesn't contain any GRANDLIB code only the development environment. A basic tutorial to use docker for GRANDLIB is available.

Minimal configuration for git/github

  • How define your token github for continuous integration (CI)

The github tokens can be generated with different rights, you must select workflow in setting github_setting_zoom

than developer setting (last line left column)/Personal access tokens/generate new token.

token_SQ

Otherwise the CI isn't executed, quality report isn't generated and isn't sent to SonarQube server.

  • How define user and mail in your commit and push

You must define the email associated with your git account before pushing. In the root git package, write these git commands

$ git config --local user.email <mail defined in github>
$ git config --local user.name <user name defined in github>

If you have only one git account, you can use --global option instead --local.

Development environment

A docker image with GRANDLIB environment is available without source of GRANDLIB. You can clone GRANDLIB package inside container or outside and add a link between container/local direction with option -v to access to your data or sources, see basic tutorial to use docker for GRANDLIB.

Don't forget to initialize grand library before use it with script setup.sh in env directory

$ git clone https://github.com/grand-mother/grand.git
$ docker run -it --rm -v $PWD:/home grandlib/dev
root@58788b7dc04e:/home# ls grand
bin  COPYING.LESSER  docs  env   examples  grand  lib  LICENSE  log.txt  quality  
README.rst  src tests  wheel
root@dc6f5b78ef35:/home# cd grand
root@dc6f5b78ef35:/home/grand# source env/setup.sh
var GRAND_ROOT=/home/grand
==============================
update PATH with grand quality
==============================
update PYTHONPATH with grand source
==============================
Install external lib gull and turtle
====================================
CLONE    turtle
MAKE     libturtle.so
CLONE    gull
MAKE     libgull.so
INSTALL  turtle.h
INSTALL  gull.h
PYTHON   _core.abi3.so
root@dc6f5b78ef35:/home/grand#

Jupyter notebook and Docker

Like for release docker image case, you can use Jupyter notebook with dev docker image, you must just initialize correctly the grand environment before launch Jupyter with your local version of grandlib package.

Example with a shared volume, so we have a grand package in current directory

$ ls grand
bin  COPYING.LESSER  docs  env   examples  grand  lib  LICENSE  log.txt  quality  README.rst  src tests  wheel
$ docker run -p 8888:8888 -it --rm -v $PWD:/home grandlib/dev
root@58788b7dc04e:/home# ls grand
bin  COPYING.LESSER  docs  env   examples  grand  lib  LICENSE  log.txt  quality  README.rst  src tests  wheel

Now initialize it and launch jupyter

root@dc6f5b78ef35:/home# cd grand
root@dc6f5b78ef35:/home/grand# . env/setup.sh 
...
etc
root@dc6f5b78ef35:/home/grand# grand_jupyter 
[I 14:17:35.829 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[I 14:17:36.012 NotebookApp] Serving notebooks from local directory: /opt/grandlib/grand
[I 14:17:36.012 NotebookApp] Jupyter Notebook 6.4.11 is running at:
[I 14:17:36.012 NotebookApp] http://58788b7dc04e:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0
[I 14:17:36.012 NotebookApp]  or http://127.0.0.1:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0
[I 14:17:36.012 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:17:36.015 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-13-open.html
    Or copy and paste one of these URLs:
        http://58788b7dc04e:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0
     or http://127.0.0.1:8888/?token=37f74751591b90d46de412a7565f23f1d6ddb1373519b4b0

Integrated Development Environment and Docker

Eclipse

See dedicated page to use Eclipse to develop GRANDLIB

PyCharm

With professionnal version of Pycharm on your computer you can use docker to define python interpreter, see tutorial

To use ROOT library a path mapping is necessary, in docker image ROOT is installed in /opt/root/lib directory, you must provide it to PyCharm like this:

  • Create on your local disk a empty directory my/path/mapping/root
  • In your project PyCharm : File>Settings>Project>Python interpreter>Path Mappings
    • add "Local Path" = my/path/mapping/root and "Remote path"=/opt/root/lib
  • In your project PyCharm : File>Settings>Project>Project Structure> Add Content Root (upper right)
    • In Select Content Root Directory, choice my/path/mapping/root

Remarks :

  1. Pycharm use word Root but it's ROOT library of CERN :-)
  2. Last Pycharm version may present a bug when configuring python interpreter in Docker. If building the docker image in Pycharm fails with message like Introspection completed with error: com.github.dockerjava.api.exception.ConflictException: Status 409: {"message":"Container c4ae9500e054ab074ac9b7c9260ce925d3d5d743f719a11f7076a0a3a470c294 is not running"} then you can use the following workaround : Open the Help menu, then Find Action, search for Registry and uncheck python.use.targets.api (see: https://youtrack.jetbrains.com/issue/PY-55444)

VScode

to do

Code quality

The quality of the code is applicable in priority on the new development.

Quality tools

Quality tools are available in package in quality directory:

  • pylint (static analysis, python rule and PEP8)
  • coverage.py (dynamic analysis of code from test case)
  • mypy (static analysis for optional typing)
  • black (code formatter)
    • To disable black on a mathematical formula for example, you can use the directives: "#fmt: off" and "#fmt: on"

see readme.md. In Docker image quality tools are already installed.

SonarQube and continuous integration

All developers have a personal project in GRAND SonarQube server associated to last branch pushed on github to improve the code quality of its own development.

CI_GRAND

Quality gates

4 levels are defined (low, medium, hight, prod), the difference are mainly on number of pylint issues threshold. You can see quality gate definition on GRAND Quality Gate page, the actual quality gate applied is tagged DEFAULT.

Application Program Interface (API) documentation

Actually, you can generate API documentation with sphinx-apidoc

Create HTML API documentation

In grandlib/dev container you can create HTML API documentation with this script

create_doc_html.sh

in directory

grand/docs/apidoc-only/sphinx

of grand git package, HTML result in

grand/docs/apidoc-only/sphinx/build/html

You can open it with your web browser.

Convention for the parameters with Sphinx

:param name_param : <[Unit]> short parameter description
:type  name_param : <type>

Example

:param fr_max: [Hz] The maximal frequency of the bandpass filter
:type fr_max: float

Development cycle proposal

Naming/use convention git branch:

  • master : last stable version used for release, protected branch developer can't commit directly
  • dev : most stable branch with new features foo_feat incomming
  • dev_foo_feat : developer branch in charge of development of new feature foo_feat

For each new feature:

  • Create a git branch from dev branch
  • Develop feature in this branch
  • Develop test associated to the feature
  • Apply code formatter black
  • Check code quality with local tools and SonarQube server
  • Update your branch with current dev branch and solve conflict
  • Create a Pull Request in GitHub
  • Code review and check by other developer before merging in dev branch

Bug:

  • Each bug must be identified by a issue in GitHub with clean explanation to reproduce it
  • Solve bug "foo" in dedicaded branch dev_bug_foo
  • Follow same development cycle as feature

Administrators

Code quality tools

SonarQube configuration

Continuous integration

Docker images generation and repository

Clone this wiki locally