Shanoir-NG - Import, manage and share neuroimaging data

Shanoir-NG (Next Generation) is a software that helps neuroimaging analysis researchers, radiologists, and MRI operators to organize and share neuroimaging datasets.

GitHub is a tool for developers if you are seeking information at a user level view please visit http://shanoir.org.

Shanoir-NG is a complete technological remake of the first version of the Shanoir application, maintaining the key concepts of Shanoir, enhancing the functionalities and the user interface and providing a great flexibility for further improvements.

Shanoir-NG is copyrighted by Inria and is now open source under the GNU General Public License v3.0. See the LICENCE file at the root of this project. If you want to contribute, please see the following page : Contribution Workflow.

⚠️ Disclaimers ⚠️

• The latest stable version of Shanoir-NG is on the branch "master".

• The latest dev version of Shanoir-NG is on "develop" (if mature -> merged into master)

• You can find the installation instructions for "master" branch below.

• Shanoir NG is still in the developement phase. While many functionalities work well, some are not developed yet and some might be unstable. Also It still misses production features like database backup.

• A few .env files in the docker-compose layer contain clear Keycloak password. Please choose your own password (check Keycloak password format policy first).

About Shanoir NG's architecture

• Shanoir-NG is based on a microservice architecture, that heavily uses Docker.
• Each Docker container integrates one microservice, mostly based on Spring Boot.
• Each microservice exposes a REST interface on using Swagger 2, as definition format.
• The front-end/web interface is implemented on using "Angular 2" (now 5) technology.
• Nginx and Keycloak (on using OpenID-Connect) are used to glue everything together.
• Internally dcm4che3 is used to handle all DICOM concerns and dcm4chee 5 arc-light as backup PACS.
• Furthermore dcm2niix is used for the DICOM to NIfTI conversion and Papaya Viewer for DICOM/NIfTI web view.

Many thanks to all these giants, on their shoulders we are standing to develop Shanoir-NG !

Access to REST-API on using Swagger3/OpenAPI

You can easily connect and investigate the REST-interface of Shanoir-NG using Swagger3. Depending on your server domain just call (e.g. for Neurinfo server):

• MS Studies: https://shanoir.irisa.fr/shanoir-ng/studies/swagger-ui/index.html and explore /api-docs
• MS Import: https://shanoir.irisa.fr/shanoir-ng/import/swagger-ui/index.html and explore /api-docs
• MS Datasets: https://shanoir.irisa.fr/shanoir-ng/datasets/swagger-ui/index.html and explore /api-docs
• MS Preclinical: https://shanoir.irisa.fr/shanoir-ng/preclinical/swagger-ui/index.html and explore /api-docs

Intialize Shanoir Downloader

Clone the shanoir-ng repository, the shanoir-downloader/ folder will be empty ; two commands must be run to get the code:

• git submodule init to initialize your local configuration file
• git submodule update to fetch all the data from shanoir and check out the appropriate commit listed in shanoir_downloader

Then the shanoir-downloader project can be simply managed as a normal git repo (as if it were a separated project) ; meaning that once your are in the shanoir-downloader/ folder, you can just git pull to get the latest changes, and commit some changes.

Windows User

• Install Docker Desktop

  • Install WSL Linux 64bits
  • Run Docker in admin mode
  • Delete, if needed, %appData%/Docker/settings.json (Docker will create another one, see https://forums.docker.com/t/solved-docker-failed-to-start-docker-desktop-for-windows/106976/6)

• Install Java 21

  • Download and install : https://www.oracle.com/java/technologies/javase/jdk21-archive-downloads.html
  • Add enviromnent variable : On Windows (as environment variable): JAVA_HOME = C:\Program Files\Java\jdk-21.0.1 On Mac (in your .bashrc or .zshrc file): export JAVA_HOME=$(/usr/libexec/java_home)

• Install Maven

  • Download : https://maven.apache.org/download.cgi
  • Install : https://maven.apache.org/install.html

• Install NodeJS (+npm)

  • Download : https://nodejs.org/en/download/

• Add a repository to maven (your_path_to_apache_folder\apache\conf\settings.xml or mvn help:effective-settings to check path) : shanoir shanoir-uploader shanoir-uploader-repo http://shanoir.gforge.inria.fr/doku.php?id=intern:shanoiruploader

• Configure git to change end of line caracters to windows' instead of linux' : git config --global core.autocrlf input

• Add this line to C:\Windows\System32\drivers\etc\hosts : 127.0.0.1 shanoir-ng-nginx viewer

• Fork GitHub project

• Clone

• Edit .env file and set SHANOIR_MIGRATION=init

• Using locally built docker images

• Run "mvn clean install -DskipTests" in shanoir-ng/shanoir-ng-parent/ folder

• Run docker-compose -f docker-compose-dev.yml up --build in shanoir-ng/

• Using github packages images

• Run docker-compose up --build in shanoir-ng/

• Load data manually from Docker Desktop :

  • Open terminal in database microservice
  • Run command : mysql -uroot -ppassword
  • use users;
  • Copy/paste the content of the scripts (src/main/resources/scripts/import.sql from shanoir-ng/shanoir-ng-datasets/users/import/studies)

Shanoir : https://shanoir-ng-nginx/shanoir-ng/home

Keycloak : http://localhost:8080/auth/admin/master/console/#/realms/shanoir-ng/roles

DEPLOY

• Install docker and docker-compose:
  • https://docs.docker.com/install/
  • https://docs.docker.com/compose/install/
• Make sur docker has enough memory to run Shanoir (6Gb should be enough)
• If you are on your developer/local machine:
  • Configure your local /etc/hosts (for windows, C:/Windows/System32/drivers/etc/hosts) and add:
  • 127.0.0.1 shanoir-ng-nginx
  • For windows 7, increase your RAM and set the port redirection (8080 and 443) for the virtual box.
• If you are on a dedicated server (e.g. shanoir-ng.irisa.fr):
  • By default Shanoir-NG is installed with the host shanoir-ng-nginx and the scheme http (dev setup)
  • If you are on a dedicated server (e.g. shanoir-ng.irisa.fr) you will have to do manual adaptions (we tried to automate as much as possible in a given time and there is still a way to go, but here we are currently)
    1. Keycloak: Open /docker-compose/keycloak/cfg/shanoir-ng-realm.json and change redirectUris and webOrigins
    2. Spring Boot: Open /.env and change the host and scheme of all three properties in the file
    3. Docker Compose: Open /docker-compose.yml and change the container_name of Nginx to e.g. shanoir-ng.irisa.fr. This is necessary, that e.g. ms users and the Keycloak CLI client can access to Keycloak (resolve the host name)
    4. Angular: Open /shanoir-ng-front/config/webpack.config.js and change SHANOIR_NG_URL_SCHEME and SHANOIR_NG_URL_HOST
  • Attention: you will have to re-compile your code after these changes with Maven!!!
• Just in case you have some old stuff of Shanoir-NG in your docker environment:
  • docker system prune -a
  • docker volume prune
  • Attention: this will clean your entire docker system!
• Go to the root folder (/shanoir-ng) and execute docker-compose up --build
  • Attention: the file .env in the root folder is used to set environment variables and will not be found if you run docker-compose elsewhere; results in errors after
• Access to shanoir-ng: https://shanoir-ng-nginx

If you want to login, please configure a user in Keycloak.

Please note, that the MS Users does for security reasons not publicly expose his REST-interface.

Requirements

To build and deploy Shanoir, you will need:

• Java 21 (since migration to Spring Boot 3.1.2)
• docker (https://docs.docker.com/install/)
• docker-compose 3 (https://docs.docker.com/compose/install/)
• maven 3
• at least 10GB of available RAM

Installation of Shanoir NG

The installation of Shanoir NG happens in three steps :

• BUILD
• CONFIGURE
• DEPLOY

The TL;DR section gives the minimal for bootstrapping a development environment. The following sections give detailed informations about each step.

TL;DR

The default docker-compose configuration is well-suited for a development environment. Each microservice is hosted in a separate container and the application data are stored in named volumes.

Shanoir is now based on github packages for the microservices images. If you want to build your own 'local' images, you have to run shanoir with docker-compose-dev.yml file.

Before deploying, some configuration is required:

• add the following line in your /etc/hosts (so that https://shanoir-ng-nginx/ is reachable from your web browser):

  127.0.0.1  shanoir-ng-nginx viewer
  

• if docker is not running natively and thus you are using docker-machine (windows/macos users), you will need to tune the virtualbox machine:
  • increase the amount of allocated RAM
  • set up tcp port redirections (at least for 8080 and 443)

The bootstrap.sh script automates the build and the deployment of shanoir.

WARNING: this script is destructive (as it will wipe out the external volumes configured in the docker-compose.yml). It is not recommended to use on a production host.

To deploy shanoir from scratch on a development machine you can just launch the following command and have a coffee.

./bootstrap.sh --clean

Once the bootstrap is complete, go on to the FIRST RUN section to create the initial users.

BUILD

The build consists of two stages: build the microservices and build the docker images.

In the source tree, each microservice is located in a separate shanoir-ng-*/ directory containing a maven project. shanoir-ng-parent is a meta-project that includes all the other projects. The contextes of the docker images are located in the docker-compose/*/ directories.

Procedure:

• Download or git clone the shanoir-ng code. The master branch should be the most stable while develop will contain the newests functionalities if you are interested in testing thoses.

• Execute the Maven build on the parent project with the following commands:

  cd shanoir-ng-parent/ && mvn install
  

  The build creates all .jar and .js executable files and stores a copy into the docker-compose/ folder to be used for building the docker images

• Build the docker images:

  docker-compose build
  

• In order to use these built images, when running "docker compose" command, you have to precise docker-compose-dev.yml file. Otherwise, github packages are used by default.

CONFIGURE

Shanoir is configured with environment variables. It is mostly handled with a a set of facade variables named SHANOIR_* (which cover the most typical setups).

Name	Value	Description
SHANOIR_URL_HOST	hostname	hostname where shanoir is reachable
SHANOIR_URL_SCHEME	http|https	https (over TLS), http (plain text, NOT RECOMMENDED)
SHANOIR_SMTP_HOST	hostname	SMTP relay for outgoing e-mails
SHANOIR_ADMIN_EMAIL	e-mail address	contact address of the administrator (for outgoing e-mails)
SHANOIR_ADMIN_NAME	name	name of the administrator (for outgoing e-mails)
SHANOIR_PREFIX	slug (optional)	prefix for container names (needed if you deploy multiple shanoir instances on the same host)
SHANOIR_X_FORWARDED	generate|trust	configures whether the nginx container generates the X-Forwarded-* HTTP headers (if running stand-alone) or trusts the existing headers (if located behind another reverse-proxy)
SHANOIR_CERTIFICATE	auto|manual	auto-generates a self-signed TLS certificate (NOT RECOMMENDED) or use a manually installed certificate
SHANOIR_MIGRATION	auto|init|never|manual|export|import	Normal runs should use auto in development and never in production. Other values are for controlling deployment and migrations (see below).
SHANOIR_KEYCLOAK_USER SHANOIR_KEYCLOAK_PASSWORD	username/password	Keycloak admin account used by shanoir for managing user accounts
SHANOIR_VIEWER_OHIF_URL_SCHEME	http|https	https (over TLS), http (plain text, NOT RECOMMENDED)
SHANOIR_VIEWER_OHIF_URL_HOST	hostname	hostname where the OHFI-Viewer is reachable

Notes

• You must ensure that the hostname SHANOIR_URL_HOST can be resolved from the clients using shanoir (the users) and from each microservice (running in the containers)
  • in development this is achieved by:
    • adding the following line to your /etc/hosts

      127.0.0.1  shanoir-ng-nginx viewer
      

    • ensuring that the nginx container name is equals to the value of SHANOIR_URL_HOST (in the default setup, they both use: shanoir-ng-nginx)
  • in production, you should have a the relevant A/AAAA configured in your DNS zone
• If docker is not running natively and thus you are using docker-machine (windows/macos users), you will need to tune the virtualbox machine:
  • increase the amount of allocated RAM
  • set up tcp port redirections (at least for 8080 and 443)
• The TLS configuration provided by the nginx container is permissive and not guaranteed to be up-to-date. You should not rely on it for facing the public internet. In production you should rather have a separate HTTP reverse-proxy, properly administrated according to your security policies.
• Upon config changes, for most cases you will just need to re-create the affected containers (run: docker-compose up -d). However if the changes also affect the keycloak container, then you will also need to re-deploy the shanoir-ng realm (see below).

DEPLOY

For all the following commands, if you want to deploy locally built images, you have to add "-f docker-compose-dev.yml" argument to the "docker compose up" command.

0. ensure all containers are stopped and all volumes are destroyed (CAUTION: this destroys all external volumes defined in docker-compose.yml)

   fig down -v
   

1. deploy the database containers and wait until they are ready to accept incoming connections

   docker-compose up -d database keycloak-database
   

2. initialise the keycloak container, then start it

   docker-compose run --rm -e SHANOIR_MIGRATION=init keycloak
   docker-compose up -d keycloak
   

3. initialise each microservice

   for ms in users studies datasets import preclinical ; do
       docker-compose run --rm -e SHANOIR_MIGRATION=init "$ms"
   done
   

4. start the remaining containers

   docker-compose up -d
   

FIRST RUN

New user accounts need to be validated by a shanoir admin. However, on the first run, there is not admin account so you will need to create it on the keycloak server directly:

1. go to Keycloak admin interface: http://localhost:8080/auth/admin/
2. sign in with the credentials configured in SHANOIR_KEYCLOAK_USER/SHANOIR_KEYCLOAK_PASSWORD' (default is admin/&a1A&a1A`)
3. go to the shanoir-ng realm
4. create/edit the new user and grant the relevant role (eg. ROLE_ADMIN). By default, new user accounts are created in Keycloak by the users microservice with temporary passwords, you may reset the password in keycloak's admin interface and receive the new password is by e-mail. In development, if you do hot have a configured SMTP relay, then you may choose to overide the password manually and set Temporary password: No to make it persistent. Go to the "Attributes" tab of your User page and create a new attribute "userId" and set the value with a random number not taken by another user.

PACS dcm4chee

Access to the backup PACS dcm4chee 5 arc-light: http://localhost:8081/dcm4chee-arc/ui2/

Local data

This installation uses Docker named volumes, find more here to handle your local data: https://docs.docker.com/storage/volumes/

Migrations

TODO