MaCWorp is a web application to give workflow developers a simple way to make their workflows easily accessible via a web browser for everyone.
- What it does for you
- Graphical user interface for
- Data up- & download
- Parameter input
- Rendered results (interactive plots, PDFs, images, tables, ...)
- Authentication
- Simple file based authentication (easy to set up, recommended for smaller teams, labs or institutes)
- OpenIDConnect (e.g. connect to ELIXIR's Life Science Login)
- Distributed execution of a workflow on different workers
- Coming soon(ish):
- Support for multiple workflow engines (currently Nextflow is working)
- Fine grained access control to projects and workflows including sharing with others.
- Graphical user interface for
- What it does not for you
- Scaling - This is still the job of the workflow engine. E.g. you can configure Nextflow to use K8s or Slurm when started by MaCWorP workers but setting these executors up is still a separate job for an admin.
A web API written in Flask for managing/scheduling workflows.
A web interface written in NuxtJS. This is basically a GUI for the web API.
A worker written in python which runs the scheduled workflows.
Make sure the dependencies in
are installed.
# Create environment
conda env create -f environment.yml
# Already creates the environment and need updates???
conda env update -f environment.yml --prune
# Upgrade pip and setuptools
conda activate nf_cloud
# Install node requirements
yarn --cwd ./frontend install
# Shell 1
docker-compose up
# Shell 2
python -m nf_cloud_backend database migrate
python -m nf_cloud_backend utility rabbitmq prepare
honcho -e dev.env start
Component | Access | User | Password |
---|---|---|---|
Frontend | http://localhost:5001 |
||
API | http://localhost:3001 |
||
Fusionauth | http://localhost:9011 |
[email protected] |
developer |
For development, Flask is configured to add CORS-Headers by default.
To keep track of database changes, this project uses peewee_migrate
.
Migrations are located in nf_cloud_backend/migrations/
. For now this needs to be added manually when using pw_migrate
.
pw_migrate create --directory nf_cloud_backend/migrations --database postgresql://postgres:[email protected]:5434/nf_cloud "<description>"
pw_migrate migrate --database 'postgresql://postgres:[email protected]:5434/nf_cloud' --directory nf_cloud_backend/migrations
psql postgresql://postgres:[email protected]:5434/nf_cloud
In production NF-Cloud is designed to run as Gunicorn service behind a NginX reverse proxy. This setup can be tested with make production-test
This will build and start back- & frontend containers, start multiple instances and put them behind a NginX reverse proxy, running on the port 16160 (NF-Cloud) and 16161 (Fusionauth). Both applications are served under HTTPS with a self signed certificate for testing and the hosts name as domain.
A worker can be started with make production-worker-test
Use python -m nf_cloud_backend database seed --drop
to insert some test records (e.g. workflows) into the database.
Set the consumer_timeout
to a high value. Otherwise scheduled workflows can't be acknowledged after finishing and getting rescheduled
The following part contains CLI usage of nf_cloud_backend
, which assume you use the nativ installation. If you run nf_cloud_backend
in docker just replace python -m nf_cloud_backend
with docker run mpc/nf_cloud_backend
.
Create the docker image with
docker build -t mpc/nf-cloud-backend:latest -f docker/backend.dockerfile .
You can use the build arguments (--build-arg
) NEW_MAMBA_USER_ID
and NEW_MAMBA_USER_GID
to change the user and group ID of the container user. Useful for harmonizing the ownership of the uploaded files with a local user.
docker build -t mpc/nf-cloud-frontend:latest -f docker/frontend.dockerfile .
The images entrypoint is the nf_cloud_backend
command line interface, so users can start using command line arguments (e.g. --help
). If the option serve --gunicorn
is given, the image will start NF-Cloud as Gunicorn application, unlike the native installation which prints the Gunicorn parameters.
NF-Cloud is running in the folder /home/app
. So please mount a local configuration accordingly when running the images, e.g.: docke run -v <some-local-config>:/home/app/nf_cloud.local.config.yaml
. To persist the uploaded files a persistent volume or bind-mount should be used.
Attention: The worker needs to be deployed natively to use Docker containers in workflows.
Clone the repository and use Conda or Miniconda to create an environment with all necessary dependencies: conda env create -f environment.yml
(the environment name will be nf_cloud
)
Make sure to activate the environment and go into the root folder of the repository before using nf_cloud_backend
or use conda run -n nf_cloud python -m nf_cloud_worker
Run conda run -n nf_cloud python
python -m nf_cloud_backend utility config create .
this will create a new config named nf_cloud.local.config.yaml
in the current directory. You can also print the configuration with python -m nf_cloud_backend utility config print
(useful for piping the results from a Docker container).
- Try to move Nextflow intermediate result folders into a subfolder.
- Add some more inputs
Radio
Select
- Add description to workflows
- Improve logging
- Save Nextflow stdout/stderr
- Enable report and show resulting HTML in web interface
- Rework docker deployment
- Investigate if the worker is able to run Nextflow in Docker containers when running in container itself