Add automatic generation of conda binary packages for Windows, macOS and Linux in robotology channel #652

traversaro · 2021-03-09T13:33:41Z

This PR fixes #620 . In particular, it adds machinery to generate the conda recipes for each subproject of the superbuild, and a GitHub Action job that build binaries once a week for all the generated recipes for the latest release of the robotology-superbuild projects and loads them in the robotology conda channel, that builds on top of the conda-forge similarly to who the robostack channel for ROS software build on top of conda-forge as well.

It is unfortunately a relatively large PR, but I preferred to experiment with it a bit to understand there is something functional before adding it to the official robotology-superbuild repo. In particular, the test binaries were generated in the channel https://anaconda.org/robotology-staging . The modifications can be largely divided in 5 main categories:

Changes in the conda directory, that contain all the machinery required to generate conda recipes out for all superbuild projects. These internal mechanism are described in doc/conda-recipe-generation.md document
The GitHub Action generate-conda-packages ( .github/workflows/generate-conda-packages.yaml ) that uses the machinery described before to actually compile the conda binaries and upload them to robotology conda channel
Modifications in the cmake/Build***.cmake scripts, that are basically just adding metadata required to generate conda packages
Modification in developer-oriented documentation in doc/developers-faqs.md (https://github.com/robotology/robotology-superbuild/pull/652/files#diff-f9b520063217385eaf8b07ab7805449c26b59ccb1158241d51497bff72e92100) that are oriented towards package and profile mantainers describing how to add the required conda metadata when they add a new package
Modification in user-oriented documentation in doc/conda-forge.md ( https://github.com/robotology/robotology-superbuild/pull/652/files#diff-58f523b9fc5050a95da9ce381d330a2cf3ad7f1c88e6fb407fad9fd87d09d091 ): this are meant for end-users of robotology software that want to install the robotology packages directly in binary format. As this is still experimental and the robotology channel is still empty, this documentation is still not linked in the main README.

Given the size of the PR, I think it would be ideal if reviewer could focus their attention to points 4. and 5. , while the first three points are mainly internal implementation details.

…and Linux in robotology channel

traversaro · 2021-03-09T15:12:53Z

As just a bunch of technical details is not a fun, this is a quick peek to what can be done now that we have (obviously all the instructions use robotology-staging until this PR is merged)

Use robotology-superbuild software built with ROBOTOLOGY_USES_MATLAB in MATLAB online

If you are a user of MATLAB online, you may want to run your scripts directly on the cloud, but this could be problematic if you depend on some MATLAB or Simulink libraries provided by the robotology-superbuild, given that MATLAB online does not permit you to install build tools packages via apt or other system package managers. Thanks to conda, you can easily install MATLAB libraries such as the YARP Matlab bindings of the robotology-superbuild simple with these MATLAB commands:

system('curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh')
system('sh Miniforge3-Linux-x86_64.sh -b')
system('~/miniforge3/condabin/conda install -c robotology-staging yarp-matlab-bindings -y')
addpath('~/miniforge3/mex')

(At this moment idyntree MATLAB bindings and wb-toolbox do not work due to #653, but this should be hopefully easy to fix).

Quickly install software on Windows

While the process of installing the robotology-superbuild and its dependencies has been historically relatively smooth on Ubuntu and macOS, it has always been quite complex on Windows. The typical pattern indeed was of Windows user that had to run Docker, a Virtual Machine or WSL just to install it. The conda binaries drastically simplify the process of installing robotology packages on Windows, for example if you have conda installed (see instructions in https://github.com/robotology/robotology-superbuild/blob/32fa9c33fb597b83aef936528e2d476576e96a5d/doc/install-miniforge.md) you just need to run the following commands:

conda create -n robsubenv
conda activate robsubenv
conda install -c robotology-staging icub-models gazebo-yarp-plugins

to be able to run Gazebo and an iCub simulation:

Easily run Continuous Integration of C++ libraries that depend on robotology libraries

Thanks to the conda binaries provided by the robotology channel, you can easily setup a continuous integration setup using GitHub Actions that runs on Windows, macOS and Linux without having to manually compile your dependencies or manually handling cache. For example, this is the script that is necessary to run CI for bipedal-locomotion-framework:

name: C++ CI Workflow with conda-forge dependencies

on:
  push:
  pull_request:
  schedule:
  # * is a special character in YAML so you have to quote this string
  # Execute a "nightly" build at 2 AM UTC 
  - cron:  '0 2 * * *'

jobs:
  build:
    name: '[${{ matrix.os }}@${{ matrix.build_type }}@conda]'
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        build_type: [Release]
        os: [ubuntu-latest, windows-latest, macOS-latest]
      fail-fast: false

    steps:
    - uses: actions/checkout@v2

    - uses: conda-incubator/setup-miniconda@v2
      with:
        mamba-version: "*"
        channels: conda-forge,defaults
        channel-priority: true

    - name: Dependencies
      shell: bash -l {0}
      run: |
        # Compilation related dependencies 
        mamba install cmake compilers make ninja pkg-config
        # Actual dependencies
        mamba install -c robotology-staging idyntree yarp matio-cpp lie-group-controllers eigen qhull casadi cppad manif spdlog catch2
        
    - name: Configure [Linux&macOS]
      if: contains(matrix.os, 'macos') || contains(matrix.os, 'ubuntu')
      shell: bash -l {0}
      run: |
        mkdir -p build
        cd build
        cmake -GNinja -DBUILD_TESTING:BOOL=ON -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} ..
      
    - name: Configure [Windows]
      if: contains(matrix.os, 'windows')
      shell: bash -l {0}
      run: |
        mkdir -p build
        cd build
        cmake -G"Visual Studio 16 2019" -DBUILD_TESTING:BOOL=ON -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} ..

    - name: Build 
      shell: bash -l {0}
      run: |
        cd build
        cmake --build . --config ${{ matrix.build_type }}
        
    - name: Test
      shell: bash -l {0}
      run: |
        cd build
        ctest --output-on-failure -C ${{ matrix.build_type }}

If you have a CMake project, the only line that you will need to modify to adapt it to your project is just the line:

mamba install -c robotology-staging idyntree yarp matio-cpp lie-group-controllers eigen qhull casadi cppad manif spdlog catch2

using your dependencies instead of the one of bipedal-locomotion-framework .

Quickly create small Docker images

The conda binaries provided by the robotology conda channel can also be used to create Docker images, that are able to quickly be ready as there is no need to recompile the libraries from source every time the Docker image is rebuilt.
See for example Deploying conda environments in (Docker) containers - how to do it right by Uwe Korn for a nice article on this.

traversaro · 2021-03-09T15:18:54Z

I also plan to eventually do in-person training on this new Conda-based tools to group of people, but before reaching that point I would prefer to have all the package generation machinery up and running.

traversaro · 2021-03-09T15:33:35Z

cc @Tobias-Fischer

Tobias-Fischer · 2021-03-09T21:18:55Z

cmake/BuildICUB.cmake

+if(NOT APPLE)
+  list(APPEND ICUB_CONDA_DEPENDENCIES freeglut)
+endif()
+if(CMAKE_SYSTEM_NAME STREQUAL "Linux")


I've often seen "UNIX AND NOT APPLE" - not sure which one is better.

UNIX AND NOT APPLE is typically used to include *bsd systems, as conda-forge do not support any *bsd or similar target at the moment I think it is more clear to indicate it as "do something if we are on linux". If conda-forge will ever support other non-Linux targets (such as WebAssembly ( pyodide/pyodide#795 ), that would be a dream!) we will eventually fix this.

Tobias-Fischer · 2021-03-09T21:19:46Z

cmake/BuildYARP.cmake

@@ -5,10 +5,6 @@
 include(YCMEPHelper)
 include(FindOrBuildPackage)

-find_package(ACE QUIET)


Out of interest, doesn't that break non-conda builds?

ACE needs to be installed in the system on all platforms supported by the robotology-superbuild. In general, in the superbuild build files there were a few of these find_package for dependencies that are not handled by the superbuild, but to avoid problems in the conda recipe generation process I just removed them for now, so that eventually we can add them back consistently for all packages.

Tobias-Fischer · 2021-03-09T21:22:33Z

conda/conda_build_config.yml

@@ -0,0 +1,3 @@
+# bipedal-locomotion-framework requires 10.13 for std::bad_cast_any
+MACOSX_DEPLOYMENT_TARGET:      # [osx]


If you use this, make sure to add the __osx run dependency: https://conda-forge.org/docs/maintainer/knowledge_base.html

Done in 05593fd .

Tobias-Fischer · 2021-03-09T21:23:32Z

conda/python/generate_conda_recipes_from_metadatadata.py

@@ -0,0 +1,139 @@
+import argparse


Mistake in filename? metadatadata?

Good catch.

Tobias-Fischer · 2021-03-09T21:24:59Z

conda/recipe_template/meta.yaml

+{% for dep in dependencies %}    - {{ dep }}
+{% endfor %}
+{# Handle specific packages required for gl on Linux, see https://conda-forge.org/docs/maintainer/knowledge_base.html?#libgl #}
+{% if require_opengl_linux %}    - xorg-libxfixes {% endif %}


Add the osx run stuff here I guess

Done in 05593fd .

Tobias-Fischer · 2021-03-09T21:25:34Z

doc/conda-forge.md

+
+If you want to develop some C++ code on the top of this libraries, it is recommended to also install the necessary compiler and development tools directly in the same environment:
+~~~
+conda installl compilers cmake pkg-config make ninja


Typo installl

Tobias-Fischer · 2021-03-09T21:25:55Z

doc/conda-recipe-generation.md

+## Rationale
+
+See the [documentation on conda-forge and the robotology-superbuild](conda-forge.md) for a general discussion on how the robotology-superbuild and conda can work together. 
+This document describes how to 


Ran out of energy there? :)

Yes. Fixed in f0c615e .

Tobias-Fischer · 2021-03-09T21:26:28Z

doc/developers-faqs.md

+* If your package needs to set or modify some environment variables to work correctly, it should provide a pair [multisheller](https://github.com/wolfv/multisheller) scripts name `<condaPkgName>_activate.msh` and `<condaPkgName>_activate.msh` in the `conda/multisheller` directory to describe how the environment should be modified, see the existing scripts for more details.
+
+
+## How ofter are conda binary packages generated?


ofter -> often

Fixed in cfe30cd .

Tobias-Fischer · 2021-03-09T21:27:33Z

Amazing work, well done @traversaro. This is super exciting :)

…recipes_from_metametadata.py

traversaro · 2021-03-11T14:39:56Z

Amazing work, well done @traversaro. This is super exciting :)

Thanks a lot for the feedback @Tobias-Fischer ! I should have fixed all of them.

doc/developers-faqs.md

doc/conda-forge.md

doc/developers-faqs.md

Co-authored-by: Nuno Guedelha <[email protected]>

nunoguedelha · 2021-03-23T20:48:51Z

Sorry @traversaro , there is a minor typo I forgot to mention, and actually is not among the modified files. It's in https://github.com/robotology/robotology-superbuild/blob/master/doc/install-miniforge.md#install-1:

# Download
curl -fsSLo Miniforge3.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-$(uname -m).sh
# Install with default options
bash ./Miniforge3-MacOSX-$(uname -m).sh

Second line should be bash ./Miniforge3.sh.

traversaro · 2021-03-24T07:58:49Z

Sorry @traversaro , there is a minor typo I forgot to mention, and actually is not among the modified files. It's in https://github.com/robotology/robotology-superbuild/blob/master/doc/install-miniforge.md#install-1:
# Download
curl -fsSLo Miniforge3.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-$(uname -m).sh
# Install with default options
bash ./Miniforge3-MacOSX-$(uname -m).sh
Second line should be bash ./Miniforge3.sh.

Fix for this proposed in #664 .

traversaro · 2021-03-24T08:00:11Z

Thanks @nunoguedelha !

As robotology/robotology-superbuild#652 has been merged, we now have officially generated packages coming from the robotology-superbuild in the [robotology anaconda channel](https://anaconda.org/robotology), so we can switch to use that instead of the experimental and test-only `robotology-staging`.

traversaro force-pushed the submit_conda_upstream branch from 040f529 to da7265a Compare March 9, 2021 13:40

Add automatic generation of conda binary packages for Windows, macOS …

d520cb2

…and Linux in robotology channel

traversaro force-pushed the submit_conda_upstream branch from da7265a to d520cb2 Compare March 9, 2021 14:08

traversaro added 3 commits March 9, 2021 15:08

fixup

fa364e8

Update ci.yml

15b5647

Update CondaGenerationOptions.cmake

5078361

traversaro changed the title ~~[WIP] Add automatic generation of conda binary packages for Windows, macOS and Linux in robotology channel~~ Add automatic generation of conda binary packages for Windows, macOS and Linux in robotology channel Mar 9, 2021

Update Buildwalking-controllers.cmake

d504067

Tobias-Fischer reviewed Mar 9, 2021

View reviewed changes

traversaro and others added 7 commits March 10, 2021 10:30

Update conda_build_config.yml

553bd5d

Update meta.yaml

05593fd

Update conda-forge.md

903aa77

Rename generate_conda_recipes_from_metadatadata.py to generate_conda_…

c989941

…recipes_from_metametadata.py

Update conda-recipe-generation.md

f0c615e

Update developers-faqs.md

cfe30cd

Merge branch 'master' into submit_conda_upstream

2b0d329

traversaro requested a review from nunoguedelha March 11, 2021 17:13

traversaro mentioned this pull request Mar 20, 2021

Allow building the superbuild with "Shallow" cloned repos #662

Closed

nunoguedelha requested changes Mar 23, 2021

View reviewed changes

doc/developers-faqs.md Outdated Show resolved Hide resolved

doc/conda-forge.md Outdated Show resolved Hide resolved

doc/conda-forge.md Outdated Show resolved Hide resolved

doc/developers-faqs.md Outdated Show resolved Hide resolved

Apply suggestions from code review

7b1fa58

Co-authored-by: Nuno Guedelha <[email protected]>

traversaro requested a review from nunoguedelha March 23, 2021 20:07

nunoguedelha approved these changes Mar 23, 2021

View reviewed changes

traversaro merged commit 1600741 into robotology:master Mar 24, 2021

traversaro deleted the submit_conda_upstream branch March 24, 2021 08:00

traversaro mentioned this pull request Mar 24, 2021

Add option to run generate-conda-packages job without uploading and fix generate-conda-packages #665

Merged

nunoguedelha mentioned this pull request Mar 26, 2021

Run the simulator with the robotology binaries installed through Conda ami-iit/matlab-whole-body-simulator#19

Closed

4 tasks

traversaro mentioned this pull request Mar 30, 2021

conda-forge-ci: Switch to use official robotology channel ami-iit/bipedal-locomotion-framework#263

Merged

traversaro mentioned this pull request Mar 30, 2021

conda-forge-ci: Switch to use official robotology channel robotology/idyntree#842

Merged

nunoguedelha mentioned this pull request Apr 2, 2021

Run the simulator in MATLAB online with the robotology binaries installed through Conda ami-iit/matlab-whole-body-simulator#21

Closed

lrapetti mentioned this pull request May 13, 2021

Replace Windows CI with conda-based CI robotology/wearables#114

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automatic generation of conda binary packages for Windows, macOS and Linux in robotology channel #652

Add automatic generation of conda binary packages for Windows, macOS and Linux in robotology channel #652

traversaro commented Mar 9, 2021 •

edited

Loading

traversaro commented Mar 9, 2021 •

edited

Loading

traversaro commented Mar 9, 2021

traversaro commented Mar 9, 2021

Tobias-Fischer Mar 9, 2021

traversaro Mar 11, 2021

Tobias-Fischer Mar 9, 2021

traversaro Mar 10, 2021

Tobias-Fischer Mar 9, 2021

traversaro Mar 11, 2021

Tobias-Fischer Mar 9, 2021

traversaro Mar 10, 2021

Tobias-Fischer Mar 9, 2021

traversaro Mar 10, 2021

Tobias-Fischer Mar 9, 2021

Tobias-Fischer Mar 9, 2021

traversaro Mar 11, 2021

Tobias-Fischer Mar 9, 2021

traversaro Mar 11, 2021

Tobias-Fischer commented Mar 9, 2021

traversaro commented Mar 11, 2021

nunoguedelha commented Mar 23, 2021

traversaro commented Mar 24, 2021

traversaro commented Mar 24, 2021

		@@ -0,0 +1,3 @@
		# bipedal-locomotion-framework requires 10.13 for std::bad_cast_any
		MACOSX_DEPLOYMENT_TARGET: # [osx]

		* If your package needs to set or modify some environment variables to work correctly, it should provide a pair [multisheller](https://github.com/wolfv/multisheller) scripts name `<condaPkgName>_activate.msh` and `<condaPkgName>_activate.msh` in the `conda/multisheller` directory to describe how the environment should be modified, see the existing scripts for more details.


		## How ofter are conda binary packages generated?

Add automatic generation of conda binary packages for Windows, macOS and Linux in robotology channel #652

Add automatic generation of conda binary packages for Windows, macOS and Linux in robotology channel #652

Conversation

traversaro commented Mar 9, 2021 • edited Loading

traversaro commented Mar 9, 2021 • edited Loading

Use robotology-superbuild software built with ROBOTOLOGY_USES_MATLAB in MATLAB online

Quickly install software on Windows

Easily run Continuous Integration of C++ libraries that depend on robotology libraries

Quickly create small Docker images

traversaro commented Mar 9, 2021

traversaro commented Mar 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tobias-Fischer commented Mar 9, 2021

traversaro commented Mar 11, 2021

nunoguedelha commented Mar 23, 2021

traversaro commented Mar 24, 2021

traversaro commented Mar 24, 2021

traversaro commented Mar 9, 2021 •

edited

Loading

traversaro commented Mar 9, 2021 •

edited

Loading