Enable Python bindings in conda-forge CI and fix Python bindings compilation and tests on MSVC

[traversaro]

• Fixed compilation of Python bindings on Windows on MSVC. The problem was that the pybind11 bindings for LeggedOdometry references some LeggedOdometry::resetEstimator overloads that actually belonged to the parent class FloatingBaseEstimator.
• Fixed loading of tests on MSVC by copying the BLF dll to the directory in which the Python binding dll is found, to avoid problem in loading blf dll as on Windows there is no RPATH
• Switch GitHub Actions conda CI to use Ninja as a generator on Windows, to workaround Python tests do not work on Multi-config CMake generators #382 . As the ninja MSVC generator only works on Command Prompt, we also switched the relevant steps to run on Command Prompt as well.

[traversaro]

MSVC is failing with this error:

2021-07-27T12:21:47.9708534Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(61,18): error C2672: 'operator __surrogate_func': no matching overloaded function found [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:47.9999984Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2784: 'unknown-type pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()(Return (__cdecl *)(const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &)) noexcept const': could not deduce template argument for 'Return (__cdecl *)(const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &)' from 'overloaded-function' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0036473Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(812): message : see declaration of 'pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0053597Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2782: 'unknown-type pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()(Return (__cdecl Class::* )(const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &),std::false_type) noexcept const': template parameter 'Class' is ambiguous [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0064292Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(816): message : see declaration of 'pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0067774Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): message : could be 'BipedalLocomotion::Estimators::FloatingBaseEstimator' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0330314Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): message : or       'BipedalLocomotion::Estimators::LeggedOdometry' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0347197Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2784: 'unknown-type pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()(Return (__cdecl Class::* )(const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &),std::false_type) noexcept const': could not deduce template argument for 'Return (__cdecl Class::* )(const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &)' from 'bool (__cdecl BipedalLocomotion::Estimators::LeggedOdometry::* )(const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &)' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0353054Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(816): message : see declaration of 'pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0357780Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2780: 'unknown-type pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()(Return (__cdecl Class::* )(const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &) const,std::true_type) noexcept const': expects 2 arguments - 1 provided [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0362263Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(820): message : see declaration of 'pybind11::detail::overload_cast_impl<const BipedalLocomotion::Estimators::FloatingBaseEstimators::InternalState &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0365942Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(64,18): error C2672: 'operator __surrogate_func': no matching overloaded function found [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0370616Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2784: 'unknown-type pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()(Return (__cdecl *)(const Eigen::Quaterniond &,const Eigen::Vector3d &)) noexcept const': could not deduce template argument for 'Return (__cdecl *)(const Eigen::Quaterniond &,const Eigen::Vector3d &)' from 'overloaded-function' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0374759Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(812): message : see declaration of 'pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0378663Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2782: 'unknown-type pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()(Return (__cdecl Class::* )(const Eigen::Quaterniond &,const Eigen::Vector3d &),std::false_type) noexcept const': template parameter 'Class' is ambiguous [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0382588Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(816): message : see declaration of 'pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0386024Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): message : could be 'BipedalLocomotion::Estimators::FloatingBaseEstimator' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0389874Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): message : or       'BipedalLocomotion::Estimators::LeggedOdometry' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0394895Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2784: 'unknown-type pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()(Return (__cdecl Class::* )(const Eigen::Quaterniond &,const Eigen::Vector3d &),std::false_type) noexcept const': could not deduce template argument for 'Return (__cdecl Class::* )(const Eigen::Quaterniond &,const Eigen::Vector3d &)' from 'bool (__cdecl BipedalLocomotion::Estimators::LeggedOdometry::* )(const Eigen::Quaterniond &,const Eigen::Vector3d &)' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0399442Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(816): message : see declaration of 'pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0403280Z D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\bindings\python\FloatingBaseEstimators\src\LeggedOdometry.cpp(34,54): error C2780: 'unknown-type pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()(Return (__cdecl Class::* )(const Eigen::Quaterniond &,const Eigen::Vector3d &) const,std::true_type) noexcept const': expects 2 arguments - 1 provided [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]
2021-07-27T12:21:48.0407395Z C:\Miniconda\envs\test\Library\include\pybind11\detail/common.h(820): message : see declaration of 'pybind11::detail::overload_cast_impl<const Eigen::Quaterniond &,const Eigen::Vector3d &>::operator ()' [D:\a\bipedal-locomotion-framework\bipedal-locomotion-framework\build\bindings\python\pybind11_blf.vcxproj]

[traversaro]

MSVC is failing with this error:

The problem seems that the pybind11 bindings for LeggedOdometry references some LeggedOdometry::resetEstimator overloads that actually belonged to the parent class FloatingBaseEstimator. This should be fixed by 9ddee6f .

[traversaro]

Now that tests are compiling, they were not running correctly. The problem was that in Windows there is no RPATH. This is tipically workarounded by placing all the .dll in the same directory, so dependent dll are found easily, but in this case the python bindings are placed in a different directory, so this created problem. After searching for ways for tweaking the PATH, the easiest solution was just to copy the blf .dll in the build directory of the bindings.

[traversaro]

Due to #382, I switched the Windows Conda CI from VS generator to Ninja.

[traversaro]

Due to #382, I switched the Windows Conda CI from VS generator to Ninja.

After this, the CI is finally green. This is now ready for review, if necessary I can rework the commit to match the three main changes summarized in first post, otherwise we can just merge with squash.

[traversaro]

@prashanthr05 your review was mainly for the commit 9ddee6f .

[diegoferigo]

I trust you and CI on Windows, I don't have better ideas about how to solve it nicely. I'm just wondering, would the tests succeed against the installed tree (inside the active site-packages folder) instead of the build tree stored in the build folder? Maybe in this case the PATH is already configured system-wide.

[traversaro]

I'm just wondering, would the tests succeed against the installed tree (inside the active site-packages folder) instead of the build tree stored in the build folder? Maybe in this case the PATH is already configured system-wide.

Yep, in that case everything would work fine also for Multi-Config generators, and the copy of dll will be not necessary, but also not problematic.

[diegoferigo]

I'm just wondering, would the tests succeed against the installed tree (inside the active site-packages folder) instead of the build tree stored in the build folder? Maybe in this case the PATH is already configured system-wide.

Yep, in that case everything would work fine also for Multi-Config generators, and the copy of dll will be not necessary, but also not problematic.

Ok clear, thanks! In Python ⩾ 3.8 I guess an alternative workaround would be using os.add_dll_directory pointing to the lib dir within the build folder.

[traversaro]

Ok clear, thanks! In Python ⩾ 3.8 I guess an alternative workaround would be using os.add_dll_directory pointing to the lib dir within the build folder.

I tried to use that, but it appeared not to work, not sure why (something related to conda builds?) or if I did something wrong. Tweaking the os.environ['PATH'] seems to work fine instead, but I did not found any easy way to inject such Python code easily in all the tests

[GiulioRomualdi]

Thank you @traversaro to deal with this

[GiulioRomualdi]

Could you please reorganise the commits has you wrote here #380 (comment) ?

[traversaro]

Could you please reorganise the commits has you wrote here #380 (comment) ?

Done!

[diegoferigo]

Ok clear, thanks! In Python ⩾ 3.8 I guess an alternative workaround would be using os.add_dll_directory pointing to the lib dir within the build folder.

I tried to use that, but it appeared not to work, not sure why (something related to conda builds?) or if I did something wrong. Tweaking the os.environ['PATH'] seems to work fine instead, but I did not found any easy way to inject such Python code easily in all the tests

Since the need of using this workaround for importing modules on Windows has been kind of reverted in conda, I would not be surprised that the usage of the function does not behave as it should. Thanks for the additional details!

[GiulioRomualdi]

The macos failure shouldn't be related to this PR. Am I correct?

[GiulioRomualdi]

Pcl version in brew updated 14 hours ago Homebrew/homebrew-core@b2e4d3b

[traversaro]

The macos failure shouldn't be related to this PR. Am I correct?

Exactly, the same failure is in the scheduled job: https://github.com/dic-iit/bipedal-locomotion-framework/actions/runs/1077070309 .

[GiulioRomualdi]

Since this PR is not related to the failure we decided to merge the PR