CMake: Bringing back ArcusConfig.cmake generation #138

thopiekar · 2022-04-26T19:45:08Z

I don't see why it has been removed, but I need it to build Cura myself.

thopiekar · 2022-04-26T19:55:49Z

Just noticed I need to put more time into bringing this back..

thopiekar · 2022-04-26T20:06:48Z

That looks better for me now.. 😇

jellespijker · 2022-04-26T20:33:34Z

@thopiekar can you explain why you need it.
We are switching to Conan, see the readme in
https://github.com/Ultimaker/libArcus/tree/CURA-9177_fix_CI_CT

And I want to keep the CMake logic for to a bare minimum. We will be using Conan to consume this package. Otherwise we will have to maintain multiple methods.

Imo CMake is great for compiling and linking, but not so much for dependencies and installing. Conan handles those much better.

I have a feeling that Conan could serve you as well. So if you could explain what your workflow is maybe I can accommodate that in our new workflow as well.

jellespijker · 2022-04-26T20:40:03Z

For an example in a consumer side look at the workflow in Curaengine https://github.com/Ultimaker/CuraEngine/blob/CURA-9177_Fix_CI_CT/.github/workflows/conan-package.yml

It is now possible to install the (specific/latest) Arcus package with a single comand and consume it downstream.

conan install arcus/latest@ultimaker/stable -pr:b cura_build.jinja -pr:h cura_release.jinja --build=missing

Should take care of installing Arcus, Protobuf and zlib and generate the cake find packages and modules.

Keep in mind that linking is now done with arcus::libarcus in cmake.

jellespijker · 2022-04-26T21:09:46Z

Check the readme of that Curaengine branch for more information

thopiekar · 2022-04-26T21:22:19Z

Well, letting conan generate stuff that is missing in our CMake projects is something is really dislike.
The whole point of Flatpaks is to build everything from source. So using conan to download the software itself and the dependency chain is just nothing the Freedesktop folks want.
Sure, I allow flatpak-builder to allow network connections during build, but as I said, I won't make it this way into flathub.

So would prefer to either let CMake do the job and let it generate the files needed for further development or call conan to generate them for us during target build.

Don't get me wrong it is really great that someone is reviewing all the build scripts after years and I saw good improvements e.g. installing libArcus (or other projects - I might mix things up here) to an prefix, but having these Find*.cmake or *Config.cmake files missing is a real pain.

thopiekar · 2022-04-26T21:23:21Z

Additionally: Same applies for PPA builds, which I plan to drop to support in any way.. During builds via launchpad.net your build is disconnected from the internet.

jellespijker · 2022-04-27T14:18:57Z

@thopiekar good points. I am wondering if CMake and the compiler also build from source and all the python dependencies are also build from source. Or that you build the Cura C++ dependencies from source to ensure binary compatibility?

I don't know how much you known about Conan so I might be explaining stuff you already know. But the way it works is that it looks in the requirements specified in conanfile.txt or conanfile.py (in our case). It then checks the local package repository (usually found in ~/.conan/data) for a conan package which correspondence with the names and versions specified by our project. If it can't be found locally it can look online for the recipes. But this can be disabled if required/wanted.
Normally if the recipe can be downloaded from the conan-center https://conan.io/center/ or in our case from our own Artfifactory server. It will obtain it from there.

If will then generate a unique hash, depending on the OS, Architecture, Compiler and package options. and it will look if the package was previously build locally, if not it can look online in the above mentioned remotes. If it finds the binaries there it can download them, if not these dependencies will be build locally. To make sure they have the specified OS, Architecture, Compiler and options. But if they are online you can download them if you want to. This is the reason for a huge speed up in our build setup. It will only build if required, if they are build we'll upload them to our Artifactory such that they can be used by others or subsequent runs.

When Conan build the binaries it acts as nothing more than a wrapper. A CMake project such as arcus will just call the CMake configure and CMake build commands. Build artifacts, source files and resources are then packaged by Conan. With Additional information about the Compiler flags, environmental settings used and the used dependencies.

That package information is used by Conan to generate the information for the build systems in consuming packages. Such as
CMake find modules if the consumer is a CMake module as is used by most of our projects or modules for Autotools, Visual Studio, XTools, etc. It uses all the information above together with that unique has for a binary, such as mnt/projects/conan/.conan/data/arcus/5.0.0+46/ultimaker/stable/package/e659312091e0e7ff631f370c2375087144bf3122/ to ensure binary compatibility.

But it can also auto generate a virtual environment such as the one below

echo Capturing current environment in "/mnt/projects/ultimaker/cura/hackaton_22/uranium/venv/bin/deactivate_activate_uranium.sh"
echo echo Restoring environment >> "/mnt/projects/ultimaker/cura/hackaton_22/uranium/venv/bin/deactivate_activate_uranium.sh"
for v in PYTHONPATH LD_LIBRARY_PATH DYLD_LIBRARY_PATH PATH DYLD_FRAMEWORK_PATH VIRTUAL_ENV PYTHONHOME VIRTUAL_ENV_PROMPT
do
    value=$(printenv $v)
    if [ -n "$value" ]
    then
        echo export "$v='$value'" >> "/mnt/projects/ultimaker/cura/hackaton_22/uranium/venv/bin/deactivate_activate_uranium.sh"
    else
        echo unset $v >> "/mnt/projects/ultimaker/cura/hackaton_22/uranium/venv/bin/deactivate_activate_uranium.sh"
    fi
done

echo Configuring environment variables

export PYTHONPATH="/mnt/projects/ultimaker/cura/hackaton_22/uranium/venv/lib/python3.10/site-packages:$PYTHONPATH:/mnt/projects/conan/.conan/data/arcus/5.0.0+46/ultimaker/stable/package/e659312091e0e7ff631f370c2375087144bf3122/site-packages"
export LD_LIBRARY_PATH="/mnt/projects/conan/.conan/data/arcus/5.0.0+46/ultimaker/stable/package/e659312091e0e7ff631f370c2375087144bf3122/site-packages:/mnt/projects/conan/.conan/data/arcus/5.0.0+46/ultimaker/stable/package/e659312091e0e7ff631f370c2375087144bf3122/lib:/mnt/projects/conan/.conan/data/protobuf/3.17.1/_/_/package/ffdf9e0e1cab584f18f68896558053e3df900413/lib:/mnt/projects/conan/.conan/data/zlib/1.2.12/_/_/package/dfbe50feef7f3c6223a476cd5aeadb687084a646/lib:$LD_LIBRARY_PATH"
export DYLD_LIBRARY_PATH="/mnt/projects/conan/.conan/data/arcus/5.0.0+46/ultimaker/stable/package/e659312091e0e7ff631f370c2375087144bf3122/site-packages:/mnt/projects/conan/.conan/data/arcus/5.0.0+46/ultimaker/stable/package/e659312091e0e7ff631f370c2375087144bf3122/lib:/mnt/projects/conan/.conan/data/protobuf/3.17.1/_/_/package/ffdf9e0e1cab584f18f68896558053e3df900413/lib:/mnt/projects/conan/.conan/data/zlib/1.2.12/_/_/package/dfbe50feef7f3c6223a476cd5aeadb687084a646/lib:$DYLD_LIBRARY_PATH"
export PATH="/mnt/projects/ultimaker/cura/hackaton_22/uranium/venv/bin:/mnt/projects/conan/.conan/data/protobuf/3.17.1/_/_/package/ffdf9e0e1cab584f18f68896558053e3df900413/bin:$PATH"
export DYLD_FRAMEWORK_PATH="$DYLD_FRAMEWORK_PATH"
export VIRTUAL_ENV="/mnt/projects/ultimaker/cura/hackaton_22/uranium/venv"
unset PYTHONHOME
export VIRTUAL_ENV_PROMPT="(uranium)"

I'm not that familiar with flatpack generation or building for PPA. Since I run on an Arch-based distro.
Can you describe the required steps?
I assume it will be something similar like this:

download, build and install zlib
download, build and install protobuf
download, build and install python, with and do the same for its dependencies
- zlib
- openssl
- libffi
- mpdecimal
- libuuid
- libxcrypt
- bzip2
- gdbm
- sqlite3
- tk
- ncurses
- libdb
- xz_utils
download libarcus
configure libarcus using cmake
- pointing towards the previously installed protobuf using Protobuf_ROOT=
- pointing towards the previously installed python
build libarcus (assuming you use GCC with make or Ninja
make install
download curaengine
configure curaengine
- pointing towards the previously installed protobuf using Protobuf_ROOT =
- pointing towards the previously installed protobuf using Arcus_ROOT =
build curaengine (assuming you use GCC with make or Ninja
make install
.... en then of course for all of Cura and Uranium deps the same
Then install the Python dependencies from source or do you download them from pypi?
Once these dependencies are installed pyinstaller is called to gather all binaries and python modules
These files are then collected by you and archived in a Flatpack or RPM

Is the reason that you want to build everything from source because you want to ensure binary compatibility? If so Conan takes care of that.

It offers solution for generating a Flat Package https://docs.conan.io/en/latest/integrations/deployment/flatpak.html#deployment-flatpak
or other system package managers https://docs.conan.io/en/latest/integrations/deployment/system_package_manager.html

Once I got it up and running for Cura (and therefore all of her dependencies)
Creating a flat package should be as simple as running the command below:

conan install cura/5.1.0@/ultimaker/stable -pr:b cura_build.jinja -pr:h cura_release.jinja --build=missing -g json

this will create a json file with all the relative information which can then be used to collect and package everything

And Yes, you can do the above command in such away that every Conan managed dependency is build from source each time

I want to stretch that I'm not opposed to bringing back the old ArcusConfig, but I want to guard against us having to support multiple ways of installing and managing dependencies. This is in my opinion the reason why we had such trouble (3 months) of upgrading to PyQt6 on our build server. While I got it almost up and running in less in a couple of days when I switch to a complete Conan set-up.

thopiekar · 2022-04-27T16:35:37Z

That's very interesting! I'll definitely look into how the flatpak generation works.
However, from what I see quickly, it will build the software during first run and use these files for later execution. However, flatpak is meant to provide all files in a package - just like a AppImage, but in a containerized way of thinking.
However, that's my first impression and I'll take a closer look later!

Many thanks for keeping in touch on this and for your interest!

thopiekar · 2022-04-27T16:40:32Z

PS: I noticed for clipper and stb, that conan is providing these cmake files I mentioned before. As previously said, I would prefer to use whatever works with the original project, since these files are injected by conan and thus building the whole thing will only work with conan this way, which I don't think is wise.
On the other hand, having external packages in Cmake project is also dumb because you need to rebuild everything eventually.
But yeah, I need to know more about conan to be on the same level as you are at the moment. Very essential to talk about pros and cons 🙂

jellespijker · 2022-04-27T18:56:28Z

@nallath fyi

We should also discuss maintaining multiple installation methods next Monday.

jellespijker · 2022-04-27T19:03:54Z

The most pragmatic solution for now is to merge this PR, since that would allow you to keep your current way of working and once everything is finished you could check if Conan and it's deploy generator could also serve you.

Which I believe it will, because I'm planning to create an AppImage generator, NSIS and DMG generator.

But keep in mind that you probably need to patch things up downstream, such as Curaengine. Since we change the CMake targets and namespace there to all lowercase.
Maintaining multiple naming schemes brings an unwanted complexity to those CMakeLists.txt which I for one do not want to maintain.

jellespijker · 2022-04-27T19:05:02Z

But I will discuss things with @nallath next Monday and I will come back to you

jellespijker · 2022-04-27T19:25:15Z

The clipper recipe shows indeed that it uses a CMakeLists.txt, but this is nothing more then making sure the settings provided by Conan can be used.

The add_subdirectory still includes the original CMakeLists.txt in the source_folder for the cop module.

https://github.com/conan-io/conan-center-index/blob/master/recipes/clipper/all/conanfile.py

thopiekar · 2022-04-27T19:44:32Z

Sure, but give me some time to complete my Flatpak PoC. Doesn't make sense if I don't get my work done, you merge my stuff and mess up your workflow (eventually) for no good reason.
Don't feel any rush. I'm just the kind of person if I have any finished bits ready - which might be in the interest of others - to share them as a PR 😉

krop · 2022-06-20T16:48:52Z

Thanks for this PR, it made me gain some time for my libArcus package. A couple more things are needed to still have a working build system when conan is useless

Installing headers:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6bb4a73..ddf28bd 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -16,14 +16,6 @@ set(arcus_SRCS
     src/Error.cpp
 )
 
-set(arcus_HDRS
-    src/Socket.h
-    src/SocketListener.h
-    src/Types.h
-    src/MessageTypeStore.h
-    src/Error.h
-)
-
 set(ARCUS_VERSION 5.0.0)
 set(ARCUS_SOVERSION 5)
 
@@ -80,6 +72,10 @@ install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ArcusConfig.cmake
         DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/Arcus
 )
 
+install(DIRECTORY arcus_include/Arcus
+        DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
+)
+
 # Create the Python bindings
 if(BUILD_PYTHON)
     list(APPEND CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/cmake)

and fix the versioning which got broken by the recent changes:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 69e62ab..2f44d0a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -27,6 +27,10 @@ set(ARCUS_SOVERSION 5)
 
 if(BUILD_SHARED_LIBS)
     add_library(Arcus SHARED ${arcus_SRCS})
+    set_target_properties(Arcus PROPERTIES
+        VERSION ${ARCUS_VERSION}
+        SOVERSION ${ARCUS_SOVERSION}
+    )
 else()
     add_library(Arcus STATIC ${arcus_SRCS})
 endif()

CMake: Bringing back ArcusConfig.cmake generation

7301cd1

I don't see why it has been removed, but I need it to build Cura myself.

thopiekar changed the title ~~CMake: Bringing back ArcusConfig.cmake generation~~ [WIP] CMake: Bringing back ArcusConfig.cmake generation Apr 26, 2022

thopiekar force-pushed the patch-1 branch from 508ab14 to 7301cd1 Compare April 26, 2022 20:05

thopiekar changed the title ~~[WIP] CMake: Bringing back ArcusConfig.cmake generation~~ CMake: Bringing back ArcusConfig.cmake generation Apr 26, 2022

jellespijker mentioned this pull request Jul 8, 2022

Introduce Conan, JFrog and GitHub Actions to build and manage Cura (and her dependencies Ultimaker/Cura#12708

Merged

41 tasks

Joeydelarago closed this in Ultimaker/Cura#12708 Jul 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMake: Bringing back ArcusConfig.cmake generation #138

CMake: Bringing back ArcusConfig.cmake generation #138

thopiekar commented Apr 26, 2022

thopiekar commented Apr 26, 2022

thopiekar commented Apr 26, 2022

jellespijker commented Apr 26, 2022 •

edited

Loading

jellespijker commented Apr 26, 2022 •

edited

Loading

jellespijker commented Apr 26, 2022

thopiekar commented Apr 26, 2022

thopiekar commented Apr 26, 2022

jellespijker commented Apr 27, 2022 •

edited

Loading

thopiekar commented Apr 27, 2022

thopiekar commented Apr 27, 2022

jellespijker commented Apr 27, 2022

jellespijker commented Apr 27, 2022

jellespijker commented Apr 27, 2022

jellespijker commented Apr 27, 2022

thopiekar commented Apr 27, 2022

krop commented Jun 20, 2022

CMake: Bringing back ArcusConfig.cmake generation #138

CMake: Bringing back ArcusConfig.cmake generation #138

Conversation

thopiekar commented Apr 26, 2022

thopiekar commented Apr 26, 2022

thopiekar commented Apr 26, 2022

jellespijker commented Apr 26, 2022 • edited Loading

jellespijker commented Apr 26, 2022 • edited Loading

jellespijker commented Apr 26, 2022

thopiekar commented Apr 26, 2022

thopiekar commented Apr 26, 2022

jellespijker commented Apr 27, 2022 • edited Loading

thopiekar commented Apr 27, 2022

thopiekar commented Apr 27, 2022

jellespijker commented Apr 27, 2022

jellespijker commented Apr 27, 2022

jellespijker commented Apr 27, 2022

jellespijker commented Apr 27, 2022

thopiekar commented Apr 27, 2022

krop commented Jun 20, 2022

jellespijker commented Apr 26, 2022 •

edited

Loading

jellespijker commented Apr 26, 2022 •

edited

Loading

jellespijker commented Apr 27, 2022 •

edited

Loading