use binaryprovider to get Cairo libs [WIP]

[SimonDanisch]

At least locally this gives me:

julia> Pkg.test("Cairo")
INFO: Testing Cairo
Test Summary:   | Pass  Total
Image Surface   |    7      7
Test Summary:   | Pass  Total
Conversions     |    5      5
Test Summary:   | Pass  Total
TexLexer        |    1      1
Test Summary:   | Pass  Total
Samples         |   30     30
Test Summary:   | Pass  Total
Bitmap Painting |    6      6
Test Summary:   | Pass  Total
Vector Surfaces |    9      9
Test Summary:   | Pass  Total
Bitmap Surfaces |    3      3
Test Summary:   | Pass  Total
Assert/Status   |    7      7
Test Summary:   | Pass  Total
reset_transform |    6      6
INFO: Cairo tests passed

:)
This disables pango, though!
Pango looks easy to build, but I think I'm done for now with building more libs, so anyone else: just fire up the wizzard and follow http://www.linuxfromscratch.org/blfs/view/7.6/x/pango.html 1:1 ;)
Usually just takes a couple of minutes per dependency and can be done while doing other stuff ;)

Further discussion is needed to figure out what to do about the issues raised in #214 .

[SimonDanisch]

Picking up the discussion from #214

@tknopp wrote:

Gtk.jl has to be compatible with Cairo.jl. This won't work if they use different Cairo binaries. I would strongly prefer not doing 0.6 -> 0.7 transition simultaneously with the transition to BinaryBuilder. This BinaryBuilder thing looks like a massive effort to get fully done and this should happen in a coordinated fashion so that the user of the packages gets a smooth transition.

My answer to that:

Gtk.jl has to be compatible with Cairo.jl. This won't work if they use different Cairo binaries

One option would be to just get a clone of Cairo into GTK and use their binary/dep file in that case.
Or a require like solution, that includes the GTK deps file if present.

In any case, we obviously don't want to take away functionality from Cairo that people already rely on, so this will definitely not get merged if it reduces functionality.
If there is really no option and we won't get a solution with binary provider for the more complex backends, I guess we could do something like CairoLight as @davidanthoff proposed.
(In the context, that I don't want to have my packages depend on flaky binary dependencies anymore and for a Makie backend the Cairo library in this PR seems already to cover most functionality!)

[coveralls]

Coverage decreased (-7.7%) to 83.146% when pulling 9fcadcb on sd/binaryprovider into b8ea3a0 on master.

[tknopp]

How does the Cairo backend of Makie.jl draw something to the screen? Do you use GLFW for that?

If Makie.jl could be a replacement for Winston, then my first request will be getting this working with Gtk.jl ;-)

[SimonDanisch]

GLFW can be used for that, indeed ;) Or just opening a window and drawing pixel to it should be ~200 LOCS to cover most platforms... So there are quite a few options.

I see the main use case for cairo on systems without a display or for publications (svg/pdf), which won't need a window.

For people already invested in GTK, I restructured the OpenGL backend in a way to be able to not rely on loading GLFW (loading both segfaulted GTK). So for interactive graphics + gtk I see the future in OpenGL - for Makie at least.

[tknopp]

If you have a pure Cairo backend (?), then the most simple solution is to just combine this with Gtk. I don't see why this should be restricted to publication ready plots. People want both: use the package interactively and for publications. And Cairo is successfully being used (-> Winston) in such environments.

But maybe I don't fully understand the scope of Makie.jl. I thought that it targets being a replacement for packages like Winston and PyPlot, i.e. not only 3D as your previous packages.

[SimonDanisch]

My previous packages were never 3D only - the makie opengl backends supports (by now) mostly a super set of what Cairo offers ;) Since all backends in Makie are supposed to offer the same drawing capababilities and should only differ in platform availability + output formats, and OpenGL should be mostly available on systems that have GTK installed, this restricts cairos usability mostly to pdf/headless systems.

[tknopp]

No. On Mac Gtk does not provide an OpenGL backend and thus going through Cairo is the only option there.

I also look at this from the point of view of maintainability. If your package can do anything that Winston can (in a similar quality) Makie.jl could be a full replacement.

[SimonDanisch]

I can't find anything about this and the most relevant seems to be:
https://projects-old.gnome.org/gtkglext/ which lists osx as supported.
I also thought I got people to use Gtk + OpenGL - but not 100% sure.
Do you mean the Cairo OpenGL backend?

[tknopp]

JuliaGraphics/Gtk.jl#339

[SimonDanisch]

Oh yeah that... Jeez, looks like GTK is just trying to make me mad ;) Well in any case, I'd like to push the burdon away from Cairo purists, while not making the life of GTK users harder.

So a buildscript that checks if GTK is available and uses GTK's Cairo binary seems to be the most reasonable way forward.

Then your usecase should work pretty seamless, just make sure you have GTK installed before calling build("Cairo") and then Makie should work fine with GTK + Cairo.

[davidanthoff]

Hm, it seems really not ideal that the order in which build is called would matter... Plus, it seems quite brittle if Cairo.jl sometimes would use its own binaries and sometimes binaries shipped with Gtk.jl, that essentially means we would now have to always test everything twice etc.

I think I would almost prefer a CairoLight.jl solution... I had the same experience with Gtk.jl, it just causes so many problems (e.g. rendering the REPL more or less useless on Windows) and is something I really don't need for my use cases, that I would prefer something that just is completely separate from it.

But, if we were to create CairoLight.jl, would we have a potential problem that Cairo.jl and CairoLight.jl would each try to load shared libraries with the same name, that would actually be distinct?

[tknopp]

This has nothing to do with your Gtk.jl windows issue. That was not a build issue but it was related to how Gtk.jl handles the main loop. It makes no sense to have several Cairo packages.

But I actually do not see what BinaryBuilder aims to solve. Currently with BinDeps everything works fine and now it seems that people what to build a binary builder platform, which itself is a huge effort. The original issue raise by @SimonDanisch is pretty clear. Of course Cairo has to be build with pango support. And everything else that is required for depending packages. That this is a lot of work was clear in the first place when one decides to switch a package like Cairo to a binary builder platform.

[davidanthoff]

This has nothing to do with your Gtk.jl windows issue. That was not a build issue but it was related to how Gtk.jl handles the main loop. It makes no sense to have several Cairo packages.

Well, from my point of view I just want something that I can use to convert SVG stuff into other formats, and ideally I don't want to have dependencies on stuff that is not well maintained. Right now, Gtk.jl falls into the latter category from my point of view: a package that renders the REPL unusable simply by loading the package it is a bit of a disaster and I don't want to take a dependency on it... So from my point of view something that has no connection with Gtk.jl would be preferable. Plus, it seems that we might be able to get a pure CairoLight up and running with BinaryBuilder fairly quickly, which would also be a huge plus from my point of view.

Currently with BinDeps everything works

That is not correct. Win32 for example has been broken for a long time.

I'd still be interested to hear whether a CairoLight.jl would cause real problems somewhere, or whether a world with a Cairo.jl and a CairoLight.jl would actually work.

Oh, and I'm not sure about pango, that might actually be something I would also need for my use-case, but who knows... but it sounds as if that might be fairly easy to add.

[tknopp]

Cairo does not depend on Gtk. So there is absolutely not way how Gtk could destabilize Cairo. The only point here is that it does not make sense to build the Cairo library in such a way that Gtk cannot use it. The purpose of software collections (WinRPM, Homebrew, apt, yum, ...) is to package the binaries in such a way that they are compatible with each other. And they have solved it. You can install Cairo only and you then also install Gtk. And it will use the same Cairo binary.

CairoLight.jl pulls the complexity of binary building into user land. So you would need several flavors:

CairoLight.j, CairoLightButBuildWithPango.jl, CairoWithXButWithoutPango.jl, ...

[davidanthoff]

Cairo does not depend on Gtk. So there is absolutely not way how Gtk could destabilize Cairo.

I understood @SimonDanisch's suggestion from above such that this would change, i.e. Cairo.jl would use the binaries from Gtk.jl if Gtk.jl is installed. That is the scenario I'm not a fan of.

The purpose of software collections (WinRPM, Homebrew, apt, yum, ...) is to package the binaries in such a way that they are compatible with each other. And they have solved it.

I disagree with that, at least from a julia point of view. Homebrew.jl is a constant pain point and breaks regularly, WinRPM.jl also has its fair number of issues. It also seems pretty clear that julia as a whole is moving away from those solutions towards BinaryProvider.jl.

CairoLight.jl pulls the complexity of binary building into user land.

No, it would use BinaryProvider.jl, so that no actual building would happen on user machines.

So you would need several flavors

I'm not saying that a CairoLight.jl is ideal, but it would solve the situation for a number of use-cases.

[tknopp]

I am not disagreeing in principle that BinaryBuilder is a good thing. But if you replace the BinDeps solution with a BinaryBuilder solution, just introduce no regressions.

[SimonDanisch]

Sure, that's the crucial point of the whole discussion ;)

[bjarthur]

one can specify binary dependencies in BinaryBuilder. the GtK build_tarballs.jl could then specify CairoBuilder/build.jl, no?

[hessammehr]

Bumping this discussion as the Homebrew build of Cairo fails on mac OS 10.11 and earlier. I would be happy to lend a hand, e.g., with making builders for the other dependencies.

[davidanthoff]

So my understanding is that all of Gtk/Cairo/Rsvg needs to switch over to BinaryBuilder at the same time, and that we are missing the BinaryBuilder story for Gtk and Rsvg at this point.

I might be wrong, though :) If you managed to create these remaining builders, it would be awesome!

[tknopp]

Yes Cairo and Gtk are closely linked. These certainly need to switch at the same time. It just needs a volunteer to go ahead.

[hessammehr]

Thanks for the pointers - I'll have a crack at this tonight. Tentatively aiming for LibXML2Builder, PangoBuilder to boot.

[davidanthoff]

@hessammehr just wanted to cheer you on in trying to get this to work!! And please let us know about any updates! Fixing this would be sooo wonderful.

[hessammehr]

@davidanthoff Thank you for the encouragement! I haven't been able to spend as much time as I would like to on this. The blocking issue for me so far has been a linux-specific dependency (uuid) in one of the libraries (FontConfig if I remember correctly). I'll try to take a closer look tonight.

[davidanthoff]

@hessammehr Just wanted to cheer you on! Any updates?

[hessammehr]

@davidanthoff Terribly sorry for the lack of updates. I'm really hoping to take a closer look at the uuid issue later this week. Will keep you posted.

[davidanthoff]

Absolutely no need to apologize, it is awesome that you are taking a look in the first place!

[hessammehr]

Update: I've set up builders for libexpat and util-linux, expecting to have FontConfig building tonight.

[tknopp]

@hessammehr: Could you summarize somewhere, where all the builders live and how they are connected? I tried to get an overview about the current state bug the builders seem to be spread over several different repositories.

Have you looked into what it requires to build Gtk, once Cairo is done. Before we can do the switch, both need to be working.

[lobingera]

https://gitlab.gnome.org/GNOME/gtk/blob/master/README.md lists quite nicely the dependencies. However, the instructions for https://wiki.gnome.org/Projects/GTK%2B/OSX/Building doesn't look that nice and together with BB restrictions to build everything in cross compile ...

[tknopp]

Looks scary. But maybe one could have a look at the home-brew script (https://github.com/Homebrew/homebrew-core/blob/1e53748146209792f7a4f5cb3698984f97379ce6/Formula/gtk%2B.rb) which looks more or less as configure/make/ make install

@staticfloat: Do you think that BB can handle something like Gtk? In particular the dependencies for OSX and Linux are different. The former does not need X11.

[lobingera]

@tknopp You are aware that gtk 2.24 was released 2011?

[tknopp]

Version 2 is already released? I am still using 1.x ;-)

Sorry, I of course meant:

https://github.com/Homebrew/homebrew-core/blob/4b6c47c0386273b6350dfd448b7c325a051b86d3/Formula/gtk%2B3.rb

[tknopp]

By the way, gtk4 will be released soon and will not use autotools anymore

[hessammehr]

@davidanthoff Good news, I just managed to get FontConfig building! The Travis build is failing but we'll get it fixed.
@tknopp Here are the builders that I am aware of in terms of building Cairo.

hessammehr/FontconfigBuilder
SimonDanisch/LibpngBuilder
staticfloat/PixmanBuilder (a bit old)

I haven't started thinking carefully at the Cairo => Gtk roadmap yet. I suppose we'll need a few more libraries, e.g. Pango.

[tknopp]

As far as I have understood, the builders should be pooled here:

https://github.com/JuliaPackaging/Yggdrasil

There is for instance already a builder for Pixman. @staticfloat: Is that correct that the builders should move there?

@hessammehr: Would be great if you could have a look at Gtk as well, because Cairo.jl can only switch if Gtk.jl is migrated as well.

[mirestrepo]

Hi All - Any chance this thread could come back to life? Cairo hasn't been building for Mac OS users for a while now #271 ....this could help making the process more stable... so many libraries depend on Cairo ...🙏

[tknopp]

Yes, this is pretty frustrating. Gtk.jl is also not working on Mac anymore because of the Homebrew issues that constantly hit us. What we lack is someone experienced with BinaryBuilder that cares for Cairo / Gtk.

An alternative would be what @staticfloat did in JuliaIO/HDF5.jl#555
If I understand it correctly, the binaries are pulled from existing sources on Mac and Linux see https://github.com/JuliaPackaging/Yggdrasil/blob/master/HDF5/build_tarballs.jl
I don't know if this is more or less work than trying to build all the libs itself.

[hessammehr]

@mirestrepo I managed to make a little bit of progress on this issue back in April so I'll share my notes in case you would like to step in and give it a go.

There are build instructions for FreeType, Cario, Gtk available from the Linux from Scratch (LFS) project. Typically, you will find that there is a hierarchy of dependencies, e.g. Cairo depends on Fontconfig, which depends on FreeType and so on. Another source of inspiration, especially when it comes to building for the mac are the formulas used by the Homebrew package manager.

Many of the dependencies have builders set up for them on github already. For instance, if you google FreeTypeBuilder, it will take you here, so you won't have to build everything from scratch. :) I hope that you or someone else reading this finds the time to finally set up builders for Cairo and Gtk since they are used in so many places.

[SimonDanisch]

@tknopp if there are prebuild binaries, it should be fairly easy to make a BinaryProvider build.jl out of them... Just needs someone finding those urls, copy the hash, and your basically done.

[lobingera]

@SimonDanisch Now i'm getting slightly confused. Taking pre-compiled binaries from available sources sounds seriously like using a system binary provider like homebrew (and some repackaging) and the whole BinDeps2 story line was about NOT using provided binaries, do something smarter and cross-compile everything to provide a julia-matching-binary distribution of binaries.

[SimonDanisch]

The main goal of BinaryProvider + BinaryBuilder:

Package installation is merely downloading, verifying package integrity and extracting that tarball on the user's computer. No more compiling on user's machines. No more struggling with system package managers. No more needing sudo access to install that little mathematical optimization library.

Nothing in the philosophy says that the binaries MUST be compiled with BinaryBuilder - it's just the accompanying package that makes it much simpler to actually get those cross compiled binaries.

Homebrew gave us lots and lots and lots of struggle - so just downloading one version of a precompiled binary is much simpler regardless of who actually cross-compiled it. And not exactly the same as going over homebrew, which manages to error in many whimsical and unforeseeable ways when depending on it.

[lobingera]

Homebrew or Homebrew.jl? It's quite ironic to the read ...

"Why Package Authors should use Homebrew.jl
A common question is why bother with Homebrew formulae and such when a package author could simply compile the .dylib's needed by their package, upload them somewhere and download them to a user's installation somewhere."

I completely agree, that the BinDeps/Homebrew.jl/WinRPM.jl situation is not acceptable anymore. But the manpower behind system package managers is at least an order of magnitude higher than the manpower in the julia environment to create builders - especially in these cases of deep dependencies.

[SimonDanisch]

The thing is, the manpower behind Homebrew.jl is just one person (edit: one person who has given up on it), so it doesn't matter if there is lots of manpower behind homebrew... On the other hand, anyone can make a build.jl that downloads precompiled binaries.
Also, one can do hybrid solutions as in:
https://github.com/JuliaGL/GLFW.jl/blob/master/deps/build.jl
Or adapt the HDF5 approach... So there is really no use in complaining about the current state and hope that someone other than the person complaining will continue dealing with Homebrew.jl/BinDeps/WinRPM.. It won't happen, unless you do it! But it won't make much sense, when there are simple solutions not involving the mentioned packages within arm's reach.

[davidanthoff]

I think the idea to just grab the binaries that are pulled these days by WinRPM and Homebrew (or some older version of the homebrew binaries when things still worked) and hosting them ourselves and then using BinaryProvider.jl for the deployment story sounds like an excellent plan!

It can't get worse on Mac than the current situation, where simply everything is broken.

I still don't fully understand when one needs to use binaries that are compiled with exactly the same compiler that was used to compile julia itself. If we grab pre-existing binaries, that probably won't be the case, but maybe we'll just try and see whether it works anyways.

[tknopp]

I also vote for BP solution + grabbing the binaries for Mac (and maybe also windows?) For linux, I don't know if it would be better to remain with BinDeps because it works pretty smoothly (at least on debian based distros).

@SimonDanisch:

• Are you aware of a build script that does use system binaries on linux and downloads the binaries on Mac / Windows.
• Are you aware of "good" sources for a Cairo / Gtk binaries that can be downloaded. The binaries from Homebrew are nice because they are compiled with the quartz backend.
• I remember that there is an issue with the binaries not being relocatable on Mac. Do you know if this is an issue?

[rfourquet]

For linux, I don't know if it would be better to remain with BinDeps because it works pretty smoothly

Just as a datapoint, I'm on the Nixos Linux distribution and have a problem with installing Cairo.jl (the build phase fails), so am following this issue with interest!

[lobingera]

@rfourquet issue number?

[rfourquet]

@rfourquet Bug report?

I believe this to not be Cairo specific, I have this problem with many other non-pure-Julia packages, but AFAICT, those using BinaryProvider work well. Nixos is quite special in not following usual Linux conventions, for example /sbin/ldconfig doesn't exist (and creating a symlink there pointing to my system's ldconfig doesn't help).

[lobingera]

superceeded by #289