Package request: Revamp X11 Feedstocks, X.Org Servers including Xvfb, X11 Utility Apps

[ehfd]

This task should be done in conjunction to renovating the X11 feedstocks across conda-forge.

Guidelines for X11 package renovation from @pkgw and @ehfd for each X11 feedstock:

• Update Automake to m2-automake1.16 (in both meta.yaml and build.sh)
• Remove very old MSYS2_ARG_CONV_EXCL workaround in bld.bat
• Remove very old IFS workaround in build.sh
• Remove all other legacy clutter in build.sh and bld.bat
• Update the aclocal include path given to autoreconf
• Add run exports if they are not already added
• Split packages so that development components are moved to *-devel packages and the current package name only contains runtime components
• Migrate feedstocks to depend on xorg-xorgproto from legacy xorg-*proto packages; but using xorg-xorgproto prematurely when the packages are not formally migrated from the deepest dependency to the shallowest one WILL error out during builds of shallower dependencies in a fatal way (potentially in any package depending on X11) - This is a possible reason to initally use a development branch instead of conda-forge
• Switch to MSVC compilers, OR incorporate new updated MinGW compilers: change compiler('m2w64_c') to compiler('c'), etc. - Use the newer MSYS2 epoch with the newer MinGW, mostly compatible with GCC, or need to address various errors of the X11 codebase that is only developed for GCC and not MSVC, such as LLP64 vs LP64 datatypes, or https://mesonbuild.com/FAQ.html#my-project-works-fine-on-linux-and-mingw-but-fails-to-link-with-msvc-due-to-a-missing-lib-file-fatal-error-lnk1181-why (Windows only)
• Find out a way to not have to patch/reduce the patches for fork(), checkpid(), mmap()/munmap()/mprotect(), and other POSIX capabilities frequently used in X11 packages, where the X.Org developers had no consideration for Windows environments and MSVC - Investigate the capabilities of the newer MSYS2 and MinGW versions as Cygwin is known to have capabilities for this (Windows only)
• Remove posix dependency; add m2-{grep,m4} (Windows only)
• Change make dependency to m2-make (Windows only)
• Change m2w64-pkg-config dependency to m2-pkg-config or pkg-config (Windows only)

Update: conda-forge/xorg-libxcvt-feedstock#4 is an example of using the newest MinGW compiler pipeline. This can pretty much likely be the way forward to reduce the amount of patches as much as possible.

Note:
https://anaconda.org/search?q=mingw-w64-ucrt-x86_64-gcc
conda-forge/xorg-libxmu-feedstock#10 (comment)
conda-forge/xorg-makedepend-feedstock#9 (comment)
conda-forge/conda-forge.github.io#1891 (comment)
conda-forge/xorg-libxfixes-feedstock#8 (comment)

Update by @pkgw:
conda-forge/libglvnd-feedstock#2 (comment)

I believe there are about 33 legacy proto packages, based on:

https://www.x.org/releases/individual/proto/

Everything except xcb-proto is all merged into xorg-xorgproto now. It's also worth mentioning that the old proto packages are no longer getting updated, so this migration is necessary. There was a case where a library wouldn't build (I think it was libxinput) because it needed newer prototypes that were provided in the xorgproto version of the input prototypes, but not in the inputproto version.

Another factor: before, because there were so many proto packages I added them as run dependencies of the xorg-libx* library packages, since it would be a big pain to have to remember "OK, since I'm building against xorg-libxaw3d, I need to also depend on xorg-inputproto, xorg-xproto, xorg-fixproto, ....".

Now that there's basically only one package, I got rid of those run dependencies. So, packages that have host deps on xorg-libx* libraries will probably need to add a host dep on xorg-xorgproto as well. This is another thing that it would be nice to have a linter/migration check for.

Oh, and, at this point I believe that the core xorg-* packages are all done migrating to the new UCRT Windows runtime + xorgproto standard. I haven't checked the migration status for a while but I've stopped seeing PRs to those packages, and I did a pass over the migration status a couple of weeks ago and there weren't any xorg-* packages left in the queue.

Package name

• x11-xserver-xvfb (https://gitlab.freedesktop.org/xorg/xserver, set Meson build options to Xvfb, currently a CDT package)
• x11-xserver-* (https://gitlab.freedesktop.org/xorg/xserver, required for a full X11 stack on Conda, currently a CDT package - refer to https://formulae.brew.sh/formula/xorg-server)
• x11-apps (List: https://packages.debian.org/sid/x11-apps, ~27 binaries, none provided by Conda)
• x11-utils (List: https://packages.debian.org/sid/x11-utils, ~17 binaries, none provided by Conda)
• x11-xserver-utils (List: https://packages.debian.org/sid/x11-xserver-utils, ~16 binaries, none provided by Conda)
• x11-session-utils (List: https://packages.debian.org/sid/x11-session-utils, 3 binaries, none provided by Conda)
• xfonts-utils (List: https://packages.debian.org/sid/xfonts-utils, 6 binaries, none provided by Conda)
• x11-xfs-utils (List: https://packages.debian.org/sid/x11-xfs-utils, 4 binaries, none provided by Conda)
• virtualgl (https://github.com/VirtualGL/virtualgl/blob/main/BUILDING.md, headless GLX through EGL or hardware X11 server API, requires x86 libraries to run x86 apps?)

More to be added.

Package version

Newest

Package website

https://www.x.org/releases/individual/app/

https://www.x.org/releases/individual/util/

Package availability

https://www.x.org/releases/individual

https://gitlab.freedesktop.org/xorg/app

https://gitlab.freedesktop.org/xorg/util

https://salsa.debian.org/xorg-team/app

Additional comments

These are many numbers of command-line or GUI applications from X.org which are typically packaged in Debian or RHEL as a group of packages, such as x11-apps, x11-utils, x11-xserver-utils, and xfonts-utils.

Instead of creating >73 different feedstocks for iceauth, xrandr, xhost, xset, etc... (of which none are currently in feedstocks) it might be a good idea to build them all in one shot together and package them in categories like how Debian or RHEL does, reducing the number of feedstocks to <10.

But on the other hand, there are a lot of demo X11 applications that no feedstock would depend on in the above list, so it would be worth omitting those. But the most critical ones are the ones such as iceauth, xrandr, xhost, xset, xdpyinfo, etc., which many applications depending on X11 actively utilize automatically.

Moreover, it would be less demanding to first limit the builds of utils and apps to Linux or Mac, unlike libs.

https://salsa.debian.org/xorg-team/app
https://src.fedoraproject.org/repo/pkgs/xorg-x11-server-utils/

Looking for help from experts for this complex recipe.

Package is not available

• The package is not available on conda-forge.

No previous issues or open PRs

• No previous issue exists and no PR has been opened.

[ehfd]

CC @xhochy @hmaarrfk @pkgw and others.

[ehfd]

Additional CC @epruesse @zklaus @scopatz @erykoff @ccordoba12 @mingwandroid @n-elie

I apologize for the tag bomb.

[ehfd]

I will contribute a subset of the X11 apps packages, typically packaged standalone by Debian or RHEL (such as xorg-libcvt, xauth), which I immediately need.

[hmaarrfk]

I understand the maintenance challenge with X11 are somewhat tiring, but I'm rather hesitant to adding to our maintenance burden unless absolutely necessary.

We don't have a good way to 'transition' packages at conda-forge, so a new package name isn't necessarily an "easy swap".

Can you point us to conda-forge feedstock you need help with?

[ehfd]

@hmaarrfk
None of the packages here I listed are currently in a non-CDT feedstock. Xvfb and xorg-utils (possibly others) are CDT packages. x11-util-macros does not overlap here.

These packages are utility executables (some are important utility tools which control the interaction of an X application with X11 servers where applications depending on X11 are crippled without) with dependencies to currently packaged X11 libraries. We are not changing existing feedstocks.

And my suggestion is simply to merge >30 small feedstocks if done separately into ~5 larger feedstocks similar to Debian or RHEL.

[hmaarrfk]

Hmm. Ok. My inclination is to say that unfortunately this is quite troublesome to execute.

conda-forge/conda-forge.github.io#1894

Discusses the reverse problem, but ultimately we don't have a way of "renaming" packages and thus this is really a difficult excercise.

We ignored problem for jpeg and jpeg-turbo and it left many I knew with no option to update. Only to tear down the environment to rebuild it.

[ehfd]

Then, this would be a long-term thing, I'll leave this issue up for now.

We're not merging existing packages, but this might be a decision that should be made while creating the initial versions of such packages.

[hmaarrfk]

I think the first step in the "long term project" would be to create a table of the new package names, and which existing conda-forge package they "amalgamate".

Typically it helps to edit the first comment to keep the list alive.

[h-vetinari]

Out of curiosity, what's the use-case for having the binaries (as opposed to just the libraries)? In the bigger picture, while the Wayland future is still going to take a good while, it's coming inexorably, so I'm not sure how much of our scarce resources the X11 ecosystem still merits. 🤔

[ehfd]

@h-vetinari

In principle, utilizing the same HPC environment (Conda) researchers use for ML and AI, to deliver rootless, isolated, and portable graphical workloads (Robotics including ROS, Carla, OpenCV, PyMOL, VMD, ParaView, OpenDroneMap, etc.).

HPC facilities now mostly use either Conda or Apptainer (known as Singularity). Other rootless environments such as LinuxBrew (HomeBrew), Pkgsrc, Nix, Spack, and EasyBuild (where the latter two are basically Conda wrappers adding some more packages on their sides) exist.
One example: https://hpc.nmsu.edu/discovery/software/modules/

Out of these, LinuxBrew, Pkgsrc, and Nix have X11 server binaries and the above CLI tools, but only Conda (with Apptainer coming close) is universal to all global HPC facilities and supercomputer centers, and most importantly, considered a first-class citizen of NVIDIA and/or ROCm, either through pip or conda.

The above binaries are required for functional usage of the xorg-xserver binary (needed for XDummy) and xorg-xvfb, as well as VirtualGL (if we are stretching a bit more for headless OpenGL support).

While Wayland is coming fast, the vast majority of HPC facilities are still optimized for X11, and despite Conda already having Wayland libraries (much like X libraries), people are, as of now, not going to start a Wayland server solely inside Conda.

In that sense, Wayland and X11 are similarly limited in Conda.

Moreover, Wayland does not mean that X11 will completely disappear. There will still be legacy applications still on X11 (very prevalent in academic situations), and thus XWayland will be required. Since XWayland is another X server, it comes back to still needing the X.Org CLI tools.

Speaking of which, unlike the libraries, the X.Org CLI tools, as you've said, aren't mandatory in essence, as these feedstocks don't have explicit dependence on other feedstocks. So conversely, they don't need to be built through all sorts of patching for VC or MSYS2 in Windows.

VcXsrv and Xming already do this well (after some meticulous code patches for the whole X11 stack on their sides) for everyone in Windows, and unlike Mac or Linux, Windows doesn't have the ABI and glibc hell which otherwise requires Conda for portability.

Thus, the maintenance strain would be much less if it was only targeted for Mac and Linux because the only things probably required include autoconf boilerplate and a script for automatic build recursion (in my experience writing new feedstocks, >90% of the time spent making X11 feedstocks pass the test comes from patching Windows).
I'm also simply omitting Windows for application feedstocks that depend on X11 when they don't build and have some weird Windows incompatibilities such as fork() (see the PRs for xclip, xsel, xdotool, etc.).

[h-vetinari]

Thanks for the detailed reply! In principle it's a good idea to provide these things of course, but it might be a substantial effort, and that would need a champion. It sounds like that could potentially be you? 🙃

You might also want to join one of the core calls (biweekly Wednesday 18h UTC, next one is today, agenda not too crammed yet) to discuss the idea and how to approach it.

[ehfd]

It sounds like that could potentially be you?

It's not immediate for me in the current sense, but in a year or so if people can wait. It will be in demand eventually. If someone's interested before that, they can do it. Else, I'll do it eventually.

For now, I'm just fully focused on the depending library structures that should land for the Alma8 sysroot launch.
The future structure of mesalib and X11 libraries should be decided before Alma8, then it will benefit this cause some time later.

From conda-forge/cdt-builds#66:

This seems to be the feedstock for libxcomposite: https://github.com/conda-forge/xorg-libxcomposite-feedstock

And libxshmfence is a critical package I forgot that isn't in Conda. This is a CDT package, so should be done ASAP. @hmaarrfk

[h-vetinari]

I brought this up in the core call to get a feel for what people think about this. Overall there's no issue with doing this, though:

• we may not be able to sanely ship all binaries, in case where running certain things needs elevated privileges
• we don't yet really see use-cases in the form of other feedstocks that would want/need to use these
• to do this at the scale proposed would need long-term commitment from >=2 contributors due to the scope of the maintenance effort

Not discussed directly in the call but from my POV: any of the missing lib* packages should be added and PRs are very welcome for that.

[ehfd]

@h-vetinari Great, thank you!

I am also waiting for more potential contributors to contribute to this job.
Perhaps from my own project.

We can wait and see. Meanwhile, I'd thank the core team if they can focus on the CDT refactoring roadmap.

[ehfd]

Follow up from conda-forge/xorg-libxfixes-feedstock#8 (comment)
@hmaarrfk

I have been consistently aware of what @pkgw is trying to do (read the top part of this issue that I added and will edit further), but the problem is that a new MinGW toolchain that isn't ancient is very likely required and thus is dependent on the works of @isuruf. I think a new MinGW toolchain exists now but not the documentation or the remainder of the ecosystem.

Many feedstocks in my experience have massive issues working together with MSVC, and MSYS2 + MinGW is, still in my opinion, the way to go.

Using MSVC would possibly mean massive levels of patches to the point of substantially transforming the original codebase.

Also, it would be much more convenient if mmap(), fork(), checkpid(), and other POSIX capabilities were fully supported during Windows builds without patching them out somehow. I thought Cygwin and therefore MSYS2 did this.

[ehfd]

Note that I no longer have dedicated time for development, so I cannot sit down for this in the foreseeable future. The three new recent contributions of feedstocks were because I had two days of time and they could be done in this timeframe in order to clear my unfulfilled checklist and move it out of my head. So do not count on me to do this in the moderate future.

@hmaarrfk I have updated the top thread to outline everything known @pkgw has said in GitHub, plus my current opinions.

What I diverge from @pkgw, so far, is that using MSVC for X.Org X11 will be a major headache. It will require extensive patches in nearly hundreds of feedstocks and I am not aware of a mechanism where a patch is updated automatically between releases. Even if there is, it is prone to manual intervention.

VcXsrv (https://github.com/marchaesen/vcxsrv) is a whole repository that contains such manual patches for X.Org in order to compile on MSVC, and it will be a lot of upkeep for us to do something similar.

Thus, I strongly feel that continuing to use MinGW GCC compared with MSYS2 is the way to go, at least for X11.

[ehfd]

Thanks to the efforts of @isuruf, conda-forge/xorg-libxcvt-feedstock#4 is an example of using the newest MinGW compiler pipeline. This can pretty much likely be the way forward to reduce the amount of patches (that can arise from the incompatibilities of MSVC with code written solely for GCC) as much as possible.

CC @pkgw @hmaarrfk @h-vetinari

[ehfd]

@pkgw I have updated your notes in conda-forge/libglvnd-feedstock#2 here in the first post.

One comment is that some (core) packages including libx11 itself still have legacy build code and automake1.15, although they build with UCRT today.

Moreover, they contain:

    - xorg-libx11 >=1.6,<2
    # Additional constraintless xorg-libx11 dep below is for conda-build to add potential pinning to.
    - xorg-libx11

Which I don't think is needed as of now because of run-exports.

https://github.com/conda-forge/xorg-libxxf86vm-feedstock/blob/main/recipe/build.sh
https://github.com/conda-forge/xorg-libxxf86vm-feedstock/blob/main/recipe/meta.yaml
The links seem like pretty standard solutions for across X11.