Looking to make a FlatPak version...

[EvilSupahFly]

It's not super urgent, as I don't have much free time, but I'd like to make Amulet into a Flatpak for Linux users who don't want to mess with or understand how to work with dependencies and the like, but the Amulet project no longer uses a specifically defined "requirements.txt" file, and I was wondering if this requirement information was moved into "setup.cfg" in its entirety or if the installer makes some hard-coded assumptions. There's a lot of stuff the flatpak builder can't use or doesn't understand in "setup.cfg" meaning I can't reference it directly, so I was looking to simplify it back down to the old-school "requirements.txt", but I don't know if I need to add anything that isn't already defined in the setup file.

For what it's worth, this is what I've reduced it down to:

amulet_map_editor
amulet_core>=1.9
amulet_nbt>=2.0
black>=22.3
minecraft_resource_pack>=1.3
numpy>=1.17
packaging
Pillow>=10.0.1
platformdirs>=3.1
pre_commit>=1.11.1
pymctranslate>=1.2
pyopengl>=3.0
python_version>=3.9
Sphinx>=1.7.4
sphinx_autodoc_typehints>=1.3.0
sphinx_rtd_theme>=0.3.1
wxPython

Am I missing anything?
** Edit - Fixed some context-sensitive typos

[gentlegiantJGC]

I don't know too much about the flatpak format.
Would each library be bundled separately or all bundled into one package?

A requirements.txt file can be generated with pip freeze which will write all the installed libraries to a requirements.txt file but that will freeze all of the requirements.

As part of our build process we freeze first party dependencies to ensure that everyone running a given version of amulet_map_editor is also running the same version of amulet_core and other first party dependencies to make debugging easier.
I would like the same behaviour here.
The following code installs the dependencies and calls pip freeze and extracts only first party libraries.
https://github.com/Amulet-Team/Amulet-Map-Editor/blob/0.10/setup.py#L23C5-L23C24

Most of the dependencies you have listed are development requirements that are not needed to run the program.

build-time dependencies are listed here
https://github.com/Amulet-Team/Amulet-Map-Editor/blob/0.10/pyproject.toml#L2

runtime dependencies are listed here
https://github.com/Amulet-Team/Amulet-Map-Editor/blob/0.10/setup.cfg#L17

[gentlegiantJGC]

It may be easier to add linux wheels

[EvilSupahFly]

With a Flatpak, ideally a project can be self-contained and sandboxed with all its libraries and dependencies so it can be run on any version of Linux right out of the box, so to speak. This avoids manually creating VENs, or potentially screwing things up by skipping the virtual environment and doing a system-wide install, and also eliminates the need to muck about in the terminal or with PIP (as many Linux newcomers might not be comfortable with these). Because Flathub plugs right into the Software Manager GUI for both Gnome and KDE, users can easily install or uninstall a Flatpak with no prior understanding of anything, really.

Like GIT, it has built in version control as well, meaning users will get the newest version when they install fresh, and be notified by their system update manager when updates are available. Wheels are handy too, but still require people to do "terminal things" whether it's typing out stuff themselves, or running a script - something Flatpaks avoid. Also, wheels aren't completely sandboxed, though they can be self contained.

There's a pretty good write up about Flatpaks over on Reddit if you were curious.

[gentlegiantJGC]

Sounds kind of like a windows installer.
We use PyInstaller to bundle the code into an executable format.
An additional step is required to make a normal installer.

Is there a way to convert the output from pyinstaller to a flatpak?

Are dependencies like wxPython installed separately by the package manager or would we also bundle those with the flatpak?

[EvilSupahFly]

According to the official docs, Flatpak uses runtimes which provide the basic dependencies that are used by applications. Each application must be built against a runtime, and this runtime must be installed on a host system in order for the application to run, and Flatpak can automatically install the runtime required by an application.

Their entire runtime catalogue is listed on their website. My original plan was to test it first on my production box running Mint, then test it on a couple different VMs running other distros. The issue I have with Mint is that it's always based on the last LTS release of Ubuntu, but sometimes projects want newer versions of certain core libraries (libc is the most common offender) which can't just be manually installed unless you want to brick your system, and aren't brought in with Python's VENV because it (potentially) violates the library sanctity of the host OS or will break the project.

Since Flatpaks are bundled and sandboxed with their own runtimes, in theory, it fixes this.

[UltraBlackLinux]

super grateful that you're working on this. I've stopped using amulet years ago because I could never get the project to build.
Very happy to see some efforts towards making this somewhat portable

[EvilSupahFly]

It's a bit more challenging than I originally thought but I'm making good progress. There are a few quirky things I had to learn about FlatPak first. 😜

Edit: once I have the process tweaked, it should work for all future versions of Amulet. Trying to get it done around my actual job has been the hardest part.

[EvilSupahFly]

OK - After some initial testing, a Flatpak learning curve and some issues with Python package versioning, I think I have something workable based on Amulet 0.10.34 ----

Edit: Scratch that. There was an issue building the flatpak which needs to be addressed.

[EvilSupahFly]

I now have a working Flatpak version of Amulet built from Amulet v0.10.34, and I'm looking for testing feedback.

https://github.com/EvilSupahFly/Amulet-Flatpak

[gentlegiantJGC]

I haven't tested but it looks good.
Some of the dependencies are not needed. We don't use pygame and pre-commit, wheel and cython are compile time dependencies (unless these are for some reason needed at runtime)

[EvilSupahFly]

I've removed PyGame, PreCommit, wheel and Cython, and once I have a working install of Python 3.9 and 3.10, I'll try building the FlatPak again. Also building from 0.10.35 now.

[EvilSupahFly]

After some trial-and-error, I've realized that the issue might be with the Flatpak sandboxed system itself. It uses Python 3.11, so I made a venv for the same version, and while Amulet runs and loads worlds as expected in the venv, it still crashes when loading a world in the Flatpak.

I've added the Python module for xapp to the requirements, and added some extra OpenGL-related things to the build recipe, and rounded out support for both Wayland and X11:

finish-args:
  - --env=GDK_BACKEND=x11
  - --env=GDK_BACKEND=wayland
  
modules:
  - name: python
    buildsystem: simple
    sources:
      - type: git
        url: https://github.com/python/cpython.git
        tag: v3.11.0
  - name: opengl
    buildsystem: cmake
    sources:
      - type: git
        url: https://github.com/KhronosGroup/OpenGL-Registry.git
        tag: master
  - name: opengl-extensions
    buildsystem: cmake
    sources:
      - type: git
        url: https://github.com/KhronosGroup/OpenGL-Extension-Registry.git
        tag: master
  - name: xapp-gtk3-module
    buildsystem: meson
    sources:
      - type: git
        url: https://github.com/linuxmint/xapp.git
        tag: master
  - name: wayland
    buildsystem: cmake
    sources:
      - type: git
        url: https://gitlab.freedesktop.org/wayland/wayland.git
        tag: master
  - name: x11
    buildsystem: meson
    sources:
      - type: git
        url: https://gitlab.freedesktop.org/xorg/lib/libx11.git
        tag: 1.7.0

This should fix Failed to load module "xapp-gtk3-module" - hopefully, and stop it from crashing when loading a world.

Now ... if only it didn't take so long to actually compile everything and build the Flatpak...

[EvilSupahFly]

I finally have a working-in-debug-mode Flatpak version of the editor! As I type this, the non-debug version is being built. I figured out the problem after reading through the docs on running a flatpak app in a debug shell. I had made a somewhat stupid mistake:

Using add-build-extensions: versus inherit-extensions: makes a world of difference. The app was crashing because the OpenGL libraries were being used to build the initial app (add) but weren't being made available to the runtime inside the sandbox (inherit).

After changing

sdk-version: '23.08'
add-build-extensions:
  - org.freedesktop.Platform.GL.default
command: amulet_map_editor

To this:

sdk-version: '23.08'
inherit-extensions:
  - org.freedesktop.Platform.GL
  - org.gtk.Gtk3theme
  - org.freedesktop.Platform.GL.Debug
  - org.freedesktop.Platform.VAAPI.Intel
inherit-sdk-extensions:
  - org.freedesktop.Sdk.Debug
  - org.freedesktop.Sdk.Extension
  
command: amulet_map_editor

The Amulet Map Editor now runs flawlessly as a flatpak in the debug shell and as I write this, the non-debug version is being built and will be up as soon as possible.

[EvilSupahFly]

It finally works! It was a hell of a ride, but the flatpak actually works, so you can close this issue if you want.

[gentlegiantJGC]

How can we integrate this as part of our build system?

We currently have a github workflow which generates the sdist, wheels and pyinstaller bundle automatically when we create a release.
https://github.com/Amulet-Team/Amulet-Map-Editor/blob/0.10/.github/workflows/python-build.yml

It would be great if we could create a flatpak automatically as well.

[EvilSupahFly]

I'm not sure if GitHub can automatically generate the flatpak but I'll check around and let you know.

[gentlegiantJGC]

The github workflow can run any code on any supported OS.
As long as you can create a script to automatically create the flatpack from the repositories/pypi it should be able to run it.

[gentlegiantJGC]

Even looks like an official action
https://github.com/flatpak/flatpak-github-actions

[EvilSupahFly]

The workflow runs, but fails. Looks like it will take some tweaking as it seems Git's builder is much less forgiving than when building local.

Edit: Tweaking workflow YML seems to have worked. I'm looking at 4000 lines of output (and counting, as of the "edit" timestamp on this post) and no errors reported yet.

[EvilSupahFly]

As of the release of Amulet 0.10.36, the Flatpak version of Amulet runs the same as the "Pure Python" version, and aside from the flickering interface, the Flatpak runs pretty smoothly, even with Python debug options enabled.

Flatpak users can download the Flatpak build directly from my Releases page along with a shell script to install and run it.