Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement architecture and bitness checks #1360

Merged
merged 16 commits into from
Aug 30, 2023

Conversation

rmartin16
Copy link
Member

@rmartin16 rmartin16 commented Jul 14, 2023

Summary

  • Only download Android SDK when known to work on host platform
  • Only download JDK when known to work on host platform
  • Only download a Standalone Python known to work on host platform
  • Only build Windows Apps on wholly 64bit platforms
  • Fix i386 support for AppImages
  • (Attempt to) Build x86-64 AppImages on Apple Silicon
  • Ensure Linux System builds are actually compatible with the target platform

Changes

  • Prevent Briefcase from attempting to run in known unsupported configurations
  • Android AAB/APK
    • Java
      • Error if JDK is not supported and JAVA_HOME is not set
      • Adoptium provides JDK for:
        • Linux: arm, aarch64, x86_64
          • The arm release is for the armv7hfABI...so armv6l (e.g. RPi Zero) is not supported
        • macOS: x86_64, arm64
        • Windows: AMD64 (and technically x86)
    • Android SDK
      • Error if Android SDK is not supported and ANDROID_HOME is not set
      • The sdkmanager is a lot of fun 🙃
        • A key prerequisite for Briefcase to install the Android SDK using sdkmanager is its support to install emulator as a dependency for build-tools
        • However, the sdkmanager can only download emulator package for:
          • Linux: x86_64
            • The emulator can run on aarch64....Google even builds it in their CI....but sdkmanager won't fetch it for you...
          • macOS: x86_64, arm64
          • Windows: AMD64
            • It's unclear if running the emulator on Windows on ARM is possible, but there is "Windows Subsystem for Android"....and it definitely looks like arbitrary Android apps can run in it via adb
  • iOS App
    • Assume platform is supported since macOS is only practically available for x86_64 and arm64
  • Linux AppImage
    • Error if LinuxDeploy is not supported
    • LinuxDeploy is available for i686 and x86_64
  • Linux Flatpak
    • TODO: decide if error if Flatpak runtime support is scant
    • The existing error that a runtime is not available is pretty clear about the problem...so, leaving as is.
    • Flatpak runtime support is only practical for x86_64 and aarch64
    • Although, the Flatpak tools all "work" on the other platforms
  • Linux System
    • TODO: decide if error when Python bitness doesn't match target OS bitness
    • It's not exactly straightforward to determine if Python has a different bitness than the OS. However, pip detects this and installs 32bit wheels and the underlying packaging tools seem to create 32bit packages as well.
    • Any platform is supported by virtue of the fact the platform exists
  • macOS App
    • Assume platform is supported since macOS is only practically available for x86_64 and arm64
  • Web
    • Assume platform is supposed since web is practically platform-agnostic
  • Window App
    • Error if Python was built targeting 32bit
    • Error for unsupported platforms
    • The Briefcase template only supports AMD64
    • ARM64 support would ideally be added at some point, though
      • RCEdit is only available for AMD64...although Windows on ARM can run x86_64 binaries....so, I guess the real question is whether RCEdit can edit ARM64 binaries....
    • i686 support is unnecessary as 32bit Windows continues to disappear

Notes

  • It's a bit annoying that sdkmanager considers emulator a dependency of build-tools
    • This effectively breaks Android support on any platform that isn't a "normal PC or macbook"
    • Maybe it's possible to spoof emulator or avoid the dependency another way...
      • Putting a proper android_sdk/emulator/package.xml file is sufficient to spoof this for sdkmanager
      • A android_sdk/emulator/emulator.exe file is sufficient to spoof this for Briefcase....however, Briefcase depends on this file being an executable later on...

PR Checklist:

  • All new features have been tested
  • All new features have been documented
  • I have read the CONTRIBUTING.md file
  • I will abide by the code of conduct

@rmartin16 rmartin16 force-pushed the arch-checks branch 3 times, most recently from 33d3a10 to ea37930 Compare July 14, 2023 15:35
@freakboy3742
Copy link
Member

  • macOS

    • Supported arches: x86-64, arm64

      • Modern macOS is only x86-64 and arm64...so probably not really effective checking this
    • Supported Python: 64bit

      • I don't know when macOS dropped 32bit support but Python did with 3.7
      • Although, maybe "universal2" installers include 32bit support...kinda doubt it tbh

FWIW: The last macOS release to support 32 bit apps was Mojave (10.14), which had EOL almost exactly 2 years ago; even that release had copious warnings whenever you tried to run a 32 bit app.

Universal2 doesn't (and can't) include 32 bit support. Python packaging theoretically supports:

  • "universal", supporting x86_64, i386, ppc64 and ppc
  • "intel", "fat" and "fat32" supporting i386 and x86_64

However, in practice, I don't think I've ever seen one of those tags in use.

tl;dr - 32-bit macOS effectively doesn't exist at this point.

@rmartin16
Copy link
Member Author

rmartin16 commented Jul 18, 2023

To help me keep this straight in my head, I put it in a spreadsheet instead. I believe this represents our current situation....although, I'm considering it up for debate based on what the team wants to actually support.

A goal I think I'd like to achieve for this PR is strictly enforcing the supported platforms....instead of letting (for instance) the JDK download fail because there isn't a arm64 version to download on Windows (although, msft does have one for download), Briefcase should provide a relevant error message to the user.

So, first, I would like to confirm we align on this goal of defined error states vs "if it works, then go ahead". So, practically, I think this means committing to an actual definition of what Briefcase supports.

A couple of things to point out:

  • Do we really need all these arm versions?
    • maybe...
      • Pi zero will report armv6l
      • Pi 3B+ will report armv7l
      • Pi 4B with 32bit OS will report armv7l or armv8l (need to confirm)
        • Older Linux kernels jury-rigged Pi support by hardcoding some broadcom chip so armv7l would always be reported by certain tools. need to confirm if this is still the case.
      • Pi 4B with 64bit OS will report aarch64
    • The good news is this is only really relevant to Linux System packages and we don't have to include a Python support package for those apps
      • So, this avoids all the complications of the ARM ABIs prior to aarch64
  • What about 32bit Python on 64bit OS?
    • On Windows, this will be an error
    • On macOS, this is moot...so a check probably isn't even necessary
    • On Linux...it'll probably be necessary to confirm Python's bitness matches that of the OS
      • Needs more testing....although, it's also a bit difficult to get 32bit Python running on 64bit Linux nowadays
  • Flatpak i686 and arm support
    • While runtimes aren't generally available, if one was found, this would all work..
    • So, we could just show a warning to users....or simply enforce that only x86_64 and aarch64 are supported
  • Other architectures
    • FWIW, other popular targets are ppc64, ppc64le, s390x
    • I don't really see the need for Briefcase to target these platforms...but future work could always add them
  • The build directory doesn't really support targeting different arches
    • I'm unlikely to address this in this PR...but calling it out here
    • For instance, it's currently possible to target both debian:bullseye and i386/debian:bullseye....but they will use the same build sub-directory.
    • This obviously isn't ideal...and will definitely be exacerbated once explicitly targeting other architectures is supported
    • (Although, it should be noted this doesn't actually work...since Briefcase thinks its building a amd64 package)

A prettier Google Sheets version

Platform Format Host OS Host Arch Supported? Notes
Android Gradle Linux i686  
      x86_64  
      armv6l 🟨 Requires user-installed Android SDK and JDK
      armv7l 🟨 Requires user-installed Android SDK
      armv8l 🟨 Requires user-installed Android SDK
      aarch64 🟨 Requires user-installed Android SDK
    macOS x86_64  
      arm64  
    Windows x86 🟨  
      x86_64  
      arm64 🟨 Requires user-installed Android SDK and JDK
iOS Xcode macOS x86_64  
      arm64  
Linux AppImage Linux i686  
      x86_64  
      armv6l Support from LinuxDeploy is unlikely
      armv7l Support from LinuxDeploy is unlikely
      armv8l Support from LinuxDeploy is unlikely
      aarch64 🟠 Pending aarch64 release from LinuxDeploy
    macOS x86_64 Requires Docker on host
      arm64 🟠 Requires Docker on host; pending aarch64 release from LinuxDeploy
  Flatpak Linux i686 Runtime support was dropped
      x86_64  
      armv6l Runtime support was dropped
      armv7l Runtime support was dropped
      armv8l Runtime support was dropped
      aarch64  
  System Linux i686  
      x86_64  
      armv6l  
      armv7l  
      armv8l  
      aarch64  
    macOS x86_64  
      arm64  
macOS App macOS x86_64  
      arm64  
  Xcode macOS x86_64  
      arm64  
Web Static Linux Any  
    macOS Any  
    Windows Any  
Windows App Windows x86 Windows 11 doesn’t support x86 and it’s scant on Windows 10
      x86_64  
      arm64 The stub app in the template needs to be built for arm64
  VisualStudio Windows x86 Windows 11 doesn’t support x86 and it’s scant on Windows 10
      x86_64  
      arm64 The VS project needs arm64 added as a compile target

Legend

Icon Description
Supported
🟨 Supported with user prerequisites
Requires BeeWare development for support
🟠 Requires upstream development to add support
Support not possible and/or desired

Background

  • Windows
    • Supported arches: AMD64
      • the stub app is pre-compiled for x86-64
      • the Visual Studio project requires a lot of nontrivial changes to support x86 compilation
      • Although, arm64 support is more interesting and also nontrivial to add to the project
    • Supported Python: 64bit
      • pip will download a 32bit version of cffi with 32bit Python which prevents the stub app from loading cffi since it'll be a x64 process
  • macOS
    • Supported arches: x86-64, arm64
      • Modern macOS is only x86-64 and arm64...so probably not really effective checking this
    • Supported Python: 64bit
      • I don't know when macOS dropped 32bit support but Python did with 3.7
      • Although, maybe "universal2" installers include 32bit support...kinda doubt it tbh
  • Linux
    • Supported arches: varied
      • AppImage requires x86-64 or i386 to run LinuxDeploy
      • Flatpak can be built on x86-64 and aarch64
        • Support for armv5/6/7 (32 bit) was deprecated
        • While Flatpak runs on i386, there aren't any supported runtimes for it (especially the Gnome runtime)....but there are i386 extensions for x86-64 runtimes that effectively allow for running x86 applications on x86-64 hardware in Flatpak
      • System packages should support all
        • seems reasonable to allow Briefcase to run if the distro supports the configuration
        • However, briefcase package --target i386/debian:bullseye on x86-64 produces an i386 deb labeled as amd64 since platform.machine() in a docker container detects the real underlying hardware
    • Support Python: varied
      • I can imagine using 32bit Python will also lead to downloading 32bit packages....which could be problematic
      • But 32bit Python must be allowed for System packages...
      • Might be best to confirm the bitness of Python is the same as the OS...

Addendum

  • Docker on i386
    • sudo apt install docker.io
    • DOCKER_BUILDKIT=1 /usr/bin/docker build --platform=linux/386 -o . https://github.com/docker/buildx.git
    • mkdir -p ~/.docker/cli-plugins/ && mv buildx ~/.docker/cli-plugins/docker-buildx
    • docker run --rm -it i386/debian:bookworm

@rmartin16
Copy link
Member Author

After experimenting a bit more, I think I've landed on an implementation that won't require defining some exhaustive set of compatible platforms.

There are two reasons a user's platform may not be compatible:

  • Briefcase, itself, does not support the platform
  • An underlying tool does not support the platform...or Briefcase's installation method doesn't support the platform

So, given these known and definable limitations, I'll just update Briefcase to:

  • prevent commands from running if Briefcase doesn't work on the platform
    • e.g. Windows commands on a arch other than AMD64
  • return a proper error if Briefcase cannot install a usable version of a tool
    • e.g. JDK can't be installed on armv6l....so instead of erroring on the download, tell the user to set JAVA_HOME
    • (although, i'ts actually a bit more complicated for JDK here....but I'll omit the details for now)

@rmartin16
Copy link
Member Author

@freakboy3742, couple questions when you get a break from pycon...

As it relates to using LinuxDeploy, was Briefcase able to create an AppImage on macOS for Apple Silicon? It seems this would try to download LinuxDeploy using arm64 as the architecture...which isn't going to work.

class LinuxDeploy(LinuxDeployBase, ManagedTool):
    @property
    def file_name(self) -> str:
        return f"linuxdeploy-{self.tools.host_arch}.AppImage"

    @property
    def download_url(self) -> str:
        return (
            "https://github.com/linuxdeploy/linuxdeploy/"
            f"releases/download/continuous/{self.file_name}"
        )

Also, this makes me wonder how Docker behaves on macOS for Apple Silicon. Is the architecture inside the Docker container x86_64? or aarch64?

Docker docs say they recommend rosetta...so, my presumption is the Linux VM that Docker Desktop runs the containers in is running x86_64 in emulation.

@rmartin16 rmartin16 force-pushed the arch-checks branch 8 times, most recently from 06c9afc to 30ee4a8 Compare July 21, 2023 00:46
@rmartin16
Copy link
Member Author

I'm implemented all the checks I've identified as necessary and described in the PR description...save one.

The are issues when running a 32bit Python and/or a 32bit Linux on 64bit hardware. It is probably pretty uncommon nowadays for x86-64 hardware to be running a 32bit Linux...if only because most of the big distros have dropped 32bit support. However, this is more common on ARM hardware. Even now, the Raspberry Pi Imager software for imaging SD cards puts the 32bit version front and center in the UI with "recommended" tagged on it.

AppImage (in Docker anyway) and Flatpak seem to be ok since they are built in sandboxes.

However, the System packages have most of the issues since they can become a mix of 32 and 64 bit. I'm gonna spend some more time trying to tweak some of these trouble spots to better source the information.

Moreover, though, I'm not sure there's a reliable way (or at least straightforward way) to detect this situation. While it's pretty straightforward to determine the bitness of Python....it can be tricky to know the bitness of the OS or hardware in an arbitrary environment. As an example, here's systemd trying to figure it out the arch.

@rmartin16 rmartin16 force-pushed the arch-checks branch 2 times, most recently from 8a4fd5c to 8b020d6 Compare July 22, 2023 13:28
@rmartin16
Copy link
Member Author

rmartin16 commented Jul 22, 2023

However, the System packages have most of the issues since they can become a mix of 32 and 64 bit. I'm gonna spend some more time trying to tweak some of these trouble spots to better source the information.

After spending a lot of time trying to understand how the distros internalize architectures, I've landed on an implementation that at least appears to work (although, it should be no worse than where we started)...I am, at least, quite confident that the strategy for Debian is reliable.

Instead of using platform.machine() to determine the ABI tag for system packages, Briefcase now defers to the build environment for this information:

  • Debian: dpkg --print-architecture
  • RHEL: rpm --eval '%_target_cpu'
  • Arch: pacman-conf Architecture

This approach basically does all the groundwork to eventually implement #1128. However, there is still the problem that the build subdirectory path does not include the target's ABI; that will be necessary to fully support targeting non-native architectures.

@rmartin16 rmartin16 force-pushed the arch-checks branch 7 times, most recently from 138d446 to df0c0f6 Compare July 25, 2023 23:17
Comment on lines +56 to +59
def verify_tools(self):
"""Verify the AppImage LinuxDeploy tool and its plugins exist."""
super().verify_tools()
LinuxDeploy.verify(tools=self.tools)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify LinuxDeploy before building an entire Docker image and then informing the user that LinuxDeploy can't run on ARM.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completely agreed that we should verify LinuxDeploy before building the Docker image.

However, this isn't working for me on my M1. When I run briefcase package linux appimage, it downloads the x86_64 linuxdeploy, then builds the Docker image, then downloads the GTK plugin, then starts building the app image (with a warning that the image being used is linux/amd64), and then fails due to a bad file descriptor. Log file attached.

briefcase.2023_07_31-11_57_12.package.log

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is something I've discovered in #1392; if you don't specify an explicit platform to use and the image only supports a platform that's different than the host platform, you get this warning.

Nonetheless, (and ignoring the completely corrupted PATH setting), I think this is AppImage/AppImageKit#965 which suggests that building x86-64 AppImages on Apple Silicon may not currently be possible.

I haven't fully processed that issue....so, I look more at this later.

@rmartin16
Copy link
Member Author

rmartin16 commented Jul 31, 2023

This is true....unless we're willing to pursue emulation in CI. GitHub makes it fairly straightforward to run actions/jobs inside a Docker container....and those containers can be for an arbitrary platform. Then, Linux System and Flatpak could be tested on aarch64 and an AppImage could be built on i386.

I may have overstated just how "straightforward" it is to run CI actions/jobs in a Docker image for a different architecture. This is because GitHub's built-in support for running inside a container doesn't support emulation. And this is because GitHub's default configuration for Docker Engine in the runner doesn't support emulation. Docker has actions available to help get a GitHub runner in to a position to perform emulation. The run-on-arch-action action helps encapsulate all this to be more succinct.

Nonetheless, I put a POC together so we could at least see this.

- On Apple Silicon, Docker Desktop runs x86-64 containers in an
  emulation mode in an arm64 Linux VM; however, due to present issues
  in the implementation of QEMU used, AppImages cannot successfully
  run. Since linuxdeploy itself is an AppImage, it cannot run and
  AppImages cannot be created.
@rmartin16
Copy link
Member Author

Current status:

  • The whole AppImage on Apple Silicon is a bit moot not since I disabled the possibility
    • Although, I'd be interested in still clarifying our misalignment if worthwhile
  • Still need to add documentation for host platform support
  • I also want to merge in the Command-line Tools and JDK bumps before this is merged
  • Interested in thoughts on emulation in CI
    • I think my biggest concern (outside of complexity and some duplication) is how slow the emulation is
    • To that end, we may be able to speed CI up a bit if we create more jobs....probably another job matrix level for platform or maybe even output format...have to evaluate the overhead of setting up everything more times

Copy link
Member

@freakboy3742 freakboy3742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 All very straightforward.

Copy link
Member

@freakboy3742 freakboy3742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops - sorry - that last comment (and approval) was for #1398, but I had a browser tab malfunction :-)

@freakboy3742
Copy link
Member

At any rate, my real intention has become to ensure a defined runtime behavior for Briefcase on any platform. Previously, Briefcase would error at an arbitrary point for an incompatibility that wasn't well defined...or worse, create a broken/incompatible package.

👍

that we're not able to validate in an automated fashion.

This is true....unless we're willing to pursue emulation in CI. GitHub makes it fairly straightforward to run actions/jobs inside a Docker container....and those containers can be for an arbitrary platform. Then, Linux System and Flatpak could be tested on aarch64 and an AppImage could be built on i386.

A more intensive option would be to set up a self-hosted aarch64 GitHub runner to do this on....that would be native hardware and probably a better overall test of the functionality.

I'll take "Headaches I can live without" for $400, Alex. :-)

ARM64 runners for macOS are supposed to be coming later this year; but even without that, we've been able to maintain reasonable M1 support for Briefcase. Admittedly, this is at least in part because at least one of the maintainers (cough me cough) is using M1 as a daily driver; but even still, it does lend credence to the fact that we are able to maintain at least baseline levels of quality without explicitly having a platform to test on.

At the very least, I think it we need to add some form of documentation on where our platform support is, and the guarantees we can make on each. The support chart you've compiled is a good starting point for this; we should formalise that a little into a formal support guarantee.

That makes sense. What about a table at the top of each Platform page that details host machine support similar to the Toga support tables? A note could also be added underneath if the limiting factor is a third-party tool that doesn't support the platform.

Additionally, I suppose those individual tables could be combined (as Toga does) in to a single page to show what platforms can build which packages.

That sounds like a good idea to me. Differentiating between "Platforms we actively check in CI (x86_64/*)", "Platforms we consider critical (macOS on M1)" and "Platforms where we'll do our best, but YMMV" would also be a good idea.

@rmartin16
Copy link
Member Author

I finally added the platform support documentation; definitely open to style critique.

That should wrap this up, though :)

Copy link
Member

@freakboy3742 freakboy3742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to take your word that this works on armv6 and similar esoteric platforms, but the code certainly looks like it should work, and the platforms I can test all work well.

I've done some tweaking of the support tables; but the changes are almost entirely cosmetic (but OMFG do I want to migrate to MyST... the limitations of ReST are really starting to bug me).

Once again - a great piece of work to make Briefcase that much more solid.

@freakboy3742 freakboy3742 merged commit 7ceb159 into beeware:main Aug 30, 2023
32 of 35 checks passed
@rmartin16 rmartin16 deleted the arch-checks branch August 30, 2023 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants