Implement architecture and bitness checks #1360
Conversation
rmartin16
force-pushed
the
arch-checks
branch
3 times, most recently
from
33d3a10
to
ea37930
Compare
FWIW: The last macOS release to support 32 bit apps was Mojave (10.14), which had EOL almost exactly 2 years ago; even that release had copious warnings whenever you tried to run a 32 bit app. Universal2 doesn't (and can't) include 32 bit support. Python packaging theoretically supports:
However, in practice, I don't think I've ever seen one of those tags in use. tl;dr - 32-bit macOS effectively doesn't exist at this point. |
|
To help me keep this straight in my head, I put it in a spreadsheet instead. I believe this represents our current situation....although, I'm considering it up for debate based on what the team wants to actually support. A goal I think I'd like to achieve for this PR is strictly enforcing the supported platforms....instead of letting (for instance) the JDK download fail because there isn't a arm64 version to download on Windows (although, msft does have one for download), Briefcase should provide a relevant error message to the user. So, first, I would like to confirm we align on this goal of defined error states vs "if it works, then go ahead". So, practically, I think this means committing to an actual definition of what Briefcase supports. A couple of things to point out:
A prettier Google Sheets version
Legend
Background
Addendum
|
|
After experimenting a bit more, I think I've landed on an implementation that won't require defining some exhaustive set of compatible platforms. There are two reasons a user's platform may not be compatible:
So, given these known and definable limitations, I'll just update Briefcase to:
|
|
@freakboy3742, couple questions when you get a break from pycon... As it relates to using LinuxDeploy, was Briefcase able to create an AppImage on macOS for Apple Silicon? It seems this would try to download LinuxDeploy using class LinuxDeploy(LinuxDeployBase, ManagedTool):
@property
def file_name(self) -> str:
return f"linuxdeploy-{self.tools.host_arch}.AppImage"
@property
def download_url(self) -> str:
return (
"https://github.com/linuxdeploy/linuxdeploy/"
f"releases/download/continuous/{self.file_name}"
)Also, this makes me wonder how Docker behaves on macOS for Apple Silicon. Is the architecture inside the Docker container Docker docs say they recommend rosetta...so, my presumption is the Linux VM that Docker Desktop runs the containers in is running |
rmartin16
force-pushed
the
arch-checks
branch
8 times, most recently
from
06c9afc
to
30ee4a8
Compare
|
I'm implemented all the checks I've identified as necessary and described in the PR description...save one. The are issues when running a 32bit Python and/or a 32bit Linux on 64bit hardware. It is probably pretty uncommon nowadays for x86-64 hardware to be running a 32bit Linux...if only because most of the big distros have dropped 32bit support. However, this is more common on ARM hardware. Even now, the Raspberry Pi Imager software for imaging SD cards puts the 32bit version front and center in the UI with "recommended" tagged on it. AppImage (in Docker anyway) and Flatpak seem to be ok since they are built in sandboxes. However, the System packages have most of the issues since they can become a mix of 32 and 64 bit. I'm gonna spend some more time trying to tweak some of these trouble spots to better source the information. Moreover, though, I'm not sure there's a reliable way (or at least straightforward way) to detect this situation. While it's pretty straightforward to determine the bitness of Python....it can be tricky to know the bitness of the OS or hardware in an arbitrary environment. As an example, here's |
rmartin16
force-pushed
the
arch-checks
branch
2 times, most recently
from
8a4fd5c
to
8b020d6
Compare
After spending a lot of time trying to understand how the distros internalize architectures, I've landed on an implementation that at least appears to work (although, it should be no worse than where we started)...I am, at least, quite confident that the strategy for Debian is reliable. Instead of using
This approach basically does all the groundwork to eventually implement #1128. However, there is still the problem that the build subdirectory path does not include the target's ABI; that will be necessary to fully support targeting non-native architectures. |
rmartin16
force-pushed
the
arch-checks
branch
7 times, most recently
from
138d446
to
df0c0f6
Compare
| def verify_tools(self): | ||
| """Verify the AppImage LinuxDeploy tool and its plugins exist.""" | ||
| super().verify_tools() | ||
| LinuxDeploy.verify(tools=self.tools) |
There was a problem hiding this comment.
Verify LinuxDeploy before building an entire Docker image and then informing the user that LinuxDeploy can't run on ARM.
There was a problem hiding this comment.
Completely agreed that we should verify LinuxDeploy before building the Docker image.
However, this isn't working for me on my M1. When I run briefcase package linux appimage, it downloads the x86_64 linuxdeploy, then builds the Docker image, then downloads the GTK plugin, then starts building the app image (with a warning that the image being used is linux/amd64), and then fails due to a bad file descriptor. Log file attached.
There was a problem hiding this comment.
I think this is something I've discovered in #1392; if you don't specify an explicit platform to use and the image only supports a platform that's different than the host platform, you get this warning.
Nonetheless, (and ignoring the completely corrupted PATH setting), I think this is AppImage/AppImageKit#965 which suggests that building x86-64 AppImages on Apple Silicon may not currently be possible.
I haven't fully processed that issue....so, I look more at this later.
I may have overstated just how "straightforward" it is to run CI actions/jobs in a Docker image for a different architecture. This is because GitHub's built-in support for running inside a container doesn't support emulation. And this is because GitHub's default configuration for Docker Engine in the runner doesn't support emulation. Docker has actions available to help get a GitHub runner in to a position to perform emulation. The run-on-arch-action action helps encapsulate all this to be more succinct. Nonetheless, I put a POC together so we could at least see this. |
- On Apple Silicon, Docker Desktop runs x86-64 containers in an emulation mode in an arm64 Linux VM; however, due to present issues in the implementation of QEMU used, AppImages cannot successfully run. Since linuxdeploy itself is an AppImage, it cannot run and AppImages cannot be created.
|
Current status:
|
There was a problem hiding this comment.
Oops - sorry - that last comment (and approval) was for #1398, but I had a browser tab malfunction :-)
👍
I'll take "Headaches I can live without" for $400, Alex. :-) ARM64 runners for macOS are supposed to be coming later this year; but even without that, we've been able to maintain reasonable M1 support for Briefcase. Admittedly, this is at least in part because at least one of the maintainers (cough me cough) is using M1 as a daily driver; but even still, it does lend credence to the fact that we are able to maintain at least baseline levels of quality without explicitly having a platform to test on.
That sounds like a good idea to me. Differentiating between "Platforms we actively check in CI (x86_64/*)", "Platforms we consider critical (macOS on M1)" and "Platforms where we'll do our best, but YMMV" would also be a good idea. |
|
I finally added the platform support documentation; definitely open to style critique. That should wrap this up, though :) |
There was a problem hiding this comment.
I'll have to take your word that this works on armv6 and similar esoteric platforms, but the code certainly looks like it should work, and the platforms I can test all work well.
I've done some tweaking of the support tables; but the changes are almost entirely cosmetic (but OMFG do I want to migrate to MyST... the limitations of ReST are really starting to bug me).
Once again - a great piece of work to make Briefcase that much more solid.
Summary
Changes
JAVA_HOMEis not setarm,aarch64,x86_64armrelease is for thearmv7hfABI...soarmv6l(e.g. RPi Zero) is not supportedx86_64,arm64AMD64(and technicallyx86)ANDROID_HOMEis not setsdkmanageris a lot of fun 🙃sdkmanageris its support to installemulatoras a dependency forbuild-toolssdkmanagercan only downloademulatorpackage for:x86_64aarch64....Google even builds it in their CI....butsdkmanagerwon't fetch it for you...x86_64,arm64AMD64adbx86_64andarm64i686andx86_64TODO: decide if error if Flatpak runtime support is scantx86_64andaarch64TODO: decide if error when Python bitness doesn't match target OS bitnesspipdetects this and installs 32bit wheels and the underlying packaging tools seem to create 32bit packages as well.x86_64andarm64AMD64ARM64support would ideally be added at some point, thoughRCEditis only available forAMD64...although Windows on ARM can runx86_64binaries....so, I guess the real question is whetherRCEditcan editARM64binaries....i686support is unnecessary as 32bit Windows continues to disappearNotes
sdkmanagerconsidersemulatora dependency ofbuild-toolsemulatoror avoid the dependency another way...android_sdk/emulator/package.xmlfile is sufficient to spoof this forsdkmanagerandroid_sdk/emulator/emulator.exefile is sufficient to spoof this for Briefcase....however, Briefcase depends on this file being an executable later on...PR Checklist: