Support the use conda as a base environment for Briefcase apps

[freakboy3742]

There is a significant part of the Python ecosystem (especially in the science/data space) that uses Conda to manage their Python environments. Some tools are easier to install via Conda; it therefore follows that some apps may benefit from being based on a Conda environment, rather than the default "vanilla" Python installation.

Briefcase should provide, as an option, the ability to build the app on a Conda base environment, rather than the "vanilla" Python install.

This will involve providing a configuration option in pyproject.toml to specify the base/foundation that an app wishes to use (with the current "vanilla Python" behavior being the default); then, depending on the configured foundation, Briefcase should change:

1. The default application template
2. The default support package
3. The tool used to install app packages

Ideally, this would be done via a plugin interface, so that:

1. The conda backend can be developed independently without imposing a maintenance burden on the briefcase team
2. Other backends could be developed. There isn't a huge demand for this (pip and conda really are the only options in this space), but it might be useful to be able to configure different template + support package combinations as defaults.

Implementation hints

The affected code is all contained in briefcase/commands/create.py:

• The support_package_url property
• The app_template_url property
• The install_app_dependencies() method

If these three attributes/methods could be abstracted behind an interface, with Briefcase providing a default implementation for that interface that provides the existing behavior, an alternate Conda implementation of that interface could then be installed.

The implementation to be used would be selected early in the lifecycle of the Create command, based upon a configuration option on the app looking up a key in a list of pkg_resources plugins (see setup.cfg and briefcase/platforms/__init__.py for details on how plugins are used at present to define the platforms). Ideally, the plugin interface would allow any command to have overridden entry points (so - Briefcase would define a briefcase.foundation.default.create plugin, and a Conda plugin for briefcase would define briefcase.foundation.conda.create; allowing for briefcase.foundation.default.build and briefcase.foundation.conda.build et al in future, if required).

Questions/Considerations

A (probably incomplete) list of questions/considerations for this project:

1. How do you create a standalone, relocatable Conda environment? Are binaries installed into a Conda site-packages folder location independent? Can the .app folder be moved once an installation has occurred?
2. What configuration to that environment is needed to support having a site-packages folder in user space, outside the location of the packaged Conda environment.
3. Supporting Conda on non-desktop platforms (in particular iOS and Android) would be desirable, but is unlikely in the short term. How will foundation plugins notify the user that they don't support a given platform? Is there a need for a "foundation validation" step when the plugin is selected?

[freakboy3742]

Flagging @goanpeca and @tlambert03 on this since they have expressed an interest in this functionality.

[goanpeca]

Pinging @jaimergp

[jaimergp]

Hi Russel, I might work on this in the coming weeks, so I've been looking into your detailed description (it's excellent! I wish more issues were written with this amount of care) and the questions.

I'll try answer them to the best of my knowledge:

1. Once created, Conda environments are not relocatable. There's a non trivial amount of path substitution magic going on. However, Conda does support relocatable environment instructions through several ways:

• An environment.yml or a conda.lock file can be generated from an existing, solved environment. These instructions can then be used to download and install the packages in a given target. However, we trust that the channels won't be deleting any of their packages. This is mostly the case with conda-forge and defaults, but we can't guarantee other channels will follow the same policy. One way of using this, with very little external tooling, is to solve the environment, download the tarballs, and ship a script to install those tarballs in the relevant location within Briefcase.
• conda-pack will pack an existing environment in a tarball and apply the same relocation magic on the target machine through some bundled scripts. It might have some caveats, but I think it can be used.
• constructor can create an installer out of an environment.yml file. This rivals in use cases with Briefcase itself, but doesn't support as many output formats.

2. AFAIK there's no conda setting for that, but we should be able to write arbitrary locations to the Python installation files like this.
3. Conda is theoretically available for ARM platforms, but I don't know if that helps. We should possibly error out if a non-supported architecture is detected. No clue how you are handling these right now.

[freakboy3742]

Hi @jaimergp! I'm here to help in any way I can.

That said - if I'm understanding what you've described, (1) sounds like it could be a major blocker. If conda bakes in hard-coded, non-relocatable paths, that's going to be an issue. macOS doesn't require that app files are in a specific location, and it doesn't prohibit apps from being moved. So - I could create an app in ~/projects/myapp, move it to my per-user ~/Applications folder, then move it to the system /Applications folder - and the app should continue to run in each location without modification. If conda is baking full paths into the install, then relocation like this won't be possible.

Managing this as an "installation instructions" task isn't really viable, either. In order to survive the app signing process, the content of the app needs to be fixed at time of distribution. The only way I can see this working would be for the app to unpack the Conda component into a known system-level folder (e.g., /usr/local), and then rely on the conda install being in that location... but that (a) means the app is no longer completely self-contained, and (b) could lead to some interesting filesystem collisions. It would also likely fall foul of Apple's App Store restrictions (and, given the direction they've been heading with sandboxing, might not work at all in a year or two).

The only way I can see this working is if the path that is baked in is a relative path - i.e., if the "baked" conda paths are defined relative to something app specific (e.g., the location of the executable), then the app remains movable, because the baked paths are interpreted relative to something that can move. However, I can imagine that isn't something conda is set up to do easily.

[jaimergp]

Yep, I am afraid your understanding is spot-on. Some stuff like the shebangs in scripts can be reworked to be relative, but some binaries might contain the hardcoded PREFIX. I'll ask around to see if relative paths are even doable; if they are, substantial work will be needed for the conda internals.

[jaimergp]

Some relevant links:

• Relocatable conda environments conda/conda#1845
• request: conda "rename" (or conda "move") for changing name of an environment conda/conda#3097
• https://docs.conda.io/projects/conda-build/en/latest/resources/make-relocatable.html

But yes, anyway, all those feature requests would involve actively search-replacing prefixes.

[goanpeca]

Indeed the nonrelocalability of conda is an issue. I was thinking in the lines

One way of using this, with very little external tooling, is to solve the environment, download the tarballs, and ship a script to install those tarballs in the relevant location within Briefcase.

The actual app is the tarballs for all the packages needed, and the first "install". would effectively unpack the app on user space, if the app is moved running the app would either unpack things (first time) or run the actual app.

In this case there is the issue of a bash script calling another process which seems mac does not allow for security reasons.

[freakboy3742]

Ah - that's an interesting idea - each user gets their own conda install, manufactured as part of "first run". I can see how that could work. There's a bit of a bootstrapping problem - if the app is itself Python, how is that python run if the Conda environment doesn't exist yet - but I think it could be possible with some fancy footwork in a custom bootstrap executable.

As for macOS security - AIUI, that issue only exists for the primary executable - so if you my "myapp", which immediately starts a different process that becomes the app, that's a problem, because the process that is "the executable" is changing as a result of being run. However, if the core myapp is self contained, but invokes another script contained in the bundle, that should (AFAIK) be OK.

[goanpeca]

However, if the core myapp is self contained, but invokes another script contained in the bundle, that should (AFAIK) be OK.

Ohhh then I think this could work :)

[jaimergp]

There's a bit of a bootstrapping problem - if the app is itself Python, how is that python run if the Conda environment doesn't exist yet

We can use Micromamba, a static C++ binary that can bootstrap conda environments.

[goanpeca]

However, if the core myapp is self contained, but invokes another script contained in the bundle, that should (AFAIK) be OK.

Ok, on second read I do not follow this, but let's try some diagram here:

myapp/
    some-executabe          (the app executable?)
    micromamba              (a binary static library to bootstrap conda envs)
    packages/conda-packages (the actual files we need to unzip in user space with micromamba to create an environment)

some-executable here needs to call micromamba on first run, and then run the napari app that was installed on user space (some-location-on-mac/naparo/envs/napari-0-4-11/.../, but that seems to violate the security concern you mentioned @freakboy3742 ?

If the app is moved it only contains the origin al packages so running it again, would call the napari installed in a predefined location on user space.

As for macOS security - AIUI, that issue only exists for the primary executable

How could we get around this @freakboy3742?

[freakboy3742]

As long as some-executable is always "the app", I don't think it's a problem if it subshells to invoke micromamba. The security issue only emerges if micromamba becomes the primary manifestation of the app (i.e., some-executable starts micromamba and then exits. i.e., some-executable would be, at a high level:

def some-executable():
    if not bootstrapped:
        run_micromamba()
    show_app_ui()

The ideal situation in your case would be if run_micromamba() wasn't a subprocess, but was fully embedded executable code (ie., some-executable becomes a wrapper around the outside of the micromamba code).

For context - the issue we had that caused the recent change to briefcase is that "the app" was a shell script that started python -m myapp. As a result, the "actual app" wasn't the shell script that was started - and so, permission grants are given to the shell script, not the Python process. It also means the app appears in the process manager as "python.exe", not "myapp".

[mhsmith]

There's some discussion at #1270 (comment) about how this would be configured.

Also, as soon as we support this, we should add Briefcase itself to conda-forge and the Anaconda distribution. because packaging environments into apps is a notable gap in the conda ecosystem which we could fill.

I'm not sure how many of Toga's platform-specific dependencies have already been packaged for conda, but even if we only supported console apps at first, that would still be useful to a lot of people.