Upstream constructor changes from fork to project

[goanpeca]

🧰 Task

For the constructor bundle work @jaimergp used a fork and implemented a bunch if goodies.

The plan is to upstream this from the work https://github.com/jaimergp/constructor/tree/menuinst+branding

[jaimergp]

Constructor

• FIX: conda_interface should be able to handle alpha, beta, rc versions conda/constructor#496
• Add branding features on MacOS PKGs conda/constructor#469
• add windows installer signing conda/constructor#475
• add initialize_conda and register_python keys to control whether to show these options conda/constructor#507
• add support for additional files conda/constructor#500
• add support for multi_env installers conda/constructor#509
• Add cross-platform support for the new menuinst conda/constructor#474

Menuinst

• Rewrite for cross-platform compatibility (following conda/ceps#8) conda/menuinst#91

Conda

• Enable support for menuinst v2 (multiplatform shortcuts) conda/conda#11882

Conda Build

• add menuinst JSON file validation conda/conda-build#4405

Conda-standalone

• Menuinst debug (DNM) conda-forge/conda-standalone-feedstock#21

[liu-ziyang]

@jaimergp curious about the next steps on this issue, are these blocking our migration or?

[jaimergp]

Not blocking. Most of the constructor-only stuff landed on main, but the menuinst derived features are still pending.

[larsoner]

@jaimergp don't forget to check off conda/constructor#475 :)

[jaimergp]

Adding some task items here for my own sanity as I update I sync our fork to latest upstream versions to reduce the technical debt. We should have 90% parity or so!

• Cut a new menuinst alpha release from Rewrite for cross-platform compatibility (following conda/ceps#8) conda/menuinst#91
  • Rebuild and rerender jaimergp-forge/menuinst-feedstock#5
• Cut a new constructor noarch build from Add cross-platform support for the new menuinst conda/constructor#474
  • make noarch conda-forge/constructor-feedstock#63
  • Rebuild with rerender jaimergp-forge/constructor-feedstock#4
  • Update to 3.08 conda-forge/nsis-feedstock#20
• Cut a new conda build from Enable support for menuinst v2 (multiplatform shortcuts) conda/conda#11882
  • Rebuild jaimergp-forge/conda-feedstock#2
• Build a new conda-standalone using the three PRs mentioned above
  • Update jaimergp-forge/conda-standalone-feedstock#3
  • Osx arm64 fixes jaimergp-forge/conda-standalone-feedstock#4
  • support newest menuinst datas jaimergp-forge/conda-standalone-feedstock#5
• Upload builds to a new label bundle-tools-2 to not disrupt established workflows
• Update workflows to use the new label
  • sync recipe with conda-forge and use updated fork from bundle-tools-2 #42
  • Update installer infra to catch up with upstream (part two) #49
• Test everything works and if PermissionsError when accessing some directories with os module #38 is fixed

[jaimergp]

We have constructor 3.4.3 out now, which provides most of the functionality we have worked on during the last year, plus some! It's only missing the menuinst 2.x bits, so here's the plan:

Work items for menuinst

We now have a cep-devel branch on conda/menuinst! This allows us to iterate faster with smaller PRs. We no longer have to use a giga-commit PR with all the updates :D

• Finalize menuinst docs
  • Add documentation for the CEP prototype conda/menuinst#112
• CZI needs menuinst to support url:// protocol registration on Windows. This will need a change in the schema: either adding a new key for Windows options, or consolidating the existing Linux and macOS keys in a single url_protocols list in the global namespace.
  • Cross-platform support for URL and file type shortcut association (Windows) conda/menuinst#117
  • Add support for URL protocol and file type association (all OS) conda/menuinst#119
• Duplicated icons on Windows (AppUserModelID) Set AppUserModelId on Windows shortcuts conda/menuinst#133
• Revisit the permission elevation logic thanks to Re-license PyInstaller.isolated + its tests under MIT. pyinstaller/pyinstaller#7397
• Review if the current menuinst implementation is agnostic enough to support PyPI-sourced installations in the future. It shouldn't be too tied to conda!

Once these last technical requirements are implemented in the devel branch, we can kick off the upstreaming process for menuinst. Getting this accepted by the community would allow us to stop forking, maintaining and packaging conda, conda-standalone, constructor and menuinst on our own! This task needs various items that need to be completed sequentially:

• 1. Update the CEP (Conda Enhancement Proposal) with latest implementation details (new features, schema changes, etc), and propose for vote + approval. Iterate on the implementation as needed.
  • Prepare menuinst for multiplatform support conda/ceps#8
• 2. Merge cep-devel into main conda/menuinst#141
• 3. Add defensive pins for downstream users of menuinst <2. This won't be needed in principle, since menuinst 2 has backwards compatibility.
• 4. Release and package menuinst 2.0 on conda-forge & defaults.
• 5. Merge the conda PR for the new menuinst support and wait for release.
  • Enable support for menuinst v2 (multiplatform shortcuts) conda/conda#11882
• 6. Update conda-standalone accordingly.
  • See August 2023 PRs
• 7. Merge the menuinst-cep PR on constructor and release
  • Add cross-platform support for the new menuinst conda/constructor#474

Once we have all these pieces in conda-forge, we will be able to:

• Dismantle jaimergp-forge
• Dismantle the napari/label/bundle_tools_2 channel
• Migrate the installer pipelines to use conda-forge - Use upstream constructor stack (drop napari/bundle_tools_*) #103
• Amend (more like trim) our internal documentation to mostly rely on the official upstream docs, and simply document the glue that holds it together! - Update packaging guide by removing outdated constructor fork details docs#319

---

Nice to have, maintainability oriented, with the help of the community:

• Package conda-standalone from the new conda/conda-standalone repo, instead of using the feedstock as a source provider.
• Implement a better maintenance model for the conda constructor ... CLI helper on conda-standalone (not to be confused with constructor itself!). Right now this is an implementation detail on conda/conda-standalone. Probably best if done with a conda subcommand plugin. It might need its own (coupled) CEP.
• Create a conda-menuinst plugin to handle this logic outside conda (preferred long term, but it requires a new plugin hook in conda).

[jaimergp]

The above is not done yet, but we are going to update our installers to use the last features in menuinst. This is the build order:

• Create new channel label: napari/label/bundle_tools_3
• Release new menuinst build
• Update from main and release new conda build
• Update from main and release new constructor build
• Release new conda-standalone build
• Upgrade to bundle_tools_3 Prepare new constructor stack release #75
  • Build new installers
  • Test new installers - see https://github.com/napari/packaging/actions/runs/5410466633

[jaimergp]

A last update on this:

• We released menuinst v2 in November I think? We have released two patch versions since then. We have 2.0.2 right now!
• conda 23.11.0 (released in December) ships compatibility for menuinst v2
• constructor 3.6.0 shipped compatibility with menuinst v2 last week
• conda-standalone 23.11.0 with, you guessed it, menuinst v2 compatibility, has just been merged! This includes manual builds for osx-arm64.
• Use upstream constructor stack (drop napari/bundle_tools_*) #103 drops napari/label/bundle_tools_* and uses the conda-canary nightlies. I just need to adjust it so it uses conda-forge straight once all the releases land.

So, all in all, it sounds like we are basically done! This won't reach our installers till napari v0.5 is released, though. The 0.4.19 pipeline will still use the "old yet trusted" forked version in main. Once that lands, we can merge all the open PRs and upgrade to the upstream stack! We'll need some QA at some point to ensure everything is ✨.

In the meantime, I'll send a PR to update our packaging docs. An by update I mean "dramatically simplify" because now we can just omit all the hacky details of our fork.

It only took us 2.5 years 🚀

[hoechenberger]

👏👏👏

[jaimergp]

cc @mrclary, in case this is relevant for Spyder. No further need for napari/label/bundle_tools_*.