darwin: installBinaryPackage init and refactoring of darwin apps

[afh]

Description of changes

• Add darwin.installBinaryPackage unifying and simplifying how binary darwin applications are installed.
• Refactor a few packages to use darwin.installBinaryPackage
• Clean-up of packages, e.g. ensure phase pre- and post-hooks are run

NOTA BENE: This is a draft PR to get the conversation going on whether this is headed into the right direction and folks see this as beneficial and helpful.

💭 A Few Thoughts

• There are several ../os-specific/darwin/ packages declared in pkgs/top-level/all-packages.nix do folks agree that these are better declared in pkgs/top-level/darwin-packages.nix?
• TODO: Migrate os-specific/darwin packages from all-packages.nix to darwin-packages.nix if the general sentiment is positive on the previous question.

grep os-specific/darwin pkgs/top-level/all-packages.nix | awk -F'[=]' '/ *#/{next} {print $1}'

• asitop
• goku
• grandperspective
• hexfiend
• kwm
• khd
• m-cli
• pam-reattach
• reattach-to-user-namespace
• skhd
• qes
• pngpaste
• sketchybar
• spacebar
• swiftbar
• airbuddy
• aldente
• coconutbattery
• defaultbrowser
• karabiner-elements
• osx-cpu-temp
• macfuse-stubs
• osxsnarf
• plistwatch
• noah
• rectangle
• shortcat
• smimesign
• swiftdefaultapps
• sensible-side-buttons
• raycast
• dockutil
• mas
• mysides
• yabai
• ghc-standalone-archive
• duti
• wifi-password

• TODO: Refactor packages using undmg or _7zz as their nativeBuildInput to unpack a prebuilt darwin binary, if this PR is generally viewed positively.

nix run nixpkgs#silver-searcher -- -G '\.nix' -l '(undmg|_7zz)' | rev | cut -d/ -f2 | rev

• prl-tools
• install-binary-package
• aldente
• macfuse
• airbuddy
• rectangle
• raycast
• grandperspective
• hexfiend
• sensible-side-buttons
• caffeine
• archi
• 7zz
• undmg
• peazip
• realvnc-vnc-viewer
• unnaturalscrollwheels
• disk-inventory-x
• jprofiler
• composer
• mkl
• fpc
• fpc
• top-level
• spotube
• spacedrive
• mos
• iina
• suspicious-package
• rectangle-pro
• warp-terminal
• kbreakout
• libkmahjongg
• knights
• killbots
• bomber
• knetwalk
• kpat
• kdiamond
• granatier
• libkdegames
• kapman
• kgoldrunner
• kigo
• notesnook
• sequelpro
• 1password-gui
• joplin-desktop
• tableplus
• keeweb
• obsidian
• losslesscut-bin
• roam-research
• darwin
• localsend
• lens
• discord
• vk-messenger
• slack
• caprine-bin
• breitbandmessung
• synology-drive-client
• bin
• scilab-bin
• yesplaymusic
• spotify
• reaper
• p4v
• radicle-upstream
• pika
• pkgs
• zandronum
• katawa-shoujo
• anki

• What needs to be documented where, so contributors can discover darwin.installBinaryPackage and learn about its details?
• If initial review feedback is positive loop in maintainers of refactored packages and ask for their input
• @reviewers Please see inline comments and questions, especially on pkgs/os-specific/install-binary-package/*

ℹ️ All treewide: refactor to use darwin.installBinaryPackage* commits will be squashed into one. For now it's easier for me to work on this with individual commits.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[afh]

• Is this the best way to set sensible defaults, but allow customisation for packages that need it?
• Are there any other dont* flags that are relevant here?

[afh]

Does this implementation allow for enough flexibility for uncommon use-cases (see karabiner-elements) while still making "good" assumptions about the general layout/content of DMGs and PKGs, e.g. unpack any Payload* using bsdtar contained in a .pkg file?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/proposal-for-installing-prebuilt-binaries-on-darwin/40843/1

[DataHearth]

I like the abstraction made by the dynamic unpack hook for DMG and PKG. Package complexity will drop (IMO) and package's lines reduced by a good margin.

But I think we should make sure every possible option is covered. As for some specific behavior, this might even add more complexity inside the package and possibly inside the hook.

If this solution is chosen, we should create a detailed documentation to avoid looking everywhere to find options, default behaviors, etc.

Love it 👍

[afh]

Thank you for taking the time to have a look at this and your comment, @DataHearth, very much appreciated!

In your mind what is a good way to ensure that "every possible option is covered"? My first approach would be to refactor the packages in pkgs/os-specific/darwin and the ones that have undmg or _7zz as a build input.

Where would be a good place for such documentation? Somewhere in docs or rather pkgs/os-specific/darwin/README.md?

[afh]

Looking at fetchzip I wonder if that could be changed or creating a similar fetcher, e.g. fetchdmgpkg would be more? helpful than the unpack-dmg-pkg hook? 🤔

[ShamrockLee]

I love the idea of having a unified way to package binaries for MacOS.

Regarding the treewide change,

1. hello: refactor to use darwin.installBinaryPackage would be easier to read comparing to treewide: refactor to use darwin.installBinaryPackage (hello).
2. It would make this PR easier to review if we change only some packages for demonstration and testing purpose, and split others into subsequent PRs.

[afh]

Thanks for your input, @ShamrockLee, much appreciated!

1. Yes, that is a working title and when this draft PR is promoted to a PR I'll squash all those commits into one with an updated commit message

2. Agreed 🙂

Any opinion or advice you may have on the "💭 A Few Thoughts" I listed in the initial description?

[DataHearth]

Hey! Regarding creating a fetcher, I'm not a big fan of. As the "universal" way seems to fetch then unpack for most scenarios, I would keep it as a new package helper.

I think the best way to find out if you missed something, it's to check other packages which are using an unpack like 7z or unzip, etc... list all available options, and see if they're applicable to dmg and pkg files. And of course, test it on packages ^^.

I agree with @ShamrockLee , I think this would me much easier to review for a code maintainer with only 1 or 2 packages as examples.

[afh]

Thanks for the feedback, @DataHearth, that's helpful.

I'll do some code level analysis and prepare some notes. In addition to that I'll split this PR into two: one for the install binary package changes and one for the treewide package changes (with a possible follow-up).

[afh]

As requested this PR now affects fewer files and the majority of the actual package rewrites to use the changes proposed in this PR will be in #293961.
Please allow for some more time for a more methodical analysis and summarising the findings in a few notes (posted here later).

[afh]

Kindly requesting review from @spease to compare the approaches taken here and in #147567

[reckenrode]

FWIW, iTerm2 is another binary package that might be suitable for this.

[spease]

Kindly requesting review from @spease to compare the approaches taken here and in #147567

Thanks for reaching out. Unfortunately, I'm uncomfortable making further contributions to Nix at this time.

[afh]

Thanks for taking the time to reply, @spease.

[pcasaretto]

Hey afh!
One common idiom I see is using makeWrapper to make a bundled binary available at the command line.
I'm wondering if we could facilitate that here.

[afh]

I've come across that recently more often too, @pcasaretto, and yes, I think it's something that could be facilitated here too. I shall make some time to factor this in, in the coming weeks.

[pcasaretto]

I'm also the mantainer of vlc-bin, another suitable binary package. How can I contribute further?

[afh]

Happy to hear you're interested in helping moving this forward, @pcasaretto 😃 Let's coordinate next week (cw 32), if that works you?

[pcasaretto]

I'm on the last week of my vacation so I have more free time now. Starting a new job next week so I can't make any promises, but I'm sure we can work something out!

[afh]

I think the work presented in #350504 is useful and I'd like to incorporate it into this Draft once merged.

[ShamrockLee]

Are you still working on this PR? It would be handy for Darwin binary packaging.

[afh]

Apologies for the late response, @ShamrockLee.
Happy to continue work on this, but instead of trying to wade through all binary darwin packages and consolidate them all at once I think a more incremental approach is more feasible and manageable. What do you think?

[ShamrockLee]

It's common to split a large PR into smaller ones. This strategy helps push substantial changes (e.g., #352976 (comment)) forward.