Add support for URL protocol and file type association (all OS)

[aganders3]

Description

In development of #117 we realized macOS does not support passing URLs via argv when opening from a URL. See also #118 for more discussion.

This draft PR investigates an experimental launcher for the mac bundle that uses AppKit in order to respond to Apple Events, specifically to implement application(_:open:). This allows us to respond to Apple Events associated with opening a custom URL scheme. The launcher first launches the main application via NSWorkspace and hides itself, listens for relevant Apple Events and dispatches them to another CLI program, then quits if the main application exits.

The new app bundle structure is a bit more complicated, and is basically the current structure nested as a resource in a new bundle using the proposed launcher.

menuinst app name.app
└── Contents
    ├── Entitlements.plist
    ├── Info.plist  <------ this is where we register the url scheme, etc.
    ├── MacOS
    │   └── menuinst-app-name    <----- this is the new Swift launcher from this PR
    ├── PkgInfo
    ├── Resources
    │   ├── menuinst app name.app  <----- this is basically the current app bundle, but now does *not* register the url scheme
    │   ├── menuinst icon.icns
    │   └── open-url    <------ the launcher calls this with the url as arguments whenever a URL is opened
      ...

The open-url script/program should be somehow user-configurable. Right now it is not included in this PR. Here's a very barebones proof-of-concept that would communicate with a main running app via socket:

#!/bin/bash

set -ueo pipefail

for i in 1 2 3 4 5
do
    echo "${1#menuinst://}" | nc localhost 40256 && break || sleep 5
done

There's a known issue where a second dock icon will flash (very) briefly on startup and when opening a URL in the running app.

Checklist

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

N.B. @jaimergp

[conda-bot]

We require contributors to sign our Contributor License Agreement and we don't have one on file for @aganders3.

In order for us to review and merge your code, please e-sign the Contributor License Agreement PDF. We then need to manually verify your signature. We will ping the bot to refresh the PR status when we have confirmed your signature.

[aganders3]

There's a known issue where a second dock icon will flash (very) briefly on startup and when opening a URL in the running app.

I believe this is fixed with the latest commit!

[jaimergp]

Ok, I added some commits that (a) integrate the now merged #117, and (b) add some tests and examples. The tests currently fail because the association is not working somehow. To be investigated but now we have CI eyes :D

[jaimergp]

I think we also need to handle this file (from SO):

The file you seek is ~/Library/Preferences/com.apple.LaunchServices/com.apple.launchservices.secure.plist.

It holds an array called LSHandlers, and the Dictionary children that define an LSHandlerURLScheme can be modified accordingly with the LSHandlerRole.

[aganders3]

For this PR to work properly (no blip in the dock) I think the CFBundleTypeRole actually needs to be omitted. This took me hours to figure out. I was so desperate I even opened a thread on Apple's developer forum: https://developer.apple.com/forums/thread/725410

[jaimergp]

Oh, nice, thanks. I added this to the docs.

[jaimergp]

The file is binary but can be opened with Python's plistlib. This is what mine looks like:

{
  "LSHandlers": [
    {
      "LSHandlerURLScheme": "https",
      "LSHandlerRoleAll": "com.google.chrome",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerURLScheme": "com.todoist",
      "LSHandlerRoleAll": "com.todoist.mac.todoist",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      },
      "LSHandlerRoleAll": "com.microsoft.vscode",
      "LSHandlerContentTagClass": "public.filename-extension",
      "LSHandlerContentTag": "4"
    },
    {
      "LSHandlerURLScheme": "todoist",
      "LSHandlerRoleAll": "com.todoist.mac.todoist",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerContentType": "public.html",
      "LSHandlerRoleAll": "com.google.chrome",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerURLScheme": "prli",
      "LSHandlerRoleAll": "com.parallels.desktop.console",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      },
      "LSHandlerRoleAll": "com.parallels.sharedapplink.agent",
      "LSHandlerContentTagClass": "public.filename-extension",
      "LSHandlerContentTag": "lnk"
    },
    {
      "LSHandlerURLScheme": "webcal",
      "LSHandlerRoleAll": "com.google.chrome",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      },
      "LSHandlerRoleAll": "com.microsoft.vscode",
      "LSHandlerContentTagClass": "public.filename-extension",
      "LSHandlerContentTag": "8"
    },
    {
      "LSHandlerURLScheme": "web+stellar",
      "LSHandlerRoleAll": "keybase.electron",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerURLScheme": "mailto",
      "LSHandlerRoleAll": "com.google.chrome",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerURLScheme": "keybase",
      "LSHandlerRoleAll": "keybase.electron",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerURLScheme": "zoomphonecall",
      "LSHandlerRoleAll": "us.zoom.xos",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      },
      "LSHandlerRoleAll": "com.microsoft.vscode",
      "LSHandlerContentTagClass": "public.filename-extension",
      "LSHandlerContentTag": "in"
    },
    {
      "LSHandlerURLScheme": "facetime",
      "LSHandlerRoleAll": "com.apple.facetime",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    },
    {
      "LSHandlerContentType": "public.url",
      "LSHandlerRoleViewer": "com.google.chrome",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleViewer": "-"
      }
    },
    {
      "LSHandlerURLScheme": "http",
      "LSHandlerRoleAll": "com.google.chrome",
      "LSHandlerPreferredVersions": {
        "LSHandlerRoleAll": "-"
      }
    }
  ]
}

[aganders3]

Good to know - from my experiments the application is registered automatically when first launched, but it could be useful to directly manipulate this list (e.g. when installing/uninstalling multiple versions).

I think another interface to this file is the lsregister command:

❮ /System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister --help                          
lsregister: [OPTIONS] [ <path>... ]
                      [ -apps <domain>[,domain]... ]
                      [ -libs <domain>[,domain]... ]
                      [ -all  <domain>[,domain]... ]

Paths are searched for applications to register with the Launch Service database.
Valid domains are "system", "local", "network" and "user". Domains can also
be specified using only the first letter.

  -delete       Delete the Launch Services database file. You must then reboot!
  -kill         Reset the Launch Services database before doing anything else
  -seed         If database isn't seeded, scan default locations for applications and libraries to register
  -lint         Print information about plist errors while registering bundles
  -lazy n       Sleep for n seconds before registering/scanning
  -r            Recursive directory scan, do not recurse into packages or invisible directories
  -R            Recursive directory scan, descending into packages and invisible directories
  -f            force-update registration even if mod date is unchanged
  -u            unregister instead of register
  -v            Display progress information
  -gc           Garbage collect old data and compact the database
  -dump [table] Display full database contents after registration
  -h            Display this help

[jaimergp]

Running that command on the .app shortcut doesn't modify the plist database, so I guess something's off with the example I am using. Looking into it.

[aganders3]

Or I'm just wrong!

[jaimergp]

Tried a few more things with interactive debugging:

• A fresh login to the VM reveals that the apps are still running the nc listener, with the file handles open:

$ lsof -c nc
lsof: WARNING: can't stat() vmhgfs file system /Volumes/VMware Shared Folders
      Output information may be incomplete.
      assuming "dev=36000002" from mount table
COMMAND  PID   USER   FD   TYPE             DEVICE SIZE/OFF                NODE NAME
nc      4133 runner  cwd    DIR                1,4      640                   2 /
nc      4133 runner  txt    REG                1,4   203632 1152921500312782147 /usr/bin/nc
nc      4133 runner  txt    REG                1,4  2177216 1152921500312783021 /usr/lib/dyld
nc      4133 runner    0r   CHR                3,2      0t0                 314 /dev/null
nc      4133 runner    1w   REG                1,4        0         12896141635 /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpmbdiqvhmfile_types.json.out
nc      4133 runner    2u   CHR                3,2      0t0                 314 /dev/null
nc      4133 runner    3u  IPv4 0x5bf5f966daa0a557      0t0                 TCP *:40257 (LISTEN)
nc      4348 runner  cwd    DIR                1,4      640                   2 /
nc      4348 runner  txt    REG                1,4   203632 1152921500312782147 /usr/bin/nc
nc      4348 runner  txt    REG                1,4  2177216 1152921500312783021 /usr/lib/dyld
nc      4348 runner    0r   CHR                3,2      0t0                 314 /dev/null
nc      4348 runner    1w   REG                1,4        0         12896141682 /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
nc      4348 runner    2u   CHR                3,2      0t0                 314 /dev/null
nc      4348 runner    3u  IPv4 0x5bf5f966daa07007      0t0                 TCP *:40256 (LISTEN)

• Running echo XXX | nc localhost 40256 (or 40257) makes the process end, and the file handle is closed with the sent output (XXX in this example).

$ echo TEST | nc localhost 40256
$ echo TEST | nc localhost 40257
$ cat /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpmbdiqvhmfile_types.json.out
TEST
$ cat /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
TEST
$ lsof -c nc
# empty output

• Running open ~/Applications/CustomURLAssociation.app leaves the process hanging, but again ncing manually closes it.

$ open ~/Applications/CustomURLAssociation.app/
$ lsof -c nc
lsof: WARNING: can't stat() vmhgfs file system /Volumes/VMware Shared Folders
      Output information may be incomplete.
      assuming "dev=36000002" from mount table
COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF                NODE NAME
nc      31222 runner  cwd    DIR                1,4      640                   2 /
nc      31222 runner  txt    REG                1,4   203632 1152921500312782147 /usr/bin/nc
nc      31222 runner  txt    REG                1,4  2177216 1152921500312783021 /usr/lib/dyld
nc      31222 runner    0r   CHR                3,2      0t0                 314 /dev/null
nc      31222 runner    1w   REG                1,4        0         12896141682 /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
nc      31222 runner    2u   CHR                3,2      0t0                 314 /dev/null
nc      31222 runner    3u  IPv4 0x5bf5f966daa07007      0t0                 TCP *:40256 (LISTEN)
$ cat /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
# empty
$ echo TEST | nc localhost 40256
$ cat /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
TEST
$ lsof -c nc
# empty

• Running open menuinst://test/ results in the same outcome. Interestingly, you can autocomplete the menuinst protocol, which must mean is registered somewhere. This process returns 0 too, meaning that it thinks it succeeded.

$ open menuinst://test/
$ lsof -c nc                                                                               
lsof: WARNING: can't stat() vmhgfs file system /Volumes/VMware Shared Folders
      Output information may be incomplete.
      assuming "dev=36000002" from mount table
COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF                NODE NAME
nc      34562 runner  cwd    DIR                1,4      640                   2 /
nc      34562 runner  txt    REG                1,4   203632 1152921500312782147 /usr/bin/nc
nc      34562 runner  txt    REG                1,4  2177216 1152921500312783021 /usr/lib/dyld
nc      34562 runner    0r   CHR                3,2      0t0                 314 /dev/null
nc      34562 runner    1w   REG                1,4        0         12896141682 /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
nc      34562 runner    2u   CHR                3,2      0t0                 314 /dev/null
nc      34562 runner    3u  IPv4 0x5bf5f966daa07007      0t0                 TCP *:40256 (LISTEN)
$ cat /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
$ echo TEST | nc localhost 40256
$ cat /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
TEST
$ lsof -c nc

• I also ran ~/Applications/CustomURLAssociation.app/Contents/MacOS/customurlassociation directly. The executable runs and waits silently. nc ing manually has the same effect here.

$ ~/Applications/CustomURLAssociation.app/Contents/MacOS/customurlassociation 
$ lsof -c nc
lsof: WARNING: can't stat() vmhgfs file system /Volumes/VMware Shared Folders
      Output information may be incomplete.
      assuming "dev=36000002" from mount table
COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF                NODE NAME
nc      38560 runner  cwd    DIR                1,4      640                   2 /
nc      38560 runner  txt    REG                1,4   203632 1152921500312782147 /usr/bin/nc
nc      38560 runner  txt    REG                1,4  2177216 1152921500312783021 /usr/lib/dyld
nc      38560 runner    0r   CHR                3,2      0t0                 314 /dev/null
nc      38560 runner    1w   REG                1,4        0         12896141682 /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
nc      38560 runner    2u   CHR                3,2      0t0                 314 /dev/null
nc      38560 runner    3u  IPv4 0x5bf5f966daa07007      0t0                 TCP *:40256 (LISTEN)
$ echo TEST | nc localhost 40256
$ lsof -c nc

At this point we can conclude that the listener command is sucessfully run in all cases, but the url-handler is not delivering the arguments to the listener.

If I start the main executable:

$ ~/Applications/CustomURLAssociation.app/Contents/MacOS/customurlassociation 
# waits

And then call the handler manually:

$ ~/Applications/CustomURLAssociation.app/Contents/Resources/handle-url menuinst://test/

The main executable closes successfully, and the output file contains the expected output.

$ cat /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp6um0a0pvurl_protocols.json.out
menuinst://test/

So the problem is not with the url-handler itself. Modifying it to create extra output before nc (echo SOMETHING > ~/log.txt) reveals that the main executable is NOT calling the url-handler at all. So the problem is in the Swift launcher.

My next step will be to modify the Swift launcher so it writes sentinel files to disk based on the different tasks. Ideally we would use the system logging, but it doesn't work on the CI runner for some reason. This will tell us more about where the problem is.

I am assuming the macOS system on the CI doesn't have all the events enabled or something like that? We should check https://github.com/actions/runner-images/tree/main/images/macos for hints.

[aganders3]

Thanks! I observed similar but its nice to see this documented so methodically. I will do some research on Apple Events on remote/headless runners.

[jaimergp]

Erm... WHY did it pass now? 😂

[aganders3]

I was going to suggest perhaps it's prompting for some permissions and hanging? There are a few possibly related issues on the runner-images repo, for example actions/runner-images#553.

Now that it passed though...

[jaimergp]

Yea, I am clueless 🤷 My only ideas are:

• The dashed name of the logger was not cool in the runner.
• Recompiling the binary made it pass something

But a total mystery! The pending permissions dialog would have been a nice explanation too.

[jaimergp]

Hey @aganders3 @goanpeca, if you have time to take a look here and leave any comments, that'd be appreciated. It ended up being quite the PR, but I am happy with the results. I think it's good enough for a merge to the dev branch!

Thank you!

[aganders3]

Thanks for all the work on this! There are a couple small things but this looks to be in decent shape to me and merging to the dev branch seems okay.

We should probably change the title/description of the PR to reflect the additional content.

Also probably out of scope for this PR, but I would encourage a bit more testing of error handling and what happens if some step in the process fails. Should menuinst attempt to revert installation on any error, or perhaps log the errors somewhere? I think now there is a chance that some final step (for example, registering with launchservices) could fail. The user may see an exception, but the shortcut is still created.

[jaimergp]

Should menuinst attempt to revert installation on any error, or perhaps log the errors somewhere? I think now there is a chance that some final step (for example, registering with launchservices) could fail. The user may see an exception, but the shortcut is still created

The remove() endpoint should revert everything, even if the shortcut wasn't created, I think. But yes, it might need more testing, and I agree it should be done in a different PR.

[jaimergp]

I used your snippet and adjusted it a bit so it can compile. One problem with the deprecated version is that the opened process will be left lingering around, which might not be ideal... but at least it will launch something. Hopefully users on <10.15 are not too many :)

[jaimergp]

Let's do this.