Create a rudimentary MacOS packager

[baconpaul]

@baconpaul will do this tomorrow, and it matches some of the issues in #83 and #84.

To install Surge you need to

1. move Surge.component to the plugins folder
2. move the resources to the Application Support area
3. if Surge has never been installed, remover the audio unit cache

This should be done properly in a .pkg but that’s a bit of a pain. So I propose that for now I just add a script which

1. makes a dmg with the appropriate name
2. Puts the component and resources and a script and a readme in there
3. The readme says run the script but don’t if you don’t know how to debug your Mac
4. The script does the above from the images in the dmg

Then we can at least share dmgs for alpha releases more easily

Again, I’m away from my coding today but wanted to memorialize this as I think it will help get us some more Mac alpha testers, even before all the code is merged. (Like: I could run this from my zoom-ui branch and get all my bug fixes in while waiting for @kurasu to review the stream of PRs over the next few days which will come if/once #69 is merged).

[kzantow]

Some discussion on #87, but summarizing: what if we just provide one of those "drag this to this folder symlink" type of installers (or none at all, which is typical for vst plugins), and I (or someone else) could whip up some code that runs at startup to sync the /Library/Application Support/Surge/... required files from things included in the bundle? The biggest question I have about this is the configuration.xml -- someone could hose this and surge would not work. In terms of adding presets to the waveform and effects sections, would it make more sense to just merge customizations in with a locally stored version containing the bare minimum from the bundle, so even without any files there Surge would still work?

[kzantow]

Of course I didn't consider permissions... sigh

[baconpaul]

The permissions thing is solved by using ~/Library.

Yes you can hose ~/Library/Application Support and break surge. I think some better error reporting in that case is useful. Here's the current code in SurgeStorage.cpp

   string snapshotmenupath = datapath + "configuration.xml";

   if (!snapshotloader.LoadFile(snapshotmenupath.c_str())) // load snapshots (& config-stuff)
   {
#if !MAC && !__linux__
      MessageBox(::GetActiveWindow(), "Surge is not properly installed. Please reinstall.",
                 "Configuration not found", MB_OK | MB_ICONERROR);
#endif
   }

which explains why the MAC and linux users aren't seeing this error! I'll open a separate issue for that.

But I don't want to change things like configuration file behavior and split it and what not now; I think we may be better focused in getting a regular engine which can build, run, and package macOS with few feature changes. Of course I broke this rule by implementing zoom!

As to a "drag this folder to their" type thing that would be pretty good but I can't quite figure out how to automate it; and it would require two drags (one for the component and one for the ~/Library/Application Support). I can automate building a dmg with all the assets and a shell script. Let me google a bit and see if that would let me automate building a .pkg which is what we really want.

[kzantow]

@baconpaul can punt on the installer for a while, if I get all the resources included in the bundle and dumped into ~/Library/... I found this for the drag-to-folder type of installer; looks a bit manual though: https://gist.github.com/jadeatucker/5382343

[baconpaul]

Yeah hdiutil can make a dmg but that hand step to set the background is a pain. I'm just reading about .pkg files quickly. That may be the best route.

[baconpaul]

Oh and #89 has that surge storage reminder. Reading "man productbuild" now. Will report back.

[esaruoho]

Hmm, this is interesting:
https://www.techrepublic.com/article/pro-tip-use-terminal-to-create-packages-for-software-deployment/

looks like it could be done. I'll see if I can create a Surge.component installer.

[esaruoho]

$ sudo pkgbuild --install-location /Library/Audio/Plug-Ins/Components/ --component Surge.component SurgeAUinstaller.pkg
Password:
pkgbuild: Adding component at /Users/esaruoho/Downloads/Surge.component
pkgbuild: Wrote package to SurgeAUinstaller.pkg

fingers crossed

[baconpaul]

http://thegreyblog.blogspot.com/2014/06/os-x-creating-packages-from-command_2.html

shows you a lot of it. I got stuck though getting it to install in local.

#!/bin/sh
#
# http://thegreyblog.blogspot.com/2014/06/os-x-creating-packages-from-command_2.html
#
# is very useful

echo "BUILDING RELEASE"

xcodebuild build -configuration Release -project surge-au.xcodeproj || exit 1
./package-au.sh

echo ""
echo "CREATING TARGET FOLDERS"

TMP=/tmp

rm -rf $TMP/Surge.dst
mkdir -p $TMP/Surge.dst
mkdir -p $TMP/Surge.dst/component
mkdir -p $TMP/Surge.dst/appsupport

# for PKGBUILD but not for now
# mkdir -p $TMP/Surge.dst/install-scripts

mv ./products/Surge.component $TMP/Surge.dst/component
(cd resources/data && tar cf - . ) | ( cd $TMP/Surge.dst/appsupport && tar xf - )
# For PKGBUILD
# cp ./resources/osx-resources/install-scripts/* $TMP/Surge.dst/install-scripts
# for HACKY
cp ./resources/osx-resources/install-scripts/README.txt $TMP/Surge.dst


# I originally had done
# pkgbuild --analyze --root /tmp/Surge.dst/ --scripts /tmp/Surge.dst/install-scripts resources/osx-resources/Surge_pkg.plist

# rm -f /tmp/Surge.pkg

# pkgbuild --root /tmp/Surge.dst/ \
#         --scripts /tmp/Surge.dst/install-scripts \
#         --install-location ~/Library/Audio/Plug-Ins/Components/ \
#         --component-plist resources/osx-resources/Surge_pkg.plist \
#         /tmp/Surge.pkg

# which is "the right thing" but I just don't know what I'm doing enough to make it work.
hdiutil create -fs HFS+ -srcfolder /tmp/Surge.dst/ -volname Surge /tmp/Surge.dmg

is what I tried this morning.

If you'd like to take a swing at this that would be great. It's frustrating me some. I'll see if I can fix a couple of the other bugs like filesystem and no warning on corrupt.

[baconpaul]

Ahh yeah if yours works, then you build two pkgs (one for the component one for the resources) then use product build --synthesize to group them.

Like I said, I've just been googling like you and it didn't work for me. Hope my research helps!

[esaruoho]

I managed to get Surge.component to appear in /Library/Audio/Plug-Ins/Components/Surge.component..

I could attempt to create one that copies files from resources/data to that other folder. I'll see what I can do.. Ideally I'd like for there to be just one .PKG file that installs the AudioUnit and the resources.

[baconpaul]

Yeah you end up with one pkg. But the way you do that is to make two pkg (one for Surge.component and one for resources/data) and then combine those two pkgs into another pkg using product build.

So lets say you have SurgeComponent.pkg which installs the component (which you do! Awesome) and SurgeResources.pkg which install the resources (which you are working on).

Then apparently you can do

productbuild --synthesize --packate SurgeComponent.pkg --package SurgeResources.pkg Surge.pkg

then Surge.pkg will just install both. So then the only other thing you would do is

mkdir /tmp/surge.imagesource
mv Surge.pkg /tmp/surge.imagesource
hdiutil create -fs HFS+ -srcfolder /tmp/surge.imagesource -volname Surge /tmp/Surge.dmg

and you will get a dmg

Getting the first two pkgs working was what stuck me.

[esaruoho]

Fingers crossed that this works:

sudo pkgbuild --install-location ~/Library/Application\ Support/Surge/ --component resources/data SurgeDataResources.pkg

I have no idea if it should or shouldn't.I've wiped my Surge -folder just in case from Application Support.

[esaruoho]

nope, no go:

pkgbuild: Adding component at /Users/esaruoho/work/surge/resources/data
pkgbuild: error: Path "/Users/esaruoho/work/surge/resources/data" is not a valid bundle component (using destination path "/Users/esaruoho/work/surge/resources")

[esaruoho]

@baconpaul btw your work, it's good to put the plugin in /Library/Audio/Plug-Ins/Component and not ~/Library/Audio/Plug-Ins/Component. 👍

[asimilon]

Didn't see this issue until now. I built an installer using Packages, but using pkgbuild would definitely be preferable as it drops the dependency.
There is an example on using productbuild to create a blob of the necessary packages here, although not sure about how to deploy the resources, and the pkgbuild & productbuild documentation is really difficult to find, since Apple would prefer everything deployed via AppStore.

[baconpaul]

Now that @kzatow has resources in the bundle we really only need the one step pkg build on the component which is above I think.

And then permissions to make a github release too I suppose

But the two step stuff i talked about above is all obviated now the component populates application support if it isn’t there

[kzantow]

Yeah, I'd say given the current state of things one of 2 options is the best:

1. just distribute the Surge.vst, Surge.component etc. directly in .zip format (since resources are included etc.)
2. create one of those "drag to the ____ plugins folder" type installers like this:

https://stackoverflow.com/questions/10574223/how-to-create-drag-and-drop-installer-for-macos-x

[esaruoho]

the drag'n'drop installing is easy to do for .vst .vst3 .component on mac, but the data/resources copying is the things that throws me off - ideally the user should not be asked to do this ("hey, so, unzip this folder to your ~/Library/Application Support/Surge" - not cool, never seen it done like that).

but maybe there IS a way of doing it, as @baconpaul posted a rather complex link to it recently.

[esaruoho]

@kzantow query, when you say "distribute .vst .component in zip" and then follow it up with "since resources are included etc" - what does since resources are included mean? did something change within the components themselves, so that they now contain the resources/data folder? did i really drop the ball that hard and didn't notice that happening? confused

[kzantow]

@esaruoho yes, the VST and AU bundles include all the resources and automatically put them in ~/Library/Application Support/Surge as appropriate now, there's no need to run a script or anything. One caveat: they only do this in the case that directory doesn't exist (e.g. the first time you run Surge / when it's first scanned by a host). So if you've manually created that directory, the bundled resources will not be copied.

[esaruoho]

@kzantow wow that's awesome! that really, actually is, awesome. thanks!
ignore whatever i said.

[esaruoho]

@kzantow also you might find this PR interesting - i think maybe it has something going on in there that needs to be taken out #98

[kurasu]

for BW we've been using this tool to build scripted dmgs: https://github.com/andreyvit/create-dmg

For surge 1.5 I just created a sparseimage manually and converted it to a bundle. I could provide those via e-mail to look at but I don't want to put those in the repo.

[baconpaul]

create-dmg seems to be exactly what we want to do @kzantow's idea in a batch. So basically make all 3 components, make a directory with them and the application, make the soft links, then create-dmg with the 4 links and background and you get the @kzantow 3-hours-ago thing. Thank you!

[asimilon]

Just my opinion, but I don't think the plugin should be installing things at run-time, that is the job of the installer. As a temporary solution I think it is OK, but longer term I will work on creating a script that does it using pkgbuild and productbuild (rather than having an unneeded dependency on the Packages application, that is really just a GUI wrapper for those commands). That leaves the issue of an unsigned installer, but I believe you also have the same issue that unsigned DMGs are blocked by GateKeeper, so the user has to jump through the same hoops to install anyway.

[esaruoho]

@asimilon it really feels like the sooner we get to a .pkg / .dmg combo that installs the files (VST2,VST3,AU) and the plugin simple runs and doesn't muck about with files, the better :)

man i wish these pkgbuild things were easier to run - really thought i had a solution for multi-plugin installing but naah. just not enough.

[baconpaul]

I completely agree. Just I don’t know the magic for how to use pkgbuild.

I do know the final script will look like

1: Four executions of pkgbuild. 3 of them simple, since they are for .component or .vst or .vst3 with a target folder and one tricky since it packages up the data directory to go to application support

2: One execution or productbuild. —synthesize to make those 4 sub packages into one big package

3: On execution of hdiutil to make a dmg

But the exact mechanics of that completely escaped me. @asimilon or @esaruoho if you get that working then we can knock out @kzantow recursive copy from the plugin and start the train rolling to get an alpha release. We are close to the answer in the fragments in this thread.

[asimilon]

Is 3: strictly necessary? Could just as well distribute a .zip of the single file .pkg installer bundle (like on Windows).

This is pure conjecture, but until we work out the logistics of signing, might having both unsigned DMG and .pkg force the user to have to "Allow" in Security prefs twice? Regardless of that, learning pkgbuild is something that would be very useful for me, in both my day job and side projects, so I'm very keen to have a crack at it.

[kzantow]

Just to reiterate, I have a slightly different idea of the "best" thing to do in general: include these files in the bundles and do not move them to the Application Support. This makes it MUCH easier to distribute changes to the config and wavetables without users mucking around with them. And then also have a user-area for other configuration data in Application Support e.g. custom wavetables etc..

As for having an installer, if the desire is to install all variants of the plugin, the dragging-to-folder kind is probably not as good as the typical .dmg type where you just check or uncheck each plugin, then run a script to sync the resources to Application Support as it probably does right now. I'm totally on board with removing stuff from each bundle once that installer exists. And for local development, just having an rsync or copy as part of the debug build or something would be nice so people don't have to remember to do it all the time.

[baconpaul]

Yeah you know @kzantow I think this is one of those things where "the first person with a PR which does one of these things will say how we do it!". I think the pkg is more "native" and @asimilon has extra benefit from learning pkgbuild so I think we will end up there.

Remember, I "solved" this problem by telling people to run a crappy shell script outside of Xcode. So I'm definitely not an authority! Chuckle.

[esaruoho]

i would definitely appreciate a .dmg containing .pkgs . the user is supposed to know about evading gatekeeper. :)

[baconpaul]

Attaching PR #116 to this issue. I’ll give that a try and merge if it also works for me, then close this issue. Thanks @asimilon!

[baconpaul]

Closing this issue since #116 gives us far more than a rudimentary packager - it gives us a proper pkg generating packager. Thanks @asimilon