Our CI and Release Assets lack source tarballs

[dvzrv]

Describe the bug
Releases on https://surge-synthesizer.github.io/ are not reflected in tags in this repository (hence it is impossible to build e.g. 1.6.3 reproducibly on any system).
Releases on https://surge-synthesizer.github.io/ only offer pre-compiled binaries (which is great maybe for macOS and Windows, but a no-go for Linux distributions as they build from source).

I would like to be able to add surge to the official repositories on Arch Linux, but having no tarball to download and build from, I can't build the "latest stable version".

Source tarballs are a means of ensuring, that the input to a package remains the same (by verifying its checksum) and can be build reproducibly. In an ideal world, this is accompanied by a digital signature (e.g. PGP) to ensure authenticity (but that's not part of this ticket).

Please let us know your surge version
1.6.3

To Reproduce
Steps to reproduce the behavior:

1. Try to download source tarball

Expected behavior
A release on https://surge-synthesizer.github.io/ is reflected in a tag in this repository and can be downloaded as a source tarball, with all submodules added, so that validity can be checked.

Desktop (please complete the following information):

• OS: Arch Linux

Additional context
Providing the means to verify the source a specific release was built from and allowing outside parties to build a specific version reproducibly ensures the transparency and allows for a model of trust in upstream (this project).

I see no upside in intransparently generating a specific version of surge from an unspecific moment in code history.

[baconpaul]

Hi

There’s well defined branches for each release which are permanent

For instance branch ‘release/1.6.1.1’ will build 1.6.1.1. I built beta7 no problem 3 weeks ago.

Those are the exact code sets used to build the releases on the website by our ci pipeline

I agree it would be bad to unttansparentky build from random source. That’s exactly why we don’t! :)

Best

[baconpaul]

Oh and here’s the arch package the Linux folks set up already

https://aur.archlinux.org/packages/surge-synthesizer/

HTH

[dvzrv]

@baconpaul thanks for getting back to me so quickly!

There’s well defined branches for each release which are permanent

For instance branch ‘release/1.6.1.1’ will build 1.6.1.1. I built beta7 no problem 3 weeks ago.

Those are the exact code sets used to build the releases on the website by our ci pipeline

How would I be able to make sure from the outside, that they are indeed unchanging?
There is no way to ensure, that HEAD on release/1.6.1.1 will point to the same commit, without checking out the branch and comparing the commit's checksum with a previously stored checksum. This introduces significant overhead on the packager's side (a version and a commit checksum has to be tracked).

Furthermore, the releases are disconnected from this repository, as it's impossible to establish a link between any release and a commit. The prebuilt release artifacts are not revertable and could have been built from any version of the code (I'm not saying you did of course) and the source tarballs are only reflecting the contents of the releases repository (which has no relation to the source code).

This all makes me wonder, why this repository was not directly used for the releases, as just using git tags would allow transparency, (incomplete) tarballs and a simple trigger to build a release from a specific commit and optionally create a source tarball with all relevant sources inside.

I agree it would be bad to unttansparentky build from random source. That’s exactly why we don’t! :)

Sorry, but I think that is actually the case, as from the outside it's impossible to easily see what's going on and/or to even build from source without building from HEAD of a branch.

Oh and here’s the arch package the Linux folks set up already

As you can see in the build script, the source is setup as an unverified git clone of the repository. Afterwards a given release branch is checked out (again without any form of verification). This is suboptimal and nothing I'm comfortable with moving to the official repositories.

I understand, that github's created source tarballs are insufficient, in that they do not include submodules. However, this can be easily achieved by creating a script, that creates a full copy (i.e. clone a specific tag (recursively), archive the whole thing and add it to releases), triggered by tagging a new version.

[baconpaul]

so Unfortunately the set of assets needed to build the entire plugin are not in the git repository and are stored as secrets in our azure pipeline. That’s one of the reasons we have the 2 repository release strategy. If you want to discuss more detail perhaps let’s chat off the repo. Grab me in the surge irc and I can explain more. This is not the case for the lv2 or vst3 version though just the vst2.

Also you are conflating the fact that you can’t tell we are building from unverified source with the fact that we are building from unverified source. Two different things. We certainly are building from verified source.

But that’s all beside the point. I’m wondering what change you are proposing? It is open source and we are happy to take changes which keep the system building nightlies and release binaries for all our oses and pull requests. @kzantow was also involved in the ci pipeline so let me tag him. But again may be easier in irc or slack.

[baconpaul]

Oh and for my edification: do tags have changes you can detect? That’s the difference between them and branches? If it makes your life easier it would be simple enough for us to tag our repo at release point I guess. Just another command in the release pipeline side in this repo. Wonder what about a branch moving is different from a a tag moving. But as a practice we don’t move a release/ branch ever after we create it. You are right to guarantee that you need to cache the hash though which is a bummer.

[baconpaul]

Just reading this again: There’s a lot of stuff in this issue, but I think what you are asking is

With every release that right now has 4 assets (mac dmg, linux deb, win exe and win32 exe) add a source tarball of the snapshot of source right before the build that makes the binary begins. And perhaps add something like a checksum of that source tarball. And have those as two new assets in the release repo.

If that’s what you actually need, that’s incredibly easy to do. I think the other commentary, while interesting, may have hidden the actual goal. So is above the actual end goal you want?

that source tarball when unpacked, in the presence of the pre-requisites, would build the headless unit tests, AU, LV2 and VST3 without trouble. It would not build the VST2 for obvious reasons. I think that’s what you would need.

Then the arch distro becomes “download well named source tarball and prerequisite; run build commands”. I don’t understand arch but if I think of it roughly like ‘the new gentoo’ it sounds like that’s what you’d need.

Have I gotten the point? If so, like I said, adding a source tarball is super easy. We don’t do it because we can just check out the source and we are sure our processes are good, but not a problem.

Thanks

[dvzrv]

@baconpaul sorry, I've been busy!

Grab me in the surge irc and I can explain more.

Where is the IRC channel located? I couldn't find any information on that.

do tags have changes you can detect? That’s the difference between them and branches?

Well, annotated tags point to a commit. Conveniently, most git hosting infrastructure/ frontends automatically create a tarball when a repository was tagged. This means anyone can easily download that exact moment in time, a project was deemed as stable (without requiring git) and verify its checksum.
In comparison: The HEAD of a branch (or really any commit on it) is a commit. No git hosting infrastructure/ frontends automatically create a tarball from it, because that would mean creating a tarball for every commit.
In general, git tags are the way to point to the release version of a repository. Branches are not, although there exist projects, that have long living minor version branches, where they backport patches to (so they can e.g. release 1.1.1, while already working on 1.2.0). However, they use tags for all of those versions.

If it makes your life easier it would be simple enough for us to tag our repo at release point I guess.

Yes, it would be great to use the tag as the start point for your pipeline process (which ends up as release assets somewhere). This is great for transparency.

With every release that right now has 4 assets (mac dmg, linux deb, win exe and win32 exe) add a source tarball of the snapshot of source right before the build that makes the binary begins. And perhaps add something like a checksum of that source tarball. And have those as two new assets in the release repo.

A tarball containing the sources required for building surge would be ideal, yes. If you can, I would also very much appreciate a detached pgp signature for the tarball in that case! However, a checksum file is technically not really necessary, but appreciated.

Aside from a release tarball, there is also the possibility of building from the git repository directly. However, this requires an annotated and signed tag of a commit (of this repository). In case it is a lightweight tag, the tagged commit needs to have a valid pgp signature (I see that your commits are signed).

It would not build the VST2 for obvious reasons.

That's completely fine.

[baconpaul]

OK thank you. Very useful.

So first our IRC and Slack info is https://github.com/surge-synthesizer/surge#references

For things like signing a tarball, what signature do folks use when they have automated build pipelines like ours? I think it would be fairly straight forward to do a source-only release in the surge-synthesizer/surge repo without submodules but the main releases are in the release repo. I can also just tag the main repo with tags along with release branches if you want. Putting a source-only release in the surge.git repo seems a bit odd though, especially since it won’t have submodules expanded anyway.

So I think the plan of attack that would help you out is (and correct me if I’m wrong)

1. Start the habit of making a tag as well as a branch with a release And since I would do that and I have signed commits set up, that’s fine.
2. In our release pipeline, tar up the source and include that as a downloadable asset with the Su modules expanded
3. Sign that with a pgp signature (best) or a cryptographic checksum (worse) in a separate file

Then once 2 and 3 are available you add an arch thingy which grabs that source tarball, checks the pgp and/or checksum, unpacks it, and runs the build commands.

If that’s right, its easy enough to do before christmas, if you can just help me know what sort of signature and key management practice people use for unattended builds. You happy with us just making a key for ‘surge-build-bot’ and signing with that for instance?

[dvzrv]

So first our IRC and Slack info is https://github.com/surge-synthesizer/surge#references

Ah, I looked at the website... ;-)

For things like signing a tarball, what signature do folks use when they have automated build pipelines like ours?

If people really (and I mean really) trust their automated build environments (and tbh, I don't even trust my own hardware), they sign there. However, this requires compromising the private key of the cryptographic keypair and storing the password for it in a hashed secret or similar. When thinking of build environments, that are not directly under one's control, that's rather scary.
The more secure way is to create the signature on the developer's machine and upload it separately.

You happy with us just making a key for ‘surge-build-bot’ and signing with that for instance?

To be honest, I would refrain from doing that, because no real security is gained from it.

Having signed tags (i.e. git tag -s <some_version> -m "<some_message>") is already a huge improvement over the status quo (as the developer - you - does this on his/her own machine)!
A detached signature for the tarball (i.e. gpg --detached-sign -o <file>.sig <file>) would be great as well, but should not be done in an automated fashion, unless you run (as in: you have root on that machine) the build machine (on which the keypair has to be stored) yourself and have security measures for all users in place, that are >= the security measures on your own machine.

Thanks for the consideration!

[baconpaul]

Thanks for helping me understand.

Well first I agree with you that signing in our pipeline doesn’t add any security. That’s why I asked! We are having the same problem deciding what to do with apple’s notarization also.

Adding the signed tags is easy enough. Right now I have the three commands I issue when I make a release. I can add a fourth. And I can go and add signed tags back on the repo also. We are using the name “release/1.6.4” as our branch so I suppose I could use the tag name “release_1.6.4” to avoid ambiguity. Totally simple to add to my workflow.

But I’ve long long held the belief that a repeatable automated build pipeline, with the risk of the unattended build, is more secure than letting developers deploy binary assets from their machine. (In this case, the source is the binary asset). But you do end up with the problem of ‘who is the authority’ when that happens.

But also as to the sig, we’ve automated everything we can here so that first there’s no single person in the loop (several of us have write perms to the repo here; and several of us have permissions to the azure account that runs the build). I think often about how the project could sustain one of the maintainers deciding to do something else for a while, so things like ‘use this Apple ID to notarize’ or ‘use this pgp key which isn’t shared to sign’ also are things I’ve tried to avoid. How do other open source projects deal with this?

I suppose I could personally sign things and squirrel away the resulting signature somewhere, but does that really improve your security? You’ve gone from a tarball on a website with a checksum but no signature to a tarball on a website with a checksum that some internet rando has said is the right one (in this case, I am that “internet rando”).

But this is an evolving space and I’m less experienced in these signing and packaging nuances, having mostly worked in environments where this was taken care of for me, so any insights you may have are appreciated.

Thanks for the information.

[baconpaul]

OK so I get it. I set up gpg and pushed a signed tag to the main repo just now and I see it tries to create a source tarball. Fine. No submodules though, as you say.

Still debating what to do about the release source tarball with submodules expanded from our build pipeline and how to sign it. I need to figure out how to get the signature object web available with a well defined name for you.

[baconpaul]

OK here’s what I’ve done.

1. Added to our release checklist to add a signed tag of form release_id matching the branch release/id in the surge.git repo
2. Added to our release pipeline in the release repo a step to checkout the code and make a tarball.
3. Added that tarball to the release assets
4. Add a SHA256 of all the release assets from the build pipeline to the release assets

Right now this has only happened on our ‘nightly’ release. https://github.com/surge-synthesizer/releases/releases/tag/NIGHTLY - that release is definitely not something you want to put in the arch repo. Amongst other things it can change multiple times a day. But the release/ releases will also have the same asset layout, just with the appropriate code.

So would that setup let you do what you need? If so I could redo the 1.6.4 release build and then load the assets there (or maybe wait until a 1.6.4.1 release which I hope will fix our Catalina problem).

If signing the asset is really a requirement I’m not quite sure what to do. I suppose “download it, generate a signature, and upload the signature” is what one would have to do. I’m not sure exactly how much that adds though?

Your thoughts welcome.

[baconpaul]

Hi! Just an FYI - I have identified the problem in surge and macOS catalina and will probably do a 1.6.4.1 release with that fix and a couple of bits of modified content from a 3rd party contributor next Monday. That will definitely be our last official release of calendar year 2019.

The release will follow the checklist above - signed tag, tarball in the repo. But I hadn't planned to do anything about signing the tarball unless you happen to be able to answer my question below.

So no pressure - we're solving your problem here so I'm not as rushed as you are! :) But if you have things you'd like to change for a release so you can get to work on whatever your next step is, we have a few days to figure it out before we release.

[dvzrv]

So would that setup let you do what you need? If so I could redo the 1.6.4 release build and then load
the assets there (or maybe wait until a 1.6.4.1 release which I hope will fix our Catalina problem).

Either the tarball or using the signed tag will work for me. Thank you!
Signing the tarball is not a requirement, but obviously a nice-to-have. I don't see a sane way to do it automatically though. :-/
The best way would be to have a well-defined set of people (the members of the developer team of this project, that have access to the releases and assets) be able to use their PGP keypairs to sign a specific source tarball and upload the signature as an asset.
However, this would be a manual step after the pipeline build and pushed the assets (and therefore would have to be added to a release checklist)!

That aside: Good luck with the Catalina release workflow. Apple's notarization service is very painful and anti-CI. Been there...

[baconpaul]

Ok cool so 1.6.4.1 will ship this weekend with both a signed tag and a tarball so then you can do what you want! I will leave the pgp signature for now but keep it in the back of my mind.

When I’ve shipped 1641 with tarball and signed tag I will close this issue. If you find problems then please either reopen it or open a new one

And yeah all this stuff is hard...

Best!

[baconpaul]

OK we released 1.6.4.1 this morning

You can find our signed tag in this repo at https://github.com/surge-synthesizer/surge/tags
If you pull that tag you will need to do a git submodule etc...

You can find the source tarball completely expanded in our release repo
https://github.com/surge-synthesizer/releases/releases/tag/1.6.4.1

So I think that gives you what you need! I"m going to close this issue now. Please feel free to either reopen it or open a new one if you run into further issues.

Thanks

[dvzrv]

Oh, one more thing, before I forget: Please make sure to also upload your gpg public key, so it can be downloaded for verification: gpg --send-keys 1B1F4A9F98AE0403

Might take a while for the key to be propagated.

[baconpaul]

paul:~/dev/VCVRack/plugin-source/tinytricks$ gpg --send-keys 1B1F4A9F98AE0403
gpg: sending key 1B1F4A9F98AE0403 to hkps://hkps.pool.sks-keyservers.net
gpg: keyserver send failed: No route to host

Any ideas?

[dvzrv]

If you're on a Linux machine, maybe kill your dirmngr (service/executable) and make sure, that you have a proper DNS resolution of the domain in question.
I used to have a problem with some version of gpg a few years back and (for bizarre reasons) it was related to having more than one DNS server in /etc/resolve.conf.

In general you can also use another server by using the flag --key-server <domain>.
Orr... if all fails, just force push that thing to all of them! :D

[baconpaul]

paul:~/dev/VCVRack/plugin-source/tinytricks$ gpg --keyserver hkp://pgp.mit.edu --send-keys 1B1F4A9F98AE0403
gpg: sending key 1B1F4A9F98AE0403 to hkp://pgp.mit.edu

OK I did that and it worked. Does that let you do what you need?