FreeBSD Package?

[davidchisnall]

This looks like exactly what I want for off-site backups, but I really don't want my restore procedure to depend on compiling something from source or my backup procedure to rely on remembering to manually update something to make sure that it has the latest security fixes. What are the blockers for having this in the ports tree?

[someone1]

No blockers - just never considered it! I'll look into as I've never done it myself. Or if you're familiar with the process and would like to maintain this on the port tree, please PM me and we can discuss!

Thank you.

[davidchisnall]

This turns out to be nontrivial because of the external dependencies that are not on GitHub. FreeBSD ports need to be able to fetch dependencies without using git. For the GitHub dependencies, it's easy to instruct GitHub to generate a tarball for a specific revision, but for the code.google.com and golang.org dependencies I don't know how to download them.

[someone1]

I do provide pre-compiled FreeBSD packages, not sure if that makes it easier? I'm not sure its required to compile from source as this would require go as a dependency along with several version control systems to grab the entire source tree as you mentioned. The pre-compiled binary has no external dependencies. I don't see why you couldn't just submit a port with just that and update as needed.

[davidchisnall]

FreeBSD packages must be built by the the FreeBSD package build systems (and must include specific versions of the dependencies - we don't want to find that a rebuild fails because an upstream library has changed an API), for each of the supported architectures. They must also be buildable with custom make.conf options by downstream users. Go and Node.js are particularly problematic for packaging, because of their tendency to have unversioned dependencies.

It would help if the GitHub release tarballs contained all of the dependencies - I don't know how easy that is.

[someone1]

Ahh I see - I took a look at other go packages distributed via ports (example) and found they just ship with a pre-compiled binary. This seems like the logical choice to me but then again I'm not familiar with how to ship packages via ports.

I have CI on this repository that runs on every commit + every 30 days to catch for any issues with dependency changes. I could vendor all my dependencies so that they all live within this repository - this way they would ship with any tarball from github and we would never have to worry about dependency changes (but then I'd have to update more often to follow any changes to dependencies). I can consider this but don't think I'll have a decision in the short-term. I'll want to investigate my options a little more.

[davidchisnall]

Ahh I see - I took a look at other go packages distributed via ports (example) and found they just ship with a pre-compiled binary

Not sure why you'd think that. The port you link to grabs the source code via GitHub's tarball generator (it grabs https://codeload.github.com/prometheus/node_exporter/tar.gz/v0.15.0?dummy=/prometheus-node_exporter-v0.15.0_GH0.tar.gz, which means that GitHub will generate a tarball containing the contents of the node_exporter repository at the v0.15.0 tag and provide it with the prometheus-node_exporter-v0.15.0_GH0.tar.gz filename). The ports system then builds this.

For dependencies hosted on GitHub, it's easy to make the ports system grab the dependency in the same way, via GitHub's exporter. For example: mailhog grabs a load of dependencies like this. It's not ideal, because the port maintainer has to update the port every time the dependencies are updated (this one has a bunch of random commits for the dependencies, ideally they'd all be the latest release versions). It's more difficult for the dependencies that you have on places like Google Code, where there doesn't appear to be a way of downloading the code.

[someone1]

I see, my mistake then. Yes, this looks to be pretty tedious, again maybe vendoring my dependencies would help but there would still be the issue of updating frequently when dependencies update.

[davidchisnall]

Vendoring dependencies is also a bit painful. If you could push a tarball of the versions of the dependencies used to build the releases when you upload the release builds, that would probably be the easiest solution - presumably you're downloading them when you build anyway, so just tarring up the result and sticking it on the downloads page would give something easy to point the port at. Oh, and it would also help reproduceable builds for anyone else who cares about them.

[someone1]

@davidchisnall - I'm prepping a new release, are you still interested in helping with this issue?

I hope to be more responsive to updates going forward, but I don't expect there to be many releases beyond a couple a year at most.

I've added support for Go modules - does that help with anything you described above or should I vendor the dependencies and upload a tarball as you previously requested?

[davidchisnall]

Hi, it looks as if there's been some work in the generic ports infrastructure to help package go programs. I think that will make it (relatively?) easy to handle any dependencies that are on GitHub. I can hopefully try at the weekend and see if it works.

[someone1]

Please let me know if I can help in any way and thank you for taking the lead on this!

[davidchisnall]

I have a working package built from the current master as 0.4.p1. I'd prefer to submit it from a release or pre-release tag. Would it be possible to tag the current master as a pre-release? Then I can easily bump it to the real release by adjusting the tag name, rather than making up version numbers and tracking individual git commits.

[davidchisnall]

For reference, this is the draft port: https://gist.github.com/davidchisnall/d95ec8a2e07ed1f4648a6e937ef11193

[someone1]

that's great! I'm drafting a release, I believe some things surrounding encryption/signing/keyrings is a little off right now so as soon as I get proper tests in place and fix any bugs I'll put out a release - hopefully in another day or so.

[someone1]

Apologies on the delay - struggles with testing zfs on travis was very frustrating/discouraging.

New pre-release published,

[sww1235]

@davidchisnall any updates on this? If you need some help with the ports effort, let me know.

[davidchisnall]

I think I just need the release to be tagged. @someone1?

[someone1]

It should have been from my last comment, is there something else I need to do here?

https://github.com/someone1/zfsbackup-go/releases/tag/v0.4-rc1

[davidchisnall]

That still shows as a prerelease for me. Is there a release? Once there's one that you're happy with, I can update and submit the port.

[someone1]

@davidchisnall you had mentioned to tag a pre-release, will that no longer work?

The latest pre-release has been in production since May with no issues but let me review before I make it a full release.