Feature: Use download url and checksum of binaries stored in taiki-e/install-action

[NobodyXu]

We could use checksum in taiki-e/install-action to make sure the pre-built binaries are not modified.

Originally posted in taiki-e/install-action#514 (comment)

[NobodyXu]

The download url/template in taiki-e/install-action can be used to speedup package resolution.

[NobodyXu]

Just think of another solution:

We could run auto-discovery on popular-crates like quickinstall/taiki-e and cache it within repository.

This is perhaps better suited for quickinstall to do as it already have the infrastructure.

[NobodyXu]

cc @passcod @alsuren what do you think?

[passcod]

There's no point to this for binaries hosted in quickinstall as those are already signed and verified.

For others, we'd essentially have a crawler that downloads binaries, produces a checksum, and hosts that somewhere?

I think my concerns are provenance and mutability:

• provenance: we can't guarantee that the crawler downloaded the right thing. we also can't represent these checksums as being a guarantee from the original file producer, so it would need some kind of warning or careful wording.
• mutability: github release artifacts can be changed. if that happens legitimately, all binstalls for that tool version will start failing. we'd at least need to have a way to force a checksum update, in a reactive manner when breakage reports come in.

[NobodyXu]

For others, we'd essentially have a crawler that downloads binaries, produces a checksum, and hosts that somewhere?

Well, it will just contain a url and checksum for the target.

Storing binaries will make it really hard for user to verify the content.

Storing urls and checksum will be more compact, we can just put it in a branch as a json/csv to be auditable/readable.

[alsuren]

Last time I thought about this, I wrote these notes:

cargo-bins/cargo-quickinstall#56

I think that we should be comfortable thinking of quickinstall tarballs as immutable at this point. I have mostly been thinking that way from the start and only broke the rule once (can't remember why).

If the notary is handed the tarball url to checksum as soon as it is built, and the client checks the tarball against the checksum from the notary then we should be good. If we fuck something up in a minor way then we can just delete the tarball (and put a tombstone somewhere so that it will never be rebuilt) and the client can just rebuild from source. If we mess it up in a bad way then we can release a new version of the client that looks in a different place (with an epoch in the url scheme to count how many times we have brought shame upon ourselves).

[NobodyXu]

cc @alsuren @passcod I'm thinking of a json file, containing the url and optionally, the checksum.

E.g.

{
    "target": [
        {
            "url": "...",
            "checksum": "..."
        }
    ]
}

We can put it in release, or in a branch/tag.

A tag is immutable, it can be deleted and recreated but it seems that it can be disabled in github.

[passcod]

Well, it will just contain a url and checksum for the target.

Sure, but we'll still need to download stuff to compute that checksum.

[NobodyXu]

Hmmm... yeah I can't think of a good way to fix checksum, maybe we can simply omit it and only include the url?

[passcod]

That still leaves the problem of cache invalidation, though I suppose in the rare case that the URLs change we'll fallback to runtime resolution (and then to source).

I wonder if we'd better make an actual database-backed app for this express purpose, that reuses the binstall internals so we guarantee the exact same resolution algorithm, and doesn't store stuff in github releases so we're not rate limited.

shuttle.rs's free tier would likely be fine, or whatever other service https://www.shuttle.dev/pricing