Add Launcher app with managing updates

[chimp1984]

Following the idea layed out in #184 I try here to spec out a bit more the details. It will likely will require more work to figure out all details, but this should serve as starting point.

Basic concept

We have per application (desktopapp, satoshisquareapp) a binary (using java packager) which gets signed for OSX and Windows binaries by the release manager. Those apps are installed once and then download jar files and other metadata when an update is available and at restart load the new jar files. That launcher app should be as minimal as possible as it cannot get updated itself by that mechanism but would require a release of a new binary (as we do in Bisq 1 at each release).
The launcher application containst the jre, so when we would update the jre version it would require a new installation. It is not intended to support update of the jre by that updater framework.

How to download and detect new versions?

There are multipe ways how we could do that. Here are 3 variations:

• Deploy it to the Github release location and load the metadata and jars (signatures, version info) from there.
  • Downside: Dependency of a centralize, Microsoft owend infrastructure. Download likely only possible over clearnet (I guess Tor would require captchas so is not possible non-interactively).
  • Benefit: High performance infrastructure for large files (CDN). Can be assumed that they have very professional opsec so the download locations can be considered safe.
• Deploy it to a Bisq controlled server and support download via Tor.
  • Downside: Higher security risk and costs for traffic. Controlling the server adds a centalized and trusted role. Costs. Dos risks.
  • Benefit: Tor/I2P support possible. No dependency on $MS
• Deploy all via the Bisq P2P network
  • Downside: Downloading exclusively via Tor/I2P can be slower and more burden for the P2P network/nodes which provide the data. Might be a bit more work to implement. Requires authorized roles for deploying that data (same as above). More effort in efficient jar structures and more complex modular versioning system.
  • Benefit: Tor/I2P support. No dependency on $MS. No extra costs as it can be handled by existing nodes (seed nodes might add this feature). Protection against dos via P2P networks pow protection. More decentralized and censorship resistant.

Data size of jars from current Bisq 2 app

All jars: 77MB (jar)
External lib jars: 58 MB (from extracted fat jar)
Bisq jars: 30 MB (from extracted fat jar)

The largest jar from the bisq side is desktop.jar with 13 MB. The major part there are resource files (images). That could be extracted to a new module which changes very seldom, so download can be reduced to a one time download.

With a modular versioning strategy I think we can avoid downloading lot of data. It will be mainly the size of the class files if we manage to extract resources to modules which mostly do not change. I guess maybe 1-4 MB in compressed form should be feasible. Bisq 1 is currently downloading about 1-2 MB of data at startup. So I guess it should be feasible for the P2P network to handle that. It also can be downloaded in a low-priority mode and depending on granularity of provided data download could be distributed over various nodes.

I think the solution using the P2P network is the best, so the following is assuming we choose that option.

Verification

All files need to be verified using a signature matching one of the hard coded pub keys (in the launcher app). There can be multiple Bisq contributors signing deployed jars. The user can define how many signatures are required to be considered safe. By default we should use at least 2. Depending on the available number of signature the user could increase that requirement to 3 or more (assuming there are 3 or more signatures deployed). The min. number of signatures will be a requirement for deployment. This would be an improvement over current Bisq 1 where we only expect 1 signature (from out of 3 devs who have hard coded the pubkeys in the app).
As in Bisq 1 the initial download of the launcher app must be manually/externally verified and we cannot ensure that the downloaded signature and pubkeys are not manipulated by themself. We could use PGP as additional trust source for that part.

Process

The launcher app will start up a Java process and checks if there is a version metadata file and jar files in the user directory.
If not (at first start) it will download from a hard coded URL (can be passed as program argument as well) the versions meta data file. Using that version file it will downloads the jars of the most recent version from Github or a Bisq server.

We cannot download it via the P2P network/Tor at that stage as no network lib is shipped in the launcher. To ship it would introduce compatibility restrictions for the network module. We could though add support that users do download manually a fat jar from Github using Tor and extract that to the data directory.

Once the version meta data file and the jars are available (at next start of after download at first start) it load the jar files and start up the application.

Deployment of new versions

When a new version is deployed, the required signatures need to be deployed. Each jar file comes with its dedicated signature file (containing the pub key).
The deployment via the P2P network is done by the signatories. We use the authorized data storage feature which verifies the deployed data against hard coded pub keys (actually maybe we need a new specialized data store for that as we do not want to keep those update data containing the jars in memory).
Once a user receives an update-message containing the new version info they will wait until all defined data of the update (e.g. list of jars) are received from the network. After that it will move those jars to the data directory update folder and appends the new version to the version file. It will request the user to restart the app. After restart it will read the last entry from the versions file to find out the latest version and point to the directory with that version to load the jars. After successfully loaded it will delete the old jars.
The user can disable automatic update, but by default its turned on. If disabled the user still see the info that there is a new version available but it is not loaded automatically (pinned version mode).

Meta data

The version metadata file could be a simple map with the version number as key and a listof its containing jar files as value.
E.g.
map.put("v2.41", List.of(common-2.11.jar, network-2.20.jar, offer-2.01.jar,...));

The jar files can have different versions, only if the user does not has the required version it need to download it.
This will require a different versioning system as we use now where all jars would result in the same version.

Data structure

In the data directory (user space) we put the version file on the top level.
There is a jars directory which contains sub directories with the jar name and inside that another sub directory with the version number as directory name. Inside those are all jars with that version.
It is a similar structure like the .m2 repository directory for resolving java dependencies.

When the app launcher loads the jars it will look up different sub dirs for the relevant jar.

Open questions

• How do we apply the fine grained modular versioning system?
• How do we handle the downloaded jar files in the P2P network data storage? We do not want to keep that data in memory.
• Does anyone see security concerns of that approach?

Discussion

• Why not use gradle/maven?
  The whole process is very similar to downloading java dependencies by gradle, maven, etc. Can we utilize such existing infrastructure while supporting our higher requirements regarding privacy and security and not introducing undesired large dependencies? I doubt, but maybe someone has convincing arguments.

• Should we pack the initial jar files into the binary as resources which will get extracted to the data directory at first start? It would allow users to start up initially without that initial clearnet download. Drawback is that after longer time most of those jars (though likely not the external ones) have changed anyway and the user need to download the new jars, having downloaded first the outdated jars with the binary and then again the new jars. Also in case of larger changes the packaged network module might be not compatible anymore, if we go that route we would add at least more restrictions with compatibility.

• Should we optimize download by using delta packages to only download the changes?
  https://github.com/vinumeris/updatefx was using such an approach. I am not sure if the added complexity and risk is worth it. But maybe could be investigated later for future improvements. But in general as this is a very security critical area, I would prefer a more simple approach. Complexity it the enemy of security...

• We could support to add authorized pubkeys. This would make it more flexible in case contributors retire, but it adds likely more risks. As that part needs to be supported in the launcher app we should think about it carefully if we want to support that. To ban one of the pubkeys can be done via a similar mechanism like in the filter in Bisq 1. Users could ignore such filters via program arguments so the power of it is limited and protects against potential abuse.

• Maybe it is technically possible to do a hot-swap loading and replacing the java classes at runtime without a restart. But I guess that is much more risky and could easily lead to weird bugs. Also might add some security risks.

• Security-wise a major difference is that we load the jars from the user space (user data dir) not from the inside the application (on OSX the app file is a zip containing the jars) which has more restricted permission rights and comes with internal integrity checks (and signature checks in case of OSX/Windows). As we verify all jars before loading them, I do not really see what damage could be done, even if an attacker manage to insert malicious data into the data directory. As long the launcher app is not compromised and as long the signatories are not compromised I don't see a way how such an attack could succeed. But maybe @alvasw has some inputs here?

• Deleting old jar files would be probably a good feature.

[alkum]

Sounds ambitious, yet doable.

However I'm cautiously optimistic because of all the extra complexity.

If we decide to go ahead and do it, I'd suggest to first try a small-scale MVP (e.g. update Bisq 2 jars from GH repo, test launcher behavior on all 3 major platforms w.r.t. signing / evtl. OS asking of extra permissions or confirmations on jar update).

I would personally prefer a simpler approach, at the risk of being less user-friendly: why not provide Bisq as a single executable jar with a simple launch script

• Upgrading would mean downloading the new jar + verify sigs (can be done manually, automatically).
• The launch script could automatically choose the latest version jar available in that path (thus making reverts easier: simply close Bisq, delete new jar if there is an issue, relaunch and older version is loaded)
• Bisq can handle versioning for user data folder (backup before upgrade, etc)

Downside is, there would be no installer, no start menu shortcuts, no real "application".

But on the upside, its low effort to implement and it achieves a similar scope as above proposal.

So to summarize

• generally optimistic about the proposal (sounds complex, but doable)
• would suggest to do a small-scale end-to-end test before investing too much time into it (unclear how different OSes would handle this update operation, even with signed binaries)
• would prefer a simpler, launch script + one big jar approach

[chimp1984]

It is intended for Bisq 2 Satoshi Square app, which carries little risk as it has no wallet.

Your suggestion requires that users install java and as this get more painful with every new java release I think for a userfriendly solution we need a binary similar as with current Bisq.

A more simple version would be to download the fat jar as one single file with each update but that would not work well when using Tor/I2P as 70-80 MB is quite a bit for those networks. Over clearnet it would be acceptable (not using the P2P network but the server download option).

I think the 2 main issues are the modular version and to check out a good way how to deal with the jar files at the P2P network data storage level.

But I agree its more complex as initially thought and not a big fan of complexity either... Anyway not a must have for initial release but an option we could consider. The extra work would pay off over time as release manager has less work with deployement build of binaries.

[ripcurlx]

I'm pro getting rid of the shipping and signing of binaries for each release as it is a fragile setup. As soon as Apple and/or Microsoft is put under pressure and decides that in certain jurisdictions Bisq is not within the legal framework, my licenses will be revoked and I won't be able to sign any binaries anymore, which makes the installer UX nearly unusable for users.
To get more resilience for the initial app we could think about publishing it to more places than just to our website by using a tool like https://jreleaser.org/.

[chimp1984]

https://jreleaser.org

Ah thats the guy Chris has worked with a while back...
Maybe we should get in touch with him?

[aalmiray]

Howdy-ho 👋 I'm the author of JReleaser. Last year @cbeams requested a handful of features to support Bisq's release process. We've added plenty of things to the tool since then. Happy to chat with the team to figure out what may be done to streamline your release process.

[chimp1984]

Hi @aalmiray , thanks for joining! I will read up a bit more about JRealeaser and check out what was the work you have done with Chris, then get back to you and would be great to have a call then.

[zlatinb]

Hi,

In my application I use JPackage to create images for Windows, Mac and Linux. Specifically for Mac I sign the interim image and then invoke JPackage again to build the final .DMG which passes notarization.

The update on mac is possible with a little creative usage of DiskUtility. On Windows I use NSIS in silent mode. All the code and documentation is here

@aalmiray hey man good to see you I love Griffon!

[HenrikJannsen]

It is partly implemented via:
#1028
#1029
#1030

See #1032 for follow up issue