Create rmfakecloud-proxy package #448

matteodelabre · 2021-09-29T17:48:14Z

This PR provides a new package intended to ease the setup of rmfakecloud on a device. rmfakecloud is an alternative Xochitl sync server developed by ddvk that anyone can self-host. With this package, connecting to an rmfakecloud server only takes a few steps:

opkg install rmfakecloud-proxy
rmfakecloudctl set-upstream https://... (define the server to use)
rmfakecloudctl enable

One can then run rmfakecloudctl disable (or uninstall the package) to revert back to the default reMarkable servers.

Technical details

This package reuses most of its logic from the rmfakecloud-proxy installer https://github.com/ddvk/rmfakecloud-proxy/blob/master/scripts/installer.sh, but makes it more integrated with Toltec. The following steps happen when enabling rmfakecloud-proxy:

A local certificate authority and a self-signed certificate are generated for the reMarkable cloud domain names (stored in /opt/var/rmfakecloud-proxy).
The rmfakecloud-proxy service is enabled. This is a proxy server listening on 127.0.42.10 which forwards any requests it receives to the server selected by running rmfakecloudctl set-upstream.
Entries are written to the /etc/hosts file to direct requests to the reMarkable cloud to the rmfakecloud-proxy server.

Additionally, if one is already connected to the reMarkable cloud in Xochitl when enabling rmfakecloud-proxy (or connected to rmfakecloud when disabling it), the script takes care of disconnecting them and marking their files as unsynced. This prevents Xochitl from believing that the files have disappeared from the server and therefore deleting them.

The “enabled” state survives package upgrades and system upgrades. This state is stored as a file in /opt/etc/rmfakecloud-proxy/enabled which exists if and only if the proxy is enabled.

Test plan

Please make a backup of your files (in ~/.local/share/remarkable/xochitl) before testing this.

To test this PR, you’ll need access to an rmfakecloud instance. I can give temporary access to my instance if needed.

The following scenarios should be tested:

Installing and enabling rmfakecloud-proxy while previously being connected to the reMarkable cloud: you should get disconnected from the rM cloud, no files should be lost, and they should sync to the new server after connecting to your account.
Installing and enabling rmfakecloud-proxy while not being connected to the rM cloud: your files should sync to the new server after connecting to your account.
After enabling rmfakecloud-proxy, switching to a different rmfakecloud server with rmfakecloud set-upstream: you should get disconnected from the previous server and no files should be lost.
Upgrading the system while being connected to rmfakecloud: the rmfakecloud connection should be re-enabled after running toltecctl reenable.
Running rmfakecloudctl disable should remove entries from the /etc/hosts file, uninstall the self-signing CA and disconnect the user. No files should be lost.
Removing the rmfakecloud-proxy package should disable rmfakecloud as before.

This PR provides a new package intended to ease the setup of rmfakecloud on a device. [rmfakecloud](https://github.com/ddvk/rmfakecloud) is an alternative Xochitl sync server developed by ddvk that anyone can self-host. With this package, connecting to an rmfakecloud server only takes a few steps: * `opkg install rmfakecloud-proxy` * `rmfakecloudctl set-upstream https://...` (define the server to use) * `rmfakecloudctl enable` One can then run `rmfakecloudctl disable` (or uninstall the package) to revert back to the default reMarkable servers. Technical details ----------------- This package reuses most of its logic from the rmfakecloud-proxy installer <https://github.com/ddvk/rmfakecloud-proxy/blob/master/scripts/installer.sh>, but makes it more integrated with Toltec. The following steps happen when enabling rmfakecloud-proxy: * A local certificate authority and a self-signed certificate are generated for the reMarkable cloud domain names (stored in `/opt/var/rmfakecloud-proxy`). * The `rmfakecloud-proxy` service is enabled. This is a proxy server listening on `127.0.42.10` which forwards any requests it receives to the server selected by running `rmfakecloudctl set-upstream`. * Entries are written to the `/etc/hosts` file to direct requests to the reMarkable cloud to the `rmfakecloud-proxy` server. Additionally, if one is already connected to the reMarkable cloud in Xochitl when enabling rmfakecloud-proxy (or connected to rmfakecloud when disabling it), the script takes care of disconnecting them and marking their files as unsynced. This prevents Xochitl from believing that the files have disappeared from the server and therefore deleting them. The “enabled” state survives package upgrades and system upgrades. This state is stored as a file in `/opt/etc/rmfakecloud-proxy/enabled` which exists if and only if the proxy is enabled. Test plan --------- Please make a backup of your files (in `~/.local/share/remarkable/xochitl`) before testing this. To test this PR, you’ll need access to an rmfakecloud instance. I can give temporary access to my instance if needed. The following scenarios should be tested: * Installing and enabling rmfakecloud-proxy while previously being connected to the reMarkable cloud: no files should be lost and they should sync to the server. * Installing and enabling rmfakecloud-proxy while not being connected to the rM cloud: same result as before. * After enabling rmfakecloud-proxy, switching to a different rmfakecloud server with `rmfakecloud set-upstream`: no files should be lost. * Upgrading the system while being connected to rmfakecloud: the rmfakecloud connection should be re-enabled after running `toltecctl reenable`. * Running `rmfakecloudctl disable` should remove entries from the `/etc/hosts` file, uninstall the self-signing CA and disconnect the user. No files should be lost. * Removing the rmfakecloud-proxy package should disable rmfakecloud.

matteodelabre · 2021-09-29T17:51:21Z

I’ve been testing this for a few days on my rM2 and haven’t had any issues. Testers wanted! (Please see the test plan above.)

matteodelabre · 2021-09-29T17:51:35Z

Any thoughts on this @ddvk?

ddvk · 2021-09-29T18:19:41Z

nicely done 👍

ddvk · 2021-09-29T18:22:52Z

btw, there is a -c option now that accepts a yml config file

package/rmfakecloud-proxy/rmfakecloud-proxy.service

package/rmfakecloud-proxy/rmfakecloudctl

Eeems · 2021-09-29T19:37:27Z

package/rmfakecloud-proxy/rmfakecloudctl

+            if is-enabled; then
+                echo "rmfakecloud-proxy is already enabled."
+            else
+                disconnect-cloud


Should we trap various signals here that would cause an exit and disable?

You mean trapping script errors? Or do you have other cases in mind?

Script errors, also ctrl-c or another interrupt signal.

matteodelabre · 2021-10-01T17:05:12Z

I'll have to investigate further but the preremove script of this package fails when invoked as part of toltecctl uninstall.

Eeems · 2021-10-05T17:04:40Z

I'll have to investigate further but the preremove script of this package fails when invoked as part of toltecctl uninstall.

Should this be changed to a draft until you resolve this?

matteodelabre · 2021-10-05T23:30:25Z

The toltecctl uninstall issue is due to the script relying on pgrep being available (as specified in installdepends) while it seems that the uninstall command does not honor dependencies and removes pgrep before this package. As a workaround, I’ve reimplemented a very basic pgrep inside the Bash script, but we may want to look into making the uninstall command properly manage dependencies in a separate issue.

`toltecctl uninstall` will now try to uninstall packages in the reverse order of their dependencies, so that a package can use binaries it depends on during its removal scripts (see #448). Packages in a dependency cycle are removed in an unspecified order. Test plan: Upgrade `toltec-bootstrap`, install some packages, then run `toltecctl uninstall`. You should see packages removed in reverse dependency order, so for example: * `libc`, `libpthread`, `libgcc` should generally be the last removed packages * graphical apps should get removed before `display` * `toltec-bootstrap` should get removed before `coreutils-tsort`

matteodelabre · 2021-10-06T10:01:00Z

I added back the dependency on pgrep. Provided that the updated uninstall script from #456 is used, this package now uninstalls without errors during toltecctl uninstall.

Eeems

Add traps for ctrl-c/other interrupt signals to avoid leaving things in a bad state.

`toltecctl uninstall` will now try to uninstall packages in the reverse order of their dependencies, so that a package can use binaries it depends on during its removal scripts (see #448). Packages in a dependency cycle are removed in an unspecified order.

* toltecctl: Honour dependencies when uninstalling (#456) `toltecctl uninstall` will now try to uninstall packages in the reverse order of their dependencies, so that a package can use binaries it depends on during its removal scripts (see #448). Packages in a dependency cycle are removed in an unspecified order. * [rmkit] upgrade remux with rM1 support on 2.9 (#450) this brings remux up to date with rmkit-dev/rmkit@8254893 major improvement is proper support for touch gestures in remux on rM1 after rebooting. the main issue was the display was resizing after remux started, so the touch gestures were using the wrong display size. holding the center button on rM1 would cause remux to restart (and fix this problem), so there is a workaround available. * Update KOReader to 2021.10 (#467) Co-authored-by: raisjn <[email protected]> * Add cache for dependencies install (#473) * Add cache for dependencies install Co-authored-by: Mattéo Delabre <[email protected]> Co-authored-by: raisjn <[email protected]> Co-authored-by: Mattéo Delabre <[email protected]>

matteodelabre · 2021-11-27T00:43:34Z

Added an error handler to the rmfakecloudctl enable command that cleans up the install if it gets interrupted or encounters an error. To test, run the command and do Ctrl-C at any time during the install, or insert a failing command somewhere in the install() function of /opt/bin/rmfakecloudctl. Afterwards, check that the following has been cleaned up:

/usr/local/share/ca-certificates should be empty
/opt/var/rmfakecloud-proxy should be empty
/opt/etc/rmfakecloud-proxy should be empty
/etc/hosts should be free of cloud override lines
/etc/ssl/certs should be free of symlinks to /usr/local/share/ca-certificates/rmfakecloud-ca.crt

(cc @Eeems)

This PR adds a detailed step-by-step guide to setting up rmfakecloud-proxy on a device (see toltec-dev/toltec#448). It is to be merged when the package reaches the stable branch.

matteodelabre · 2021-11-27T23:39:33Z

I propose to add a guide to our website for how to use this package to connect to alternative sync servers. See toltec-dev/web#8.

Eeems · 2021-12-05T02:51:23Z

Just tried to run through the first test and it looks like rmfakecloud-proxy.service fails to start after running rmfakecloudctl enable and there is no feedback to the user that it failed.

Steps that I took:

Installed package
rmfakecloudctl set-upstream ...
rmfakecloudctl enable
ctrl-c while it's running
rmfakecloudctl enable

Looks like the set-upstream configuration was wiped before enable was called.

After a disable, set-upstream, enable it worked. I was able to connect after generating a code, and all my documents started uploading!

rmfakecloudctl should also have a status method to check on the current upstream, if it's enabled, and if it's running.

Eeems

While I think the issue I found should be resolved, and rmfakecloudctl should be enhanced, I think it's worth getting this into testing first.

Eeems · 2021-12-07T14:46:27Z

@matteodelabre poke?

matteodelabre · 2021-12-07T15:06:35Z

Thanks for the review @Eeems! As you proposed I’ll go ahead and merge this, and will make a follow-up PR to address your points.

This PR provides a new package intended to ease the setup of rmfakecloud on a device. [rmfakecloud](https://github.com/ddvk/rmfakecloud) is an alternative Xochitl sync server developed by ddvk that anyone can self-host. With this package, connecting to an rmfakecloud server only takes a few steps: * `opkg install rmfakecloud-proxy` * `rmfakecloudctl set-upstream https://...` (define the server to use) * `rmfakecloudctl enable` One can then run `rmfakecloudctl disable` (or uninstall the package) to revert back to the default reMarkable servers. Co-authored-by: Nathaniel van Diepen <[email protected]>

* Update xochitl.oxide to support templatectl (#505) * Update xochitl.oxide to support templatectl Co-authored-by: Linus K. <[email protected]> Co-authored-by: Linus K <[email protected]> * Create rmfakecloud-proxy package (#448) This PR provides a new package intended to ease the setup of rmfakecloud on a device. [rmfakecloud](https://github.com/ddvk/rmfakecloud) is an alternative Xochitl sync server developed by ddvk that anyone can self-host. With this package, connecting to an rmfakecloud server only takes a few steps: * `opkg install rmfakecloud-proxy` * `rmfakecloudctl set-upstream https://...` (define the server to use) * `rmfakecloudctl enable` One can then run `rmfakecloudctl disable` (or uninstall the package) to revert back to the default reMarkable servers. Co-authored-by: Nathaniel van Diepen <[email protected]> * Remove rmfakecloud-proxy Co-authored-by: Linus K. <[email protected]> Co-authored-by: Mattéo Delabre <[email protected]> Co-authored-by: Mattéo Delabre <[email protected]>

This PR provides a new package intended to ease the setup of rmfakecloud on a device. [rmfakecloud](https://github.com/ddvk/rmfakecloud) is an alternative Xochitl sync server developed by ddvk that anyone can self-host. With this package, connecting to an rmfakecloud server only takes a few steps: * `opkg install rmfakecloud-proxy` * `rmfakecloudctl set-upstream https://...` (define the server to use) * `rmfakecloudctl enable` One can then run `rmfakecloudctl disable` (or uninstall the package) to revert back to the default reMarkable servers. Co-authored-by: Nathaniel van Diepen <[email protected]>

matteodelabre added the packages Add or improve packages of the repository label Sep 29, 2021

Fix formatting

2ffa84e

Eeems reviewed Sep 29, 2021

View reviewed changes

package/rmfakecloud-proxy/rmfakecloud-proxy.service Outdated Show resolved Hide resolved

Eeems reviewed Sep 29, 2021

View reviewed changes

package/rmfakecloud-proxy/rmfakecloudctl Outdated Show resolved Hide resolved

Eeems reviewed Sep 29, 2021

View reviewed changes

package/rmfakecloud-proxy/rmfakecloudctl Show resolved Hide resolved

Eeems reviewed Sep 29, 2021

View reviewed changes

matteodelabre added 5 commits September 30, 2021 11:05

Add more dependencies to the service

7fbea68

Fix unbound variable in rmfakecloudctl

64541de

set-upstream: Show enable prompt in all cases if disabled

0f1d68a

Replace wrapper with yaml config file

3cb7cf0

Remove leftover checksum entry

b6b68a5

Merge branch 'testing' into package/rmfakecloud-proxy/create

21b10ab

matteodelabre marked this pull request as draft October 5, 2021 17:26

matteodelabre added 2 commits October 6, 2021 01:19

Remove pgrep dependency & properly wait for process termination

0fe270c

Force using /usr/bin/xargs

b656ee4

matteodelabre mentioned this pull request Oct 6, 2021

toltecctl: Honour dependencies when uninstalling #456

Merged

Add back dependency on pgrep + add findutils dep

4096e93

matteodelabre marked this pull request as ready for review October 6, 2021 10:01

Merge branch 'testing' into package/rmfakecloud-proxy/create

5c43b0b

Eeems requested changes Oct 12, 2021

View reviewed changes

Merge branch 'testing' into package/rmfakecloud-proxy/create

4dc0c97

Merge branch 'testing' into package/rmfakecloud-proxy/create

63badc6

Eeems and others added 3 commits November 22, 2021 09:23

Merge branch 'testing' into package/rmfakecloud-proxy/create

76f7686

Merge branch 'testing' into package/rmfakecloud-proxy/create

d8ac4ba

Rollback failed or interrupted installs

be4ea86

matteodelabre mentioned this pull request Nov 27, 2021

Add guide for using the rmfakecloud-proxy package toltec-dev/web#8

Open

Merge branch 'testing' into package/rmfakecloud-proxy/create

1a475f5

Merge branch 'testing' into package/rmfakecloud-proxy/create

26f481b

Eeems self-requested a review December 5, 2021 02:51

Eeems approved these changes Dec 5, 2021

View reviewed changes

matteodelabre merged commit 6705250 into testing Dec 7, 2021

matteodelabre deleted the package/rmfakecloud-proxy/create branch December 7, 2021 15:07

Eeems mentioned this pull request Dec 9, 2021

Documentation is unclear ddvk/rmfakecloud#113

Closed

This was referenced Dec 15, 2021

Stable merge for week 50 of 2021 #512

Merged

rmfakecloud-proxy:  Add status subcommand #513

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create rmfakecloud-proxy package #448

Create rmfakecloud-proxy package #448

matteodelabre commented Sep 29, 2021 •

edited by Eeems

Loading

matteodelabre commented Sep 29, 2021

matteodelabre commented Sep 29, 2021

ddvk commented Sep 29, 2021

ddvk commented Sep 29, 2021

Eeems Sep 29, 2021

matteodelabre Sep 30, 2021

Eeems Oct 1, 2021

matteodelabre commented Oct 1, 2021

Eeems commented Oct 5, 2021

matteodelabre commented Oct 5, 2021

matteodelabre commented Oct 6, 2021

Eeems left a comment

matteodelabre commented Nov 27, 2021

matteodelabre commented Nov 27, 2021

Eeems commented Dec 5, 2021

Eeems left a comment

Eeems commented Dec 7, 2021

matteodelabre commented Dec 7, 2021

Create rmfakecloud-proxy package #448

Create rmfakecloud-proxy package #448

Conversation

matteodelabre commented Sep 29, 2021 • edited by Eeems Loading

Technical details

Test plan

matteodelabre commented Sep 29, 2021

matteodelabre commented Sep 29, 2021

ddvk commented Sep 29, 2021

ddvk commented Sep 29, 2021

Eeems Sep 29, 2021

Choose a reason for hiding this comment

matteodelabre Sep 30, 2021

Choose a reason for hiding this comment

Eeems Oct 1, 2021

Choose a reason for hiding this comment

matteodelabre commented Oct 1, 2021

Eeems commented Oct 5, 2021

matteodelabre commented Oct 5, 2021

matteodelabre commented Oct 6, 2021

Eeems left a comment

Choose a reason for hiding this comment

matteodelabre commented Nov 27, 2021

matteodelabre commented Nov 27, 2021

Eeems commented Dec 5, 2021

Eeems left a comment

Choose a reason for hiding this comment

Eeems commented Dec 7, 2021

matteodelabre commented Dec 7, 2021

matteodelabre commented Sep 29, 2021 •

edited by Eeems

Loading