Added a role for the bridge mautrix-signal

[laszabine]

This role installs 3 services: the bridge itself, and its 2 dependencies: the signald daemon, and a postgres database. All run in docker containers.

The rest of the setup mirrors the mautrix-telegram bridge.

I also included documentation.

[ebbertd]

I suggest making it the default configuration that only user from YOUR_DOMAIN can access the bridge.

PS: I'm looking forward to using this bridge.

[laszabine]

Hi, thanks for commenting! I thought about doing that, but then I thought, perhaps some admins consider this setting too liberal: what if you don't want all the users registered to your domain to be able to use the bridge? Having no default permissions at all is safer, but may be annoying if you forget to adapt. But maybe I'm too paranoid, and it's reasonable to assume that what you suggest will work for the vast majority of admins.

[zapp42]

The container will not start if you forget to adapt. I think it should be at least working. I think "[]" is not valid yaml?

[laszabine]

Absolutely. Fixed it, and added documentation for how one can get what @ebbertd proposed.

[zapp42]

I had to add "-v /etc/passwd:/etc/passwd:ro" on the server for this to work. Otherwise it failed in chmod /var/run/postgres.

[laszabine]

Thanks for spotting that! I must have missed this during testing. I found this piece of documentation that you probably also found, which mentions the work-around you suggested as well as a second option. I committed your proposal as it was faster to do and shouldn't have any negative side-effects, even though the other options seems a bit cleaner.

[seanfarley]

Oh nice, thanks for making this PR! I'm excited to use it soon :-)

[spantaleev]

I think we should look into ways of making this use our already existing matrix-postgres Postgres database.

(even if it takes some manual effort one the user's part).

Otherwise, we're looking into maintaining multiple different Postgres instances, which also need to be backed up and upgraded (to new major Postgres versions) separately. It would be a pain to do for every service like this.

Besides this bridge, we also have others which would benefit from using Postgres.

Also, Dendrite and other services are coming, which would need their own Postgres as well. Each service using its own separate instance would be a pain.

I still haven't figured out a great way for us to share the existing matrix-postgres easily with all these services though.. I'm hoping I'll think of something good soon, so we can simplify this bridge's setup and also benefit from all the other things I mentioned.

[spantaleev]

Sorry for having this blocked for so long!

It seems like #740 is progressing, so I think it should be able to easily redo this role on top of it and make it use matrix-postgres. Perhaps in a few days :)

[spantaleev]

Now that #740 is merged, this bridge role can be greatly simplified.

Please remove the Postgres server setup stuff that is part of this role and switch to using matrix-postgres.

The way the other bridges do it is, they have a bunch of matrix_SERVICE_database_* fields that control where the bridge is pointed at. Example:

matrix-docker-ansible-deploy/roles/matrix-bridge-mautrix-facebook/defaults/main.yml

Lines 36 to 61 in 067f12b

	# Database-related configuration fields.
	#
	# To use SQLite, stick to these defaults.
	#
	# To use Postgres:
	# - change the engine (`matrix_mautrix_facebook_database_engine: 'postgres'`)
	# - adjust your database credentials via the `matrix_mautrix_facebook_postgres_*` variables
	matrix_mautrix_facebook_database_engine: 'sqlite'
	matrix_mautrix_facebook_sqlite_database_path_local: "{{ matrix_mautrix_facebook_data_path }}/mautrix-facebook.db"
	matrix_mautrix_facebook_sqlite_database_path_in_container: "/data/mautrix-facebook.db"
	matrix_mautrix_facebook_database_username: 'matrix_mautrix_facebook'
	matrix_mautrix_facebook_database_password: 'some-password'
	matrix_mautrix_facebook_database_hostname: 'matrix-postgres'
	matrix_mautrix_facebook_database_port: 5432
	matrix_mautrix_facebook_database_name: 'matrix_mautrix_facebook'
	matrix_mautrix_facebook_database_connection_string: 'postgres://{{ matrix_mautrix_facebook_database_username }}:{{ matrix_mautrix_facebook_database_password }}@{{ matrix_mautrix_facebook_database_hostname }}:{{ matrix_mautrix_facebook_database_port }}/{{ matrix_mautrix_facebook_database_name }}'
	matrix_mautrix_facebook_appservice_database: "{{
	{
	'sqlite': ('sqlite:///' + matrix_mautrix_facebook_sqlite_database_path_in_container),
	'postgres': matrix_mautrix_facebook_database_connection_string,
	}[matrix_mautrix_facebook_database_engine]
	}}"

While the bridge roles define sqlite as the default, this usually gets overriden to postgres in group_vars/matrix_servers.

matrix-docker-ansible-deploy/group_vars/matrix_servers

Lines 209 to 211 in 067f12b

	# Postgres is the default, except if not using `matrix_postgres` (internal postgres)
	matrix_mautrix_facebook_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}"
	matrix_mautrix_facebook_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.fb.db') | to_uuid }}"

The mautrix-signal bridge only supports postgres, so I guess there's no need for such stuff. You can hardcode the _engine variable to postgres or just not define it at all.

group_vars/matrix_server is also where we set the bridge's Postgres password:

matrix-docker-ansible-deploy/group_vars/matrix_servers

Line 211 in 067f12b

matrix_mautrix_facebook_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.fb.db') | to_uuid }}"

And finally, you'd need an entry in matrix_postgres_additional_databases in group_vars/matrix_servers, to tell the matrix-postgres role to create the database for you. Example:

matrix-docker-ansible-deploy/group_vars/matrix_servers

Lines 986 to 992 in 067f12b

	matrix_postgres_additional_databases: |
	{{
	([{
	'name': matrix_ma1sd_database_name,
	'username': matrix_ma1sd_database_username,
	'password': matrix_ma1sd_database_password,
	}] if (matrix_ma1sd_enabled and matrix_ma1sd_database_engine == 'postgres' and matrix_ma1sd_database_hostname == 'matrix-postgres') else [])

---

Let me know if you need any help with this!

[laszabine]

Thanks @spantaleev and everyone who was involved in making the shared db server happen!

I've updated the role now to work with this db server.

Here are some instructions for anyone who's been running this role already (based on commit cea2faa) on how to port the db to the present version (commit 84cac25). Tested on CentOS 7 using the postgres server as installed by the role matrix-postgres. NO GUARANTEES THAT THIS WILL WORK ON YOUR SYSTEM AND WON'T MESS IT UP!

Instructions for porting the db

You should probably start by running the playbook with the tags setup-all,start to make sure your signal bridge dockers are up to date.

You should probably also make sure your homeserver's postgres is up to date.

After that is done, feel free to check out the current version of this role.

On your server, run:

systemctl stop matrix-mautrix-signal
docker exec -ti matrix-mautrix-signal-db bash

Inside the docker, run:

createuser -U signal-db-user --superuser postgres
psql -U postgres postgres

Assuming that signal-db is your value of matrix_mautrix_signal_db_database (which is the default), and signal-db-user is your value of matrix_mautrix_signal_db_user (which is the default).

Assuming that you want to keep the new defaults matrix_mautrix_signal for username and db name, in the postgres shell, run:

alter user "signal-db-user" rename to "matrix_mautrix_signal"; 
alter database "signal-db" rename to "matrix_mautrix_signal";
exit

Back in the docker's bash, run:

pg_dump --create -d matrix_mautrix_signal -U matrix_mautrix_signal > /var/lib/postgresql/data/dump.sql
exit

Back on you server, you will find the file dump.sql at the path specified in matrix_mautrix_signal_db_storage_path. (Defaults to /matrix/mautrix-signal/database/.) Make a note of this path.

On your local machine, where you run the playbook, check out the current commit of this role if you haven't yet.

Then run the playbook with the tag setup-postgres:

ansible-playbook -i inventory/hosts setup.yml \
--ask-become-pass \
--tags=setup-postgres

This should create the new database for signal on the main postgres server.

Then import the dumped database into this new database by running the playbook with the tag import-postgres:

ansible-playbook -i inventory/hosts setup.yml \
--ask-become-pass \
--extra-vars="server_path_postgres_dump=/matrix/mautrix-signal/database/dump.sql" \
--tags=import-postgres

or wherever on the remote machine your dump.sql file is. This should populate the newly created database matrix_mautrix_signal with the data from your old bridge installation.

Run the playbook again to reinstall signal and start all services:

ansible-playbook -i inventory/hosts setup.yml \
--ask-become-pass \
--tags=setup-all,start

(Maybe it would suffice to run the tags setup-mautrix-signal,start, but I don't think running setup-all,start instead can hurt.)

Make sure the bridge works and all your data are there.

Manually remove the old signal database (instructions copied from https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/uninstalling.md). If you want to keep your dump.sql, move it out of /matrix/mautrix-signal/database/ first.

systemctl stop matrix-mautrix-signal-db
rm -f /etc/systemd/system/matrix-mautrix-signal-db.service
systemctl daemon-reload
docker rm matrix-mautrix-signal-db
rm -rf /matrix/mautrix-signal/database

Edit: Fixed the tag name setup-mautrix-signal.

[spantaleev]

Thanks a lot for all of your work on it, for the patience (for #740) and for redoing things for that! 👍

I've made a few minor changes (see above) and think this looks good for merging! Let me know if my last-minute changes broke something, etc.