Adds v2 and v3 source interface urls to metadata endpoint.

[zenmonkeykstop]

Status

Ready for review

Description of Changes

Fixes #4757

Changes proposed in this pull request:

Source Interface information, including versions and submission key fingerprint, is currently available via the /metadata endpoint. This PR adds the v2 and v3 source URLs to the endpoint payload, allowing for discovery and monitoring of v3 versions of Source Interfaces.

This feature relies on the files /var/lib/securedrop/source_{v2,v3}_url, which are created when the install playbook is run from version 1.0.0 onward. In the absence of said files, null values are returned instead.

(I decided against adding a postinst task to create said files if the playbook hasn't been run, as if it hasn't been run then v3 hasn't been enabled anyway. Open to debate on this!)

Testing

Dev env:

• check out this branch and run make dev, then visit localhost:8080/metadata:
  • the response includes v2_source_url and v3_source_url fields with value null
• use docker ps and docker exec -it <PID> bash to log into the dev container, then create a file /var/lib/securedrop/source_v2_url containing an invalid onion URL (ie. NOT 16 alphanumeric chars followed by .onion), with ownership user:root and permissions 777, then check the /metadata endpoint again
  • both URL fields are still null
• update the /var/lib/securedrop/source_v2_url to contain a valid v2 URL, and check /metadata again
  • the v2_source_urlcontains the onion URL specified above
• repeat the two steps above for the /var/lib/source_v3_url file and verify that the v3_source_url behaves as expected.

Staging env:

• check out this branch and run make build-debs; make staging, then check the SI url in install_files/ansible-base/app-source-ths and visit its /metadata endpoint in Tor Browser.
  • the response includes the v2 and v3 source url fields, with values matching the contents of install_files/ansible-base/app-source-ths and install_files/ansible-base/app-sourcev3-ths

Deployment

As this change relies on files created by the install playbook, it will only return non-null values on instances where the playbook has been recently run (most likely to enable v3 onion services). For instances where this is not the case, null values will be returned instead.

For new instances, the file dependencies will exist, and so the values will be populated.

Checklist

If you made changes to the server application code:

• Linting (make lint) and tests (make test) pass in the development container

If you made changes to securedrop-admin:

• Linting and tests (make -C admin test) pass in the admin development container

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

[rmol]

👍

This works well. I had one nitpicky naming suggestion and an idea for an extra test.

I went through the test plan with the dev container (having trouble with quay.io at the moment, so couldn't test staging), and everything worked as described.

[rmol]

I got through staging and that worked too, of course.

[kushaldas]

Testing (worked as expected)

Dev env:

• check out this branch and run make dev, then visit localhost:8080/metadata:
  • the response includes v2_source_url and v3_source_url fields with value null
• use docker ps and docker exec -it <PID> bash to log into the dev container, then create a file /var/lib/securedrop/source_v2_url containing an invalid onion URL (ie. NOT 16 alphanumeric chars followed by .onion), with ownership user:root and permissions 777, then check the /metadata endpoint again
  • both URL fields are still null
• update the /var/lib/securedrop/source_v2_url to contain a valid v2 URL, and check /metadata again
  • the v2_source_urlcontains the onion URL specified above
• repeat the two steps above for the /var/lib/source_v3_url file and verify that the v3_source_url behaves as expected.

Staging env:

• check out this branch and run make build-debs; make staging, then check the SI url in install_files/ansible-base/app-source-ths and visit its /metadata endpoint in Tor Browser.
  • the response includes the v2 and v3 source url fields, with values matching the contents of install_files/ansible-base/app-source-ths and install_files/ansible-base/app-sourcev3-ths

🦄

[kushaldas]

@rmol we will need a fresh approved review from you, then we can merge this one.

[kushaldas]

As a note: the metadata endpoint provides JSON data like:

{'allow_document_uploads': True,
 'gpg_fpr': '65A1B5FF195B56353CC63DFFCC40EF1228271441',
 'sd_version': '1.3.0~rc1',
 'server_os': '16.04',
 'supported_languages': ['ar',
                         'ca',
                         'cs',
                         'de_DE',
                         'el',
                         'en_US',
                         'es_ES',
                         'fr_FR',
                         'hi',
                         'is',
                         'it_IT',
                         'nb_NO',
                         'nl',
                         'pt_BR',
                         'ro',
                         'ru',
                         'sk',
                         'sv',
                         'tr',
                         'zh_Hant'],
 'v2_source_url': 'aaaaaaaaaaaaaaaa.onion',
 'v3_source_url': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.onion'}