test: Add Tokenserver integration tests #1152

ethowitz · 2021-09-17T18:22:04Z

Description

Adds a suite of integration tests for Tokenserver. These tests can be run remotely against an arbitrary Tokenserver, meaning they can be run against both the old and the new Tokenserver. All of the tests are passing against the old Tokenserver. Some of the tests were adapted from the tests in this file, but most of them are newly-written.

I also made some changes to the Rust code (there were some errors that the newly-added integration tests caught 🥲)

Testing

Add a MockOAuthVerifier to this file defined like so:

class MockOAuthVerifier(object):
    implements(IOAuthVerifier)
    
    def verify(self, token):
        decoded = jwt.decode(token, options={'verify_signature': False, 'verify_exp': False})
        
        idpclaims = {}
        if decoded.get('fxa-generation') is not None:
            idpclaims['fxa-generation'] = decoded['fxa-generation']
        return {
            'email': decoded.get('sub') + '@' + 'api-accounts.stage.mozaws.net',
            'idpClaims': idpclaims
        }

Start the old Tokenserver with this configuration file
In run.py, comment out this block of code and replace it with a call to this function
Ensure that the tests all pass
Edit the Tokenserver configuration file to comment out line 39 and uncomment lines 40 and 41
Restart Tokenserver
Edit run.py to replace the call to run_local_tests() with a call to run_end_to_end_tests()
Ensure the tests all pass

Issue(s)

Closes #1048

pjenvey · 2021-09-30T00:04:51Z

* Add a `MockOAuthVerifier` to [this file](https://github.com/mozilla-services/tokenserver/blob/master/tokenserver/verifiers.py) defined like so:

I'll note this needed an additional import jwt

 * Start the old Tokenserver with [this configuration file](https://gist.github.com/ethowitz/105744a7d4ffaa74f509b0c0bb3da605)

* In [run.py](https://github.com/mozilla-services/syncstorage-rs/blob/961ab165b7f30ee716fe9699ca04ace23927acac/tools/integration_tests/run.py), comment out [this block of code](https://github.com/mozilla-services/syncstorage-rs/blob/961ab165b7f30ee716fe9699ca04ace23927acac/tools/integration_tests/run.py#L51-L54) and replace it with a call to [this function](https://github.com/mozilla-services/syncstorage-rs/blob/961ab165b7f30ee716fe9699ca04ace23927acac/tools/integration_tests/tokenserver/run.py#L12)

* Ensure that the tests all pass

I'm seeing some errors here. Should I have set TOKENSERVER_DATABASE_* to point to the same db as the running old tokenserver's db?

...............
----------------------------------------------------------------------
Ran 15 tests in 1.408s

OK
........
----------------------------------------------------------------------
Ran 8 tests in 0.709s

OK
EE.E
======================================================================
ERROR: test_new_user_allocation (tokenserver.test_node_assignment.TestNodeAssignment)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 516, in cmd_query
    query_attrs=self._query_attrs)
_mysql_connector.MySQLInterfaceError: Duplicate entry '2-https://example.com' for key 'unique_idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_node_assignment.py", line 53, in test_new_user_allocation
    self._add_node(available=100, current_load=0, capacity=100, downed=1)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_support.py", line 106, in _add_node
    cursor = self._execute_sql(query, data)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_support.py", line 225, in _execute_sql
    cursor.execute(query, args)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 271, in execute
    raw_as_string=self._raw_as_string)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 519, in cmd_query
    sqlstate=exc.sqlstate)
mysql.connector.errors.IntegrityError: 1062 (23000): Duplicate entry '2-https://example.com' for key 'unique_idx'

======================================================================
ERROR: test_successfully_releasing_node_capacity (tokenserver.test_node_assignment.TestNodeAssignment)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 516, in cmd_query
    query_attrs=self._query_attrs)
_mysql_connector.MySQLInterfaceError: Duplicate entry '2-node4' for key 'unique_idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_node_assignment.py", line 87, in test_successfully_releasing_node_capacity
    node='node4', downed=1)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_support.py", line 106, in _add_node
    cursor = self._execute_sql(query, data)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_support.py", line 225, in _execute_sql
    cursor.execute(query, args)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 271, in execute
    raw_as_string=self._raw_as_string)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 519, in cmd_query
    sqlstate=exc.sqlstate)
mysql.connector.errors.IntegrityError: 1062 (23000): Duplicate entry '2-node4' for key 'unique_idx'

======================================================================
ERROR: test_user_creation (tokenserver.test_node_assignment.TestNodeAssignment)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 516, in cmd_query
    query_attrs=self._query_attrs)
_mysql_connector.MySQLInterfaceError: Duplicate entry '2-https://example.com' for key 'unique_idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_node_assignment.py", line 18, in test_user_creation
    self._add_node(available=0)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_support.py", line 106, in _add_node
    cursor = self._execute_sql(query, data)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/tools/integration_tests/tokenserver/test_support.py", line 225, in _execute_sql
    cursor.execute(query, args)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 271, in execute
    raw_as_string=self._raw_as_string)
  File "/media/ext/pjenvey/src/moz/syncstorage-rs/e2e-env/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 519, in cmd_query
    sqlstate=exc.sqlstate)
mysql.connector.errors.IntegrityError: 1062 (23000): Duplicate entry '2-https://example.com' for key 'unique_idx'

----------------------------------------------------------------------
Ran 4 tests in 0.318s

FAILED (errors=3)
Traceback (most recent call last):
  File "tools/integration_tests/run.py", line 57, in <module>
    sys.exit(res)
NameError: name 'res' is not defined

ethowitz · 2021-09-30T14:25:29Z

@pjenvey I have tokenserver.database_url in my local.toml set to a database I made specifically for the rust version with the schema created by running the migrations via:

 diesel --database-url mysql://your_db_url_here migration --migration-dir src/tokenserver/migrations run

But the problem here is that there's a difference between the schema created by the Rust migrations and the schema created by the old Tokenserver. Specifically, the new schema is missing a unique key constraint on (service, node); I'll add a migration to resolve that and a couple of other differences between the schemas. Thanks for the catch!

ethowitz · 2021-09-30T15:34:16Z

I added the necessary migrations and fixed the integration tests

ethowitz · 2021-09-30T15:54:34Z

Also, here is the command I use to run the tests:

TOKENSERVER_HOST=http://localhost:8000 \
    SYNC_TOKENSERVER__DATABASE_URL=mysql://sample_user:sample_password@localhost/tokenserver_rs \
    SYNC_DATABASE_URL=mysql://sample_user:sample_password@localhost/syncstorage_rs \
    SYNC_HOST=0.0.0.0 \
    SYNC_MASTER_SECRET=secret0 \
    TOKENSERVER_DATABASE_USER=sample_user \
    TOKENSERVER_DATABASE_PASSWORD=sample_password \
    TOKENSERVER_DATABASE_NAME=tokenserver_rs \
    TOKENSERVER_DATABASE_HOST=localhost \
    python tools/integration_tests/run.py http://localhost:8000#secret0

jrconlin · 2021-09-30T22:56:39Z

src/error.rs

-            res.request().clone(),
-            resp.into_body(),
-        )))
+        if res.request().path().starts_with("/1.0/") {


🤔I wonder if this pattern might creep up enough that it's worth creating a macro or function that encapsulates it?

Do you foresee other types of requests aside from TS ones for which we wouldn't want to render the default Sync 404 response?

No, probably not. Just seemed like a pattern.

To keep things simpler, I'd probably wait to add another macro/function until we have another instance of the pattern, but I'd be happy to add it now if you feel strongly about it!

pjenvey · 2021-10-04T23:07:28Z

Thanks @ethowitz, I got local tests passing w/ the new migrations applied and also the e2e tests passing.

pjenvey

Looks good. You may want to consider supporting a db style URL for the new settings vs individual TOKENSERVER_USERNAME/PASS/DATABASE etc. which is a little tedious (maybe via sqlalchemy's lower level "core" API might not be too much work) depending on how commonly devs/testers would need to set this up.

ethowitz · 2021-10-05T19:56:18Z

@pjenvey Done! That change was much easier than I thought it'd be...the APIs are pretty much identical

pjenvey

Great, thanks!

ethowitz requested a review from a team September 17, 2021 18:22

ethowitz mentioned this pull request Sep 17, 2021

feat: Tokenserver: Add validations and user updating for generation, keys_changed_at, and client_state #1145

Merged

Ethan Donowitz added 5 commits September 22, 2021 19:18

test: Add Tokenserver integration tests

ebd86a7

Closes #1048

remove unnecessary requirements

9343bf1

use absolute paths

edb91e0

fix flake8 errors

248929e

add test

7de1c09

fix schema

708ba57

ethowitz force-pushed the test/1058-add-integration-tests branch from 1082583 to 708ba57 Compare September 30, 2021 15:33

jrconlin previously approved these changes Sep 30, 2021

View reviewed changes

minor changes; remove file

722d0ce

ethowitz dismissed jrconlin’s stale review via 722d0ce October 4, 2021 20:50

flake8

172f3e6

ethowitz requested a review from jrconlin October 4, 2021 20:53

pjenvey previously approved these changes Oct 4, 2021

View reviewed changes

ethowitz mentioned this pull request Oct 5, 2021

feat: Tokenserver: Add node assignment logic #1158

Merged

use sqlalchemy

4401cb9

ethowitz dismissed pjenvey’s stale review via 4401cb9 October 5, 2021 19:55

flake8

f39978a

pjenvey approved these changes Oct 5, 2021

View reviewed changes

ethowitz merged commit 7209ccf into master Oct 5, 2021

ethowitz deleted the test/1058-add-integration-tests branch October 5, 2021 20:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Add Tokenserver integration tests #1152

test: Add Tokenserver integration tests #1152

ethowitz commented Sep 17, 2021

pjenvey commented Sep 30, 2021

ethowitz commented Sep 30, 2021

ethowitz commented Sep 30, 2021

ethowitz commented Sep 30, 2021

jrconlin Sep 30, 2021

ethowitz Oct 4, 2021

jrconlin Oct 4, 2021

ethowitz Oct 4, 2021

pjenvey commented Oct 4, 2021

pjenvey left a comment

ethowitz commented Oct 5, 2021

pjenvey left a comment

test: Add Tokenserver integration tests #1152

test: Add Tokenserver integration tests #1152

Conversation

ethowitz commented Sep 17, 2021

Description

Testing

Issue(s)

pjenvey commented Sep 30, 2021

ethowitz commented Sep 30, 2021

ethowitz commented Sep 30, 2021

ethowitz commented Sep 30, 2021

jrconlin Sep 30, 2021

Choose a reason for hiding this comment

ethowitz Oct 4, 2021

Choose a reason for hiding this comment

jrconlin Oct 4, 2021

Choose a reason for hiding this comment

ethowitz Oct 4, 2021

Choose a reason for hiding this comment

pjenvey commented Oct 4, 2021

pjenvey left a comment

Choose a reason for hiding this comment

ethowitz commented Oct 5, 2021

pjenvey left a comment

Choose a reason for hiding this comment