Skip to content
This repository has been archived by the owner on Jan 7, 2024. It is now read-only.

ci: run SDK tests against latest version of API #59

Merged
merged 7 commits into from
Mar 20, 2019

Conversation

redshiftzero
Copy link
Contributor

the idea in this PR is to add a circle CI job that runs nightly and (I can be convinced the latter is a bad idea) on each PR. the circle CI job will run the API tests against the server container

that way if there's a discrepancy between API and SDK, we can detect it (ref: #55)

(btw I did this in a fork so that my frequent force pushing doesn't create a billion notifications)

@redshiftzero
Copy link
Contributor Author

btw the tests here are running against the server in CI, the issue is that some tests change state (i.e. deleting sources/submissions) - that's the remaining work here before this is ready for review

@redshiftzero redshiftzero self-assigned this Mar 18, 2019
@redshiftzero redshiftzero force-pushed the ci-sdk-api branch 3 times, most recently from 6b1d764 to ce95c86 Compare March 18, 2019 23:24
@redshiftzero redshiftzero force-pushed the ci-sdk-api branch 5 times, most recently from 9f9bda0 to 568c72b Compare March 19, 2019 21:22
@redshiftzero redshiftzero changed the title [wip] ci: run SDK tests against latest version of API ci: run SDK tests against latest version of API Mar 19, 2019
@redshiftzero
Copy link
Contributor Author

OK this is working and ready for review. Once this is approved I'll update the required CI jobs for merge. Note that the diff looks big here, but this is mostly VCR cassettes getting updated.

What I think we want to do next - ideally in a followup since what's here so far is a concrete step forward testing-wise - is:

  1. Add to the developer documentation (ref: Add developer documentation #6) the update process for the Qubes RPC JSON files (I have not updated those test cases in this PR nor afaict has anyone since the time they were added).
  2. DRY up the TestAPI and TestAPIProxy test classes if we can - we have basically duplicated tests in two files with different decorators on them.

@redshiftzero redshiftzero requested a review from kushaldas March 19, 2019 22:17
- run:
name: Wait for server to be up and for test sources to load
command: |
dockerize -wait tcp://127.0.0.1:8080 -timeout 15m
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might not be enough time. Building the dev container is slooooooooow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haha, it feels like it takes a long time but it "only" takes around 4-5 minutes

only:
- master
jobs:
- test-against-latest-api
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: missing newline at EOF

@heartsucker
Copy link
Contributor

DRY up the TestAPI and TestAPIProxy test classes if we can - we have basically duplicated tests in two files with different decorators on them

The follow up should also include an env var / CLI opt to ignore the cassettes that way we don't have to do an rm data/*.yml ; make test ; git checkout -- data/ to "reset" everything. This could be in the decorator.

Copy link
Contributor

@kushaldas kushaldas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the one question, rest looks great. Thank you 👍 Approved.

],
"markers": "python_version >= '3.4'",
"version": "==1.3.0"
"version": "==1.2.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we going back to an older release?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We specify in our Pipfile Python 3.5 (and in my local dev env I use pyenv to use only that version). However in yarl 1.2.1 they pinned the minimum Python version to be 3.5.3. So this can cause confusing local failures for developers if they're using Python 3.5.0, so I set it back to 1.2.0 for compatibility with all Python 3.5.* since there were no security updates in the diff from yarl 1.2.0 to yarl 1.3.0.

@redshiftzero redshiftzero merged commit 6b5924c into freedomofpress:master Mar 20, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants