Release test for import from last two versions before current #197
Comments
azaslavsky
added a commit
to getsentry/sentry
that referenced
this issue
Oct 16, 2023
The goal for the export.json feature is to support two minor versions (since we're using CalVer, roughly equivalent to two months) of releases when importing. That is to say, if the latest release is 23.10.0, and the previous two releases were 23.9.0 and 23.8.0, then a self-hosted release running code from [email protected] or self-hosted@latest (aka nightly) should be able to successfully import JSON backup data generated by [email protected] and [email protected], but not necessarily [email protected]. To support this, we add a new file `test_releases.py`, that provides a snapshot test of an "exhaustive" (aka, has at least one of every exportable model) export. Every time we cut a new release, we will add a new commit directly after the release commit that renames the `test_at_head` snapshot to `test_at_<RELEASED_VERSION>`, and creates a test case for that version (see `test_at_23_9_1` in this PR for one such example). Test cases older than the 2 version support window will simultaneously be removed. In the future, we plan to automate this "release test rotation" process via GitHub actions, but for now the ospo team will take care to do this manually for every release. Some other notes of interest: - Exports cannot be compared for equality using simple string comparisons, since they have special comparators for cases like passwords, dates that may or may not have been updated, etc (see comparators.py for greater detail). To accommodate this, we extend the `insta_snapshot` fixture with an extra argument, `inequality_comparator`, that allows a custom comparison lambda to be passed in for such cases. - The `test_at_23_9_1` snapshot was generated by taking an early version of this PR, cherry-picking it onto the `23.9.1` release tag, generating a new snapshot there, then copying the result back into the main commit. Closes getsentry/team-ospo#197
azaslavsky
added a commit
to getsentry/sentry
that referenced
this issue
Oct 16, 2023
The goal for the export.json feature is to support two minor versions (since we're using CalVer, roughly equivalent to two months) of releases when importing. That is to say, if the latest release is 23.10.0, and the previous two releases were 23.9.0 and 23.8.0, then a self-hosted release running code from [email protected] or self-hosted@latest (aka nightly) should be able to successfully import JSON backup data generated by [email protected] and [email protected], but not necessarily [email protected]. To support this, we add a new file `test_releases.py` that provides a snapshot test of an "exhaustive" (aka, has at least one of every exportable model) export. Every time we cut a new release, we will add a new commit directly after the release commit that renames the `test_at_head` snapshot to `test_at_<RELEASED_VERSION>`, and creates a test case for that version (see `test_at_23_9_1` in this PR for one such example). Test cases older than the 2 version support window will simultaneously be removed. In the future, we plan to automate this "release test rotation" process via GitHub actions, but for now the ospo team will take care to do this manually for every release. Some other notes of interest: - Exports cannot be compared for equality using simple string comparisons, since they have special comparators for cases like passwords, dates that may or may not have been updated, etc (see comparators.py for greater detail). To accommodate this, we extend the `insta_snapshot` fixture with an extra argument, `inequality_comparator`, that allows a custom comparison lambda to be passed in for such cases. - The `test_at_23_9_1` snapshot was generated by taking an early version of this PR, cherry-picking it onto the `23.9.1` release tag, generating a new snapshot there, then copying the result back into the main commit. Closes getsentry/team-ospo#197
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In issue #159 I scoped out release tests, but then decided against it. I am now bringing this test regime back, but in a more limited form: we only validate compatibility with the last two non-patch versions, not all versions generally. There are several reasons to do this:
The validation pipeline will rely on the latest self-hosted release. This means that Sentry nightly must work with self-hosted at least one version back, which itself needs Sentry from that version. As a concrete example: the Sentry SaaS service currently runs 23.9.dev (ie, nightly at head), and the latest self-hosted release is 23.8.0, which itself includes [email protected] as a dependency. So we need to support at least one version back for this feature to work at all.
But what if a user submitted some slightly outdated JSON? Ie, if they take their backup on this upcoming Sept 15, but on Sept 16 we release 23.9.0, they will already be one version behind even though they have acted very reasonably. To avoid this, we give them an extra version's worth of buffer.
The text was updated successfully, but these errors were encountered: