-
Notifications
You must be signed in to change notification settings - Fork 534
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: update test-all-versions configs to test fewer package versions #2464
Conversation
The TAV tests in CI are taking 2.5h - 3h, for a total usage of over 13h compute time. E.g.: https://github.com/open-telemetry/opentelemetry-js-contrib/actions/runs/11233652488/usage This change reduces the number of tested versions from 839 to 317 (for a TAV run with Node.js v20). For a few of the .tav.yml files the tested version *range* was actually increased to capture some versions that the instrumentation supported, but were not being tested by TAV.
versions: "2.308.0 || 2.556.0 || 2.801.0 || 2.1049.0 || 2.1297.0 || 2.1546.0 || >=2.1548.0" | ||
versions: | ||
include: "^2.308.0" | ||
mode: max-7 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TIL that this mode exists. Nice. 👍
@trentm I see in the ended up finding the way to do it (re-assigning a label). It's kind of a hack though, the workflow is not triggered directly when we apply labels, as we don't use the token of an actual github user, so the action is then performed by https://github.com/apps/github-actions instead. We could use the token from @opentelemetrybot, but then we'd need to give it triage rights on the repo - I think such a change would need to go though a request in the https://github.com/open-telemetry/community/ repo first. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, looks like we're hitting some versions that are failing tests, I think we need to take care of these first before merging.
Yup, I kind of expected some possible breakage -- I guess I should have started in draft. There were a few instrumentations that supported a version range larger than what TAV had been testing. I'll work through them. |
… issue with its spdy dep that is not worth digging into Also fine-tune the restify version at which there was a slight behaviour in error handling that broke tests: behaviour change was in v8.2.0, not in v8.0.0.
Stop installing some '@nestjs/*' deps that aren't necessary for testing with more recent major versions. Also turn of nestjs logging for testing to make the output much cleaner.
I believe `@aws-sdk/client-sqs` releases 3.363.0 to 3.377.0 were bad releases (fixed in 3.378.0). open-telemetry#1828 (comment) shows how `@aws-sdk/[email protected]` was a bad release. Version 3.363.0 was when the `client-*` packages were moved to using "migrated `@smithy` packages" https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.363.0 The comment above shows a particular failing for client-s3. I don't have a smoking gun for the client-sqs old versions, but the tests fail on those versions and it isn't worth digging into the reasons. Let's just skip those versions.
Repeating the commit messages from the last 4 commits:
|
- first issue was instr or tests or both don't work with 8.0.0, but *do* with 8.0.3 and later - second issue was an unrelated flakiness in pg.test.ts
A couple issues in the pg tests: The instrumentation (or at least the tests) don't actually work with
Then there was a flakiness in pg-pool.test.ts:
cause by this test:
It was an async test case that did not return a promise, so it could be running |
I was the one who create the test I also tried to change the approach and use
But got the same issue. I created a PR with this change anyway, since I think is a good change. Then we can discuss anything else that is missing there and it doesn't lose the focus of this PR |
I believe we should avoid releases 3.363.0 to 3.377.0 for all |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
amazing improvement, thanks for taking the time to work on this 🚀
The TAV tests in CI are taking 2.5h - 3h, for a total usage of over 13h
compute time. E.g.: https://github.com/open-telemetry/opentelemetry-js-contrib/actions/runs/11233652488/usage
This change reduces the number of tested versions from 839 to 317
(for a TAV run with Node.js v20). For a few of the .tav.yml files the
tested version range was actually increased to capture some versions
that the instrumentation supported, but were not being tested by TAV.