-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Keep service tests green 💚 #1359
Comments
This is a list of failures from a recent test run on master:
|
IcedFrisby/IcedFrisby#71 will allow us to set a per-test `timeout()` and per-test `retry()`, which should allow us to keep flaky tests green most of the time. A slough of service tests are failing locally, though they are also failing in master and seem unrelated to these changes. (#1359) IcedFrisby is maturing toward a 2.0 API. There's been one breaking change to the way dependencies are installed, and probably more changes to come in the API itself. Shields uses such a small part of that API that 2.0, when it's released, may not even affect us.
Bower should be fixed in #1383. Thanks @RedSparr0w! |
Also Code Climate Fixed in #1368 Although if you look at our comments after the merge we definitely still need to fix some problems, |
Down to 13 errors
|
Suggest is for a web service that provides badge suggestions to the front end. You can try it out by pasting a github URL into the search box at the top, and clicking Suggest Badges. |
Link for reference: https://circleci.com/gh/badges/shields/643 Github hit counter for nonexistent repo is one that I see timing out really often. We may as well bump that one up. Possibly hit counter too. The others I haven't seen time out before. PHP version - maybe it's a Travis rate limit? |
Opened #1398 for suggest. |
https://circleci.com/gh/badges/shields/728 A strange new error for
The "PHP version from .travis.yml" only seems to fail on the daily Circle CI test, unsure where it is failing though as 'invalid' is returned several places, guess it could be a rate limit against Circle CI's servers. |
That's a socket timeout presented by IcedFrisby in a strange way. I'm going to re-run the daily tests and ssh into the build machine to see if I can diagnose further. |
Interesting; it's this one: https://github.com/badges/shields/blob/master/server.js#L336-L341 So the problem is that the request to the GitHub API is failing. Probably that is because it's not being sent with auth information, which also explains why the request usually works locally, but does not in CI. I'll open a PR with a fix. |
To fix service test that fails in CI (due to no github auth) #1359 (comment) - DRY getPhpReleases() - Pass named options to regularUpdate - Add json option - php-version: Move helpers before functions, and move exports to end
Took a quick pass through our failing service tests at https://circleci.com/gh/badges/shields/1355 and https://circleci.com/gh/badges/shields/1411. This one seems to be failing consistently. Is it perhaps referring to an image that has gone away?
The other failures look to be related to Github API (timeouts – not auth failures) and the Travis API. |
For #1359 This failure from https://circleci.com/gh/badges/shields/1411?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link ``` 26) Uptime Robot Uptime Robot: Percentage (valid) [ GET http://localhost:1111/uptimerobot/ratio/m778918918-3e92c097147760ee39d02d36.json ]: ValidationError: child "value" fails because ["value" with value "99.992%" fails to match the required pattern: /^[0-9]+%$/] at Object.exports.process (node_modules/joi/lib/errors.js:190:19) at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31) at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23) at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9) at node_modules/icedfrisby/lib/icedfrisby.js:703:10 at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33) at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12) at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16) at Request.self.callback (node_modules/request/request.js:186:22) at Request.<anonymous> (node_modules/request/request.js:1163:10) at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12) at endReadableNT (_stream_readable.js:1056:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9) 27) Uptime Robot Uptime Robot: Percentage (valid, with numberOfDays param) [ GET http://localhost:1111/uptimerobot/ratio/7/m778918918-3e92c097147760ee39d02d36.json ]: ValidationError: child "value" fails because ["value" with value "99.967%" fails to match the required pattern: /^[0-9]+%$/] at Object.exports.process (node_modules/joi/lib/errors.js:190:19) at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31) at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23) at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9) at node_modules/icedfrisby/lib/icedfrisby.js:703:10 at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33) at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12) at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16) at Request.self.callback (node_modules/request/request.js:186:22) at Request.<anonymous> (node_modules/request/request.js:1163:10) at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12) at endReadableNT (_stream_readable.js:1056:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9) ```
Current failing tests:NPM package bundle size
dotnet-status
Github
Gratipay - #1449
Hackage
Nexus
PHP version from PHP-Eye
suggest
PHP version from .travis.yml
|
Gratipay is gone :( For #1359 This failure from https://circleci.com/gh/badges/shields/1411?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link ``` 7) Gratipay Receiving [ GET http://localhost:1111/gratipay/Gratipay.json ]: ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /^\$[0-9]+(\.[0-9]{2})?\/week/] at Object.exports.process (node_modules/joi/lib/errors.js:190:19) at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31) at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23) at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9) at node_modules/icedfrisby/lib/icedfrisby.js:703:10 at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33) at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12) at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16) at Request.self.callback (node_modules/request/request.js:186:22) at Request.<anonymous> (node_modules/request/request.js:1163:10) at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12) at endReadableNT (_stream_readable.js:1056:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9) ```
SonarQube has started failing again: 16) SonarQube
Tech Debt (legacy API unsupported)
[ GET http://localhost:1111/sonar/4.2/http/sonarqube.com/com.github.dannil:scb-java-client/tech_debt.json ]:
Error: Request timed out after 5000 ms
at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)
17) SonarQube
Coverage (legacy API unsupported)
[ GET http://localhost:1111/sonar/4.2/http/sonarqube.com/com.github.dannil:scb-java-client/coverage.json ]:
Error: Request timed out after 5000 ms
at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35) I suspect the repo is no longer valid for those tests. Also the json badge is behaving strangely for this too: |
Another possibility, since these particular tests are about what happens when the wrong API version is configured on the badge, is that the API being out of date is manifesting in a new way. I'm curious which of those it is. |
I expect this failure in NpmLicense is due to #2196: https://circleci.com/gh/badges/shields/20848 Fixing an issue like this is a great way for a new contributor to jump in. Any thoughts on the best way to track these? |
This does not seem to be the best place to track these issues. I think it may be a good place to coordinate on the larger project, or figure out how to best support new contributors in addressing. them. Let's default to opening new issues for these failures in the daily tests. |
We want our service tests to hit the real services. This makes some of them unreliable – out of necessity, not sloppiness. There's a long discussion of the test strategy in #927.
I’d like to use this issue as a place to identify and fix flaky service tests.
IcedFrisby/IcedFrisby#71 will allow us to set a per-test
timeout()
and per-testretry()
, which should allow us to keep flaky tests green most of the time.Of course we want the requests to be fast. Right now they fail after 2 seconds, which is a fairly long time to be waiting for a badge. Since readme images are served through GitHub’s proxy (camp) which has its own timeout, perhaps we should choose a threshold in relation to that. IcedFrisby has a separate method
.expectMaxResponseTime()
which we could use to assert the response time. We could also generate a tidy report of slow badges and print it as part of our daily run. An often-red build is not the best way to accomplish this! 😄The text was updated successfully, but these errors were encountered: