Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep service tests green 💚 #1359

Closed
paulmelnikow opened this issue Dec 8, 2017 · 16 comments
Closed

Keep service tests green 💚 #1359

paulmelnikow opened this issue Dec 8, 2017 · 16 comments
Labels
developer-experience Dev tooling, test framework, and CI keep-service-tests-green Related to fixing failing tests of the services service-badge New or updated service badge

Comments

@paulmelnikow
Copy link
Member

We want our service tests to hit the real services. This makes some of them unreliable – out of necessity, not sloppiness. There's a long discussion of the test strategy in #927.

I’d like to use this issue as a place to identify and fix flaky service tests.

IcedFrisby/IcedFrisby#71 will allow us to set a per-test timeout() and per-test retry(), which should allow us to keep flaky tests green most of the time.

Of course we want the requests to be fast. Right now they fail after 2 seconds, which is a fairly long time to be waiting for a badge. Since readme images are served through GitHub’s proxy (camp) which has its own timeout, perhaps we should choose a threshold in relation to that. IcedFrisby has a separate method .expectMaxResponseTime() which we could use to assert the response time. We could also generate a tidy report of slow badges and print it as part of our daily run. An often-red build is not the best way to accomplish this! 😄

@paulmelnikow paulmelnikow added the developer-experience Dev tooling, test framework, and CI label Dec 8, 2017
@paulmelnikow
Copy link
Member Author

paulmelnikow commented Dec 8, 2017

This is a list of failures from a recent test run on master:

  1) Bower
       licence

  [ GET http://localhost:1111/bower/l/bootstrap.json ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  2) Bower
       custom label for licence

  [ GET http://localhost:1111/bower/l/bootstrap.json?label="my licence" ]:

      AssertionError: expected { name: '"my licence"', value: 'MIT' } to deeply equal { name: 'my licence', value: 'MIT' }
      + expected - actual

       {
      -  "name": "\"my licence\""
      +  "name": "my licence"
         "value": "MIT"
       }

      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:142:38)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  3) Bower
       custom label for version

  [ GET http://localhost:1111/bower/v/bootstrap.json?label="my version" ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  4) Bower
       custom label for pre version

  [ GET http://localhost:1111/bower/vpre/bootstrap.json?label="pre version" ]:
     ValidationError: child "name" fails because ["name" must be one of [pre version]]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:663:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:299:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:738:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  5) Bower
       Pre Version for Invaild Package

  [ GET http://localhost:1111/bower/vpre/it-is-a-invalid-package-should-error.json ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  6) Bower
       licence for Invaild Package

  [ GET http://localhost:1111/bower/l/it-is-a-invalid-package-should-error.json ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  7) Code Climate
       test coverage score

  [ GET http://localhost:1111/codeclimate/c/Nickersoft/dql.json ]:
     ValidationError: child "value" fails because ["value" must be one of [A, B, C, D, F, unknown]]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:663:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:299:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:738:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  8) suggest
       issues, forks, stars and twitter

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  9) suggest
       license

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  10) suggest
       license for non-existing project

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  11) suggest
       license when json response is invalid

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  12) suggest
       license when html_url not found in GitHub api response

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

paulmelnikow added a commit that referenced this issue Dec 9, 2017
IcedFrisby/IcedFrisby#71 will allow us to set a per-test `timeout()` and per-test `retry()`, which should allow us to keep flaky tests green most of the time.

A slough of service tests are failing locally, though they are also failing in master and seem unrelated to these changes. (#1359)

IcedFrisby is maturing toward a 2.0 API. There's been one breaking change to the way dependencies are installed, and probably more changes to come in the API itself. Shields uses such a small part of that API that 2.0, when it's released, may not even affect us.
@paulmelnikow
Copy link
Member Author

Bower should be fixed in #1383. Thanks @RedSparr0w!

@RedSparr0w
Copy link
Member

RedSparr0w commented Dec 22, 2017

Also Code Climate Fixed in #1368

Although if you look at our comments after the merge we definitely still need to fix some problems,
As the tests do not have 100% coverage for all of their badges, the other ones won't currently fail.

@RedSparr0w
Copy link
Member

RedSparr0w commented Dec 26, 2017

Down to 13 errors

  • Github - Seems to be consistently timing out, but looks to be running correctly without issue for me.
  • Suggest - Not sure what this is, haven't properly looked into it yet.
  • PHP version from .travis.yml - Not sure why this is failing as it actually shows as expected.

@paulmelnikow
Copy link
Member Author

Suggest is for a web service that provides badge suggestions to the front end. You can try it out by pasting a github URL into the search box at the top, and clicking Suggest Badges.

@paulmelnikow
Copy link
Member Author

Link for reference: https://circleci.com/gh/badges/shields/643

Github hit counter for nonexistent repo is one that I see timing out really often. We may as well bump that one up. Possibly hit counter too. The others I haven't seen time out before.

PHP version - maybe it's a Travis rate limit?

@paulmelnikow
Copy link
Member Author

Opened #1398 for suggest.

@RedSparr0w
Copy link
Member

https://circleci.com/gh/badges/shields/728

A strange new error for GitHub open issues by multi-word label is > zero today:

[ GET http://localhost:1111/github/issues/Cockatrice/Cockatrice/App%20-%20Cockatrice.json ]:
Error: Error parsing JSON string: Unexpected token I in JSON at position 1
Given: [IcedFrisby] Destination URL may be down or URL is invalid, Error: ESOCKETTIMEDOUT

The "PHP version from .travis.yml" only seems to fail on the daily Circle CI test, unsure where it is failing though as 'invalid' is returned several places, guess it could be a rate limit against Circle CI's servers.

@paulmelnikow
Copy link
Member Author

That's a socket timeout presented by IcedFrisby in a strange way.

I'm going to re-run the daily tests and ssh into the build machine to see if I can diagnose further.

@paulmelnikow
Copy link
Member Author

paulmelnikow commented Dec 28, 2017

Interesting; it's this one: https://github.com/badges/shields/blob/master/server.js#L336-L341

So the problem is that the request to the GitHub API is failing. Probably that is because it's not being sent with auth information, which also explains why the request usually works locally, but does not in CI. I'll open a PR with a fix.

paulmelnikow added a commit that referenced this issue Dec 29, 2017
To fix service test that fails in CI (due to no github auth) #1359 (comment)

- DRY getPhpReleases()
- Pass named options to regularUpdate
    - Add json option
- php-version: Move helpers before functions, and move exports to end
@paulmelnikow
Copy link
Member Author

Took a quick pass through our failing service tests at https://circleci.com/gh/badges/shields/1355 and https://circleci.com/gh/badges/shields/1411.

This one seems to be failing consistently. Is it perhaps referring to an image that has gone away?

  10) MicroBadger
       image size without a specified tag
         
	[ GET http://localhost:1111/microbadger/image-size/_/hello-world.json ]:
     ValidationError: child "value" fails because ["value" with value "error" fails to match the required pattern: /^[0-9]*[.]?[0-9]+\s(B|kB|MB|GB|TB|PB|EB|ZB|YB)$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

The other failures look to be related to Github API (timeouts – not auth failures) and the Travis API.

I opened #1448 and #1449 to fix other failures.

paulmelnikow added a commit that referenced this issue Jan 15, 2018
For #1359

This failure from https://circleci.com/gh/badges/shields/1411?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

```
  26) Uptime Robot
       Uptime Robot: Percentage (valid)
         
  [ GET http://localhost:1111/uptimerobot/ratio/m778918918-3e92c097147760ee39d02d36.json ]:
     ValidationError: child "value" fails because ["value" with value "99.992%" fails to match the required pattern: /^[0-9]+%$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  27) Uptime Robot
       Uptime Robot: Percentage (valid, with numberOfDays param)
         
  [ GET http://localhost:1111/uptimerobot/ratio/7/m778918918-3e92c097147760ee39d02d36.json ]:
     ValidationError: child "value" fails because ["value" with value "99.967%" fails to match the required pattern: /^[0-9]+%$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)
```
@RedSparr0w
Copy link
Member

RedSparr0w commented Feb 14, 2018

Current failing tests:

NPM package bundle size
  1) NPM package bundle size
       Testing format '/bundlephobia/:type/@:scope/:package.:format' against '/min/@cycle/core.json'
         
	[ GET http://localhost:1111/bundlephobia/min/@cycle/core.json ]:
     ValidationError: child "value" fails because ["value" with value "package not found error" fails to match the required pattern: /^[0-9]*[.]?[0-9]+\s(B|kB|MB|GB|TB|PB|EB|ZB|YB)$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  2) NPM package bundle size
       Testing format '/bundlephobia/:type/@:scope/:package.:format' against '/minzip/@cycle/core.json'
         
	[ GET http://localhost:1111/bundlephobia/minzip/@cycle/core.json ]:
     ValidationError: child "value" fails because ["value" with value "package not found error" fails to match the required pattern: /^[0-9]*[.]?[0-9]+\s(B|kB|MB|GB|TB|PB|EB|ZB|YB)$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)
dotnet-status
  3) dotnet-status
       get nuget package status
         
	[ GET http://localhost:1111/dotnetstatus/gh/jaredcnance/dotnet-status/API.json ]:
     ValidationError: child "value" fails because ["value" must be one of [up to date, out of date, processing]]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  4) dotnet-status
       get nuget package status
         
	[ GET http://localhost:1111/dotnetstatus/gh/jaredcnance/dotnet-status/invalid-project.json ]:

      AssertionError: expected { Object (name, value) } to deeply equal { Object (name, value) }
      + expected - actual

       {
         "name": "dependencies"
      -  "value": "inconclusive"
      +  "value": "project not found"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)
Github
  5) Github
       GitHub open issues by multi-word label is > zero
         
	[ GET http://localhost:1111/github/issues/Cockatrice/Cockatrice/App%20-%20Cockatrice.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  6) Github
       GitHub open issues by label (raw)
         
	[ GET http://localhost:1111/github/issues-raw/badges/shields/service-badge.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  7) Github
       GitHub open pull requests by label
         
	[ GET http://localhost:1111/github/issues-pr/badges/shields/service-badge.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  8) Github
       GitHub open pull requests by label (raw)
         
	[ GET http://localhost:1111/github/issues-pr-raw/badges/shields/service-badge.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  9) Github
       hit counter
         
	[ GET http://localhost:1111/github/search/torvalds/linux/goto.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  10) Github
       hit counter for nonexistent repo
         
	[ GET http://localhost:1111/github/search/torvalds/not-linux/goto.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)
Gratipay - #1449
  11) Gratipay
       Receiving
         
	[ GET http://localhost:1111/gratipay/Gratipay.json ]:
     ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /^\$[0-9]+(\.[0-9]{2})?\/week/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)
Hackage
  12) Hackage
       hackage version (valid)
         
	[ GET http://localhost:1111/hackage/v/lens.json ]:
" fails to match the required pattern: /^v\d+(\.\d+)?(\.\d+)?$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)
Nexus
  13) Nexus
       search release version
         
	[ GET http://localhost:1111/nexus/r/https/repository.jboss.org/nexus/jboss/jboss-client.json ]:
     ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /^v4(\.\d+)+$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  14) Nexus
       search release version of an inexistent artifact
         
	[ GET http://localhost:1111/nexus/r/https/repository.jboss.org/nexus/jboss/inexistent-artifact-id.json ]:

      AssertionError: expected { name: 'nexus', value: 'invalid' } to deeply equal { name: 'nexus', value: 'no-artifact' }
      + expected - actual

       {
         "name": "nexus"
      -  "value": "invalid"
      +  "value": "no-artifact"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  15) Nexus
       search snapshot version
         
	[ GET http://localhost:1111/nexus/s/https/repository.jboss.org/nexus/com.progress.fuse/fusehq.json ]:
     ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /-SNAPSHOT$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  16) Nexus
       search snapshot version of a release artifact
         
	[ GET http://localhost:1111/nexus/s/https/repository.jboss.org/nexus/jboss/jboss-client.json ]:

      AssertionError: expected { name: 'nexus', value: 'invalid' } to deeply equal { name: 'nexus', value: 'undefined' }
      + expected - actual

       {
         "name": "nexus"
      -  "value": "invalid"
      +  "value": "undefined"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  17) Nexus
       search snapshot version of an inexistent artifact
         
	[ GET http://localhost:1111/nexus/s/https/repository.jboss.org/nexus/jboss/inexistent-artifact-id.json ]:

      AssertionError: expected { name: 'nexus', value: 'invalid' } to deeply equal { name: 'nexus', value: 'no-artifact' }
      + expected - actual

       {
         "name": "nexus"
      -  "value": "invalid"
      +  "value": "no-artifact"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)
PHP version from PHP-Eye
  18) PHP version from PHP-Eye
       gets the package version of symfony
         
	[ GET http://localhost:1111/php-eye/symfony/symfony.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  19) PHP version from PHP-Eye
       gets the package version of symfony 2.8
         
	[ GET http://localhost:1111/php-eye/symfony/symfony/v2.8.0.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  20) PHP version from PHP-Eye
       gets the package version of yii
         
	[ GET http://localhost:1111/php-eye/yiisoft/yii.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  21) PHP version from PHP-Eye
       invalid package name
         
	[ GET http://localhost:1111/php-eye/frodo/is-not-a-package.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)
suggest
  22) suggest
       issues, forks, stars and twitter
         
	[ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  23) suggest
       license
         
	[ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)
PHP version from .travis.yml
  24) PHP version from .travis.yml
       gets the package version of symfony
         
	[ GET http://localhost:1111/travis/php-v/symfony/symfony.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  25) PHP version from .travis.yml
       gets the package version of symfony 2.8
         
	[ GET http://localhost:1111/travis/php-v/symfony/symfony/2.8.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  26) PHP version from .travis.yml
       gets the package version of yii
         
	[ GET http://localhost:1111/travis/php-v/yiisoft/yii.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  27) PHP version from .travis.yml
       invalid package name
         
	[ GET http://localhost:1111/travis/php-v/frodo/is-not-a-package.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

paulmelnikow added a commit that referenced this issue Mar 4, 2018
Gratipay is gone :(

For #1359

This failure from https://circleci.com/gh/badges/shields/1411?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

```
  7) Gratipay
       Receiving

	[ GET http://localhost:1111/gratipay/Gratipay.json ]:
     ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /^\$[0-9]+(\.[0-9]{2})?\/week/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)
```
@RedSparr0w
Copy link
Member

SonarQube has started failing again:

  16) SonarQube
       Tech Debt (legacy API unsupported)
         
	[ GET http://localhost:1111/sonar/4.2/http/sonarqube.com/com.github.dannil:scb-java-client/tech_debt.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  17) SonarQube
       Coverage (legacy API unsupported)
         
	[ GET http://localhost:1111/sonar/4.2/http/sonarqube.com/com.github.dannil:scb-java-client/coverage.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

I suspect the repo is no longer valid for those tests.

Also the json badge is behaving strangely for this too:
SonarQube json badge
image

@paulmelnikow
Copy link
Member Author

Another possibility, since these particular tests are about what happens when the wrong API version is configured on the badge, is that the API being out of date is manifesting in a new way. I'm curious which of those it is.

@paulmelnikow
Copy link
Member Author

I expect this failure in NpmLicense is due to #2196:

https://circleci.com/gh/badges/shields/20848

Fixing an issue like this is a great way for a new contributor to jump in. Any thoughts on the best way to track these?

@paulmelnikow
Copy link
Member Author

This does not seem to be the best place to track these issues. I think it may be a good place to coordinate on the larger project, or figure out how to best support new contributors in addressing.

them. Let's default to opening new issues for these failures in the daily tests.

@paulmelnikow paulmelnikow added keep-service-tests-green Related to fixing failing tests of the services service-badge New or updated service badge labels Nov 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
developer-experience Dev tooling, test framework, and CI keep-service-tests-green Related to fixing failing tests of the services service-badge New or updated service badge
Projects
None yet
Development

No branches or pull requests

2 participants