Operator-sdk scorecard basic spec check occasionally produces timeout with no logs #5452

tkrishtop · 2021-12-13T09:41:57Z

Bug Report

operator-sdk v.1.14.0 
scorecard-test v1.14.0

I'm from the DCI team. As a part of our daily cert suite, we regularly run operator-sdk scorecard --selector=test=basic-check-spec-test for two operators: simple-demo-operator and testpmd-operator, 10 tests per day.

Normally, basic-check-spec-test should be green for both. But it fails occasionally in 20% of cases for both operators in a row with timeout error error running tests context deadline exceeded. To increase timeout with wait-time up to 300s doesn't help. Also, the test is always failing for both operators in a row and it looks like a 10-20 min of some internal API.

To have more information, could you please add more information in the logs about where did exactly the timeout happen?

What did you do?

operator_sdk scorecard \
--output json \
--selector=test=basic-check-spec-test \
--kubeconfig {{ kubeconfig_path }} \
--namespace scorecard-testing \
--service-account default \
--config {{ scorecard_config_path }} \
--verbose \
--wait-time 300s \
{{ scorecard_operator_dir }}

What did you expect to see?

The results of basic-check-spec-test should be stable. In the case of timeout, it would be nice to have logs to identify what is the reason for this timeout.

What did you see instead? Under which circumstances?

Timeout in 20% of cases error running tests context deadline exceeded with no detailed logs.

Environment

Operator type:

-
  name: "testpmd-operator"
  version: "v0.2.9"
  image: "quay.io/rh-nfv-int/testpmd-operator-bundle@sha256:5e28f883faacefa847104ebba1a1a22ee897b7576f0af6b8253c68b5c8f42815"
  index_image: "quay.io/tkrishtop/index-testpmd-operator-bundle:v0.2.9"
-
  name: "simple-demo-operator"
  version: "v0.0.3"
  image: "quay.io/opdev/simple-demo-operator-bundle@sha256:eff7f86a54ef2a340dbf739ef955ab50397bef70f26147ed999e989cfc116b79"
  index_image: "quay.io/opdev/simple-demo-operator-catalog:v0.0.3"

Kubernetes cluster type:

Happens randomly for the latest stable OCP 4.7, OCP 4.8, OCP 4.9, OCP 4.10

$ operator-sdk version

operator-sdk v.1.14.0 
scorecard-test v1.14.0

Possible Solution

It would be nice to have more detailed logs to identify what is the reason for this timeout.

The text was updated successfully, but these errors were encountered:

porridge · 2022-04-06T05:00:20Z

FTR, I encountered a failure recently with verbose logging turned on. AFAICT the interesting part (i.e. what it is that's actually timing out) is not getting logged, but posting here in case it's useful to someone investigating this.

/go/src/github.com/stackrox/rox/operator//bin/operator-sdk-1.14.0 scorecard --storage-image="quay.io/rhacs-eng/mirror-busybox:@sha256:fe9664ceda9c13b7fd65f5bc3c0bca5060de3a66fedc39d33926b573599da3b7" --wait-time="40s" --verbose quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404
DEBU[0000] Debug logging is set                         
DEBU[0000] resolving                                     host=quay.io
DEBU[0000] do request                                    host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/manifests/v3.69.0-nightly-20220404"
DEBU[0000] fetch response received                       host=quay.io response.header.connection=keep-alive response.header.content-length=112 response.header.content-type=application/json response.header.date="Mon, 04 Apr 2022 05:58:50 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.14.1 response.header.www-authenticate="Bearer realm=\"https://quay.io/v2/auth\",service=\"quay.io\",scope=\"repository:rhacs-eng/stackrox-operator-bundle:pull\"" response.status="401 UNAUTHORIZED" url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/manifests/v3.69.0-nightly-20220404"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://quay.io/v2/auth\",service=\"quay.io\",scope=\"repository:rhacs-eng/stackrox-operator-bundle:pull\"" host=quay.io
DEBU[0000] do request                                    host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/manifests/v3.69.0-nightly-20220404"
DEBU[0000] fetch response received                       host=quay.io response.header.connection=keep-alive response.header.content-length=940 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 04 Apr 2022 05:58:50 GMT" response.header.docker-content-digest="sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/manifests/v3.69.0-nightly-20220404"
DEBU[0000] resolved                                      desc.digest="sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12" host=quay.io
DEBU[0000] resolved name: quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] fetched                                       bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" digest="sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12" dir=bundle-421604376
DEBU[0000] {application/vnd.docker.distribution.manifest.v2+json sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12 940 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] fetch                                         digest="sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12" mediatype=application/vnd.docker.distribution.manifest.v2+json size=940
DEBU[0000] do request                                    digest="sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=opm/alpha request.method=GET size=940 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/manifests/sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12"
DEBU[0000] fetch response received                       digest="sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.connection=keep-alive response.header.content-length=940 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 04 Apr 2022 05:58:50 GMT" response.header.docker-content-digest="sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=940 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/manifests/sha256:1f8462f55f91968c31cf86d54b6e9c2f8ef2e8144f108d095aaeb0a0929e9d12"
DEBU[0000] fetched                                       bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" digest="sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8" dir=bundle-421604376
DEBU[0000] {application/vnd.docker.image.rootfs.diff.tar.gzip sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8 480 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] fetch                                         digest="sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=480
DEBU[0000] fetched                                       bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" digest="sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258" dir=bundle-421604376
DEBU[0000] fetched                                       bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" digest="sha256:85dc11d7744b426c884c46a03f87365e54ed43477a62dbcc6cb2737616e10e7b" dir=bundle-421604376
DEBU[0000] {application/vnd.docker.image.rootfs.diff.tar.gzip sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258 29090 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] {application/vnd.docker.container.image.v1+json sha256:85dc11d7744b426c884c46a03f87365e54ed43477a62dbcc6cb2737616e10e7b 5623 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] fetch                                         digest="sha256:85dc11d7744b426c884c46a03f87365e54ed43477a62dbcc6cb2737616e10e7b" mediatype=application/vnd.docker.container.image.v1+json size=5623
DEBU[0000] fetch                                         digest="sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=29090
DEBU[0000] fetched                                       bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" digest="sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097" dir=bundle-421604376
DEBU[0000] {application/vnd.docker.image.rootfs.diff.tar.gzip sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097 406 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] fetch                                         digest="sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=406
DEBU[0000] do request                                    digest="sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=opm/alpha request.method=GET size=480 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8"
DEBU[0000] do request                                    digest="sha256:85dc11d7744b426c884c46a03f87365e54ed43477a62dbcc6cb2737616e10e7b" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=opm/alpha request.method=GET size=5623 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:85dc11d7744b426c884c46a03f87365e54ed43477a62dbcc6cb2737616e10e7b"
DEBU[0000] do request                                    digest="sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=opm/alpha request.method=GET size=29090 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258"
DEBU[0000] do request                                    digest="sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=opm/alpha request.method=GET size=406 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097"
DEBU[0000] fetch response received                       digest="sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.content-length=480 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Apr 2022 05:58:51 GMT" response.header.etag="\"51f66689c707be0494d193fc7650a5f9-1\"" response.header.last-modified="Mon, 04 Apr 2022 05:28:28 GMT" response.header.server=AmazonS3 response.header.x-amz-id-2="3dW6VmlVBdHl6fMvifIGpq0dWEcd8kDG9cpMSH9+QJzBpTB2Bd9QMnX5lAqBktyjbZluGbwpgNo=" response.header.x-amz-replication-status=COMPLETED response.header.x-amz-request-id=JKTGPACKW93BBZ6C response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=hawtuRC76z9_U20jVHGUYfpHW8MdEeIH response.status="200 OK" size=480 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8"
DEBU[0000] fetch response received                       digest="sha256:85dc11d7744b426c884c46a03f87365e54ed43477a62dbcc6cb2737616e10e7b" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.content-length=5623 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Apr 2022 05:58:51 GMT" response.header.etag="\"1b7f8046facca76d808acb9350446a89-1\"" response.header.last-modified="Mon, 04 Apr 2022 05:28:29 GMT" response.header.server=AmazonS3 response.header.x-amz-id-2="q4gHx9LgX6QvEl6RM0H2DKjDo6z+cIYIySZnWBvbOMaDDbARUuM0d74aYbfgE7yri71i1WX71RI=" response.header.x-amz-replication-status=COMPLETED response.header.x-amz-request-id=JKTNA878PQ0CN6KH response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=xY5B4I9EAaNpprD1lCoha71KhEvvS.u_ response.status="200 OK" size=5623 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:85dc11d7744b426c884c46a03f87365e54ed43477a62dbcc6cb2737616e10e7b"
DEBU[0000] fetch response received                       digest="sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.content-length=406 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Apr 2022 05:58:51 GMT" response.header.etag="\"b041299f20acd151b6be2316d966ae21-1\"" response.header.last-modified="Mon, 04 Apr 2022 05:28:28 GMT" response.header.server=AmazonS3 response.header.x-amz-id-2="R9MCOj3SrFzgDT5d7Snp0WlhgkPky19ior+lpfV7KVbpjq0/Vcfwe7LnJBS3qaSmfbLXW9JtnCk=" response.header.x-amz-replication-status=COMPLETED response.header.x-amz-request-id=JKTTWG8J16AQG14R response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=i7UK3EsZch9IgD6o0OGoB5bMxBwlu7hr response.status="200 OK" size=406 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097"
DEBU[0000] fetch response received                       digest="sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.content-length=29090 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Apr 2022 05:58:51 GMT" response.header.etag="\"a78a659753af9dcca852fab61394e9ba-1\"" response.header.last-modified="Mon, 04 Apr 2022 05:28:28 GMT" response.header.server=AmazonS3 response.header.x-amz-id-2="NU0J2CLLlW4jq5J1tarRPv4iBDw1CfMMl6FlbCRq75SQz3uL/OkXX4rJJO+XDfQEIjB4SASaZaM=" response.header.x-amz-replication-status=COMPLETED response.header.x-amz-request-id=JKTNTDYNJ7QT3XT9 response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=XqQhNIlRTb3aOb8QsMDVQtwbiAg3buHp response.status="200 OK" size=29090 url="https://quay.io/v2/rhacs-eng/stackrox-operator-bundle/blobs/sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258"
DEBU[0000] unpacking layer: {application/vnd.docker.image.rootfs.diff.tar.gzip sha256:7e654a7e8519f043e19d3f91b5709b424dc06c0c7787ee8cfd4a0eb14ad63258 29090 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] unpigz not found, falling back to go gzip     error="exec: \"unpigz\": executable file not found in $PATH"
DEBU[0000] unpacking layer: {application/vnd.docker.image.rootfs.diff.tar.gzip sha256:80583ed86a84ec80184a5bb7917a04b6d568b5ff1464f202986f7629b84c0097 406 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
DEBU[0000] unpacking layer: {application/vnd.docker.image.rootfs.diff.tar.gzip sha256:82023726a6e890b5b0db920e4eea2841c8557f931994c36a39b1b5a4b76a9dd8 480 [] map[] <nil>}  bundle="quay.io/rhacs-eng/stackrox-operator-bundle:v3.69.0-nightly-20220404" dir=bundle-421604376
Error: error running tests context deadline exceeded
Usage:
  operator-sdk scorecard [flags]

Flags:
  -c, --config string            path to scorecard config file
  -h, --help                     help for scorecard
      --kubeconfig string        kubeconfig path
  -L, --list                     Option to enable listing which tests are run
  -n, --namespace string         namespace to run the test images in
  -o, --output string            Output format for results. Valid values: text, json, xunit (default "text")
  -l, --selector string          label selector to determine which tests are run
  -s, --service-account string   Service account to use for tests (default "default")
  -x, --skip-cleanup             Disable resource cleanup after tests are run
  -b, --storage-image string     Storage image to be used by the Scorecard pod (default "docker.io/library/busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af")
  -t, --test-output string       Test output directory. (default "test-output")
  -u, --untar-image string       Untar image to be used by the Scorecard pod (default "registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7")
  -w, --wait-time duration       seconds to wait for tests to complete. Example: 35s (default 30s)

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution
      --verbose           Enable verbose logging

FATA[0040] error running tests context deadline exceeded 
make: *** [Makefile:401: bundle-test-image] Error 1

openshift-bot · 2022-07-05T07:03:55Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

tkrishtop · 2022-07-05T12:47:36Z

/remove-lifecycle stale

openshift-bot · 2022-10-04T01:01:06Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

porridge · 2022-10-04T05:18:46Z

/remove-lifecycle stale

…

On Tue, 4 Oct 2022 at 03:01, OpenShift Bot ***@***.***> wrote: Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen. If this issue is safe to close now please do so with /close. /lifecycle stale — Reply to this email directly, view it on GitHub <#5452 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADXPTAQ6KM5PFD6GPDGGULWBN6V7ANCNFSM5J5U4XVQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

tkrishtop · 2022-11-22T08:14:20Z

Hello team, that's an old issue but we keep seeing it regularly while running Preflight which currently runs operator-sdk 1.22.2.

Here are the logs from preflight.log (BasicSpecCheck.json is not created at all):

time="2022-11-21T02:42:04Z" level=debug msg="running check: ScorecardBasicSpecCheck"
time="2022-11-21T02:42:04Z" level=trace msg="Running operator-sdk scorecard check for quay.io/rh-nfv-int/testpmd-operator-bundle@sha256:5e28f883faacefa847104ebba1a1a22ee897b7576f0af6b8253c68b5c8f42815"
time="2022-11-21T02:42:04Z" level=trace msg="--selector=[test=basic-check-spec-test]"
time="2022-11-21T02:42:04Z" level=debug msg="Using quay.io/operator-framework/scorecard-test@sha256:b939bdf25de65d0c4256ac799b02a58335b705e24cbdd66d61e344982eb6156d as the scorecard test image"
time="2022-11-21T02:42:04Z" level=trace msg="running scorecard with the following invocation[operator-sdk scorecard --output json --selector=test=basic-check-spec-test --kubeconfig /kubeconfig --wait-time 240s --namespace preflight-testing --service-account default --config /tmp/scorecard-test-config-4476853.yaml --verbose /tmp/preflight-3276679150/fs]"
time="2022-11-21T02:46:04Z" level=debug msg="operator-sdk scorecard failed to run properly."
time="2022-11-21T02:46:04Z" level=debug msg="stderr: time=\"2022-11-21T02:42:04Z
\" level=debug msg=\"Debug logging is set
\"\nError: error running tests context deadline exceeded
\nUsage:\n  operator-sdk scorecard [flags]\n\n
Flags:\n  -c, --config string            path to scorecard config file\n  -h, --help                     help for scorecard\n      --kubeconfig string        kubeconfig path\n  -L, --list                     
Option to enable listing which tests are run\n  -n, --namespace string         namespace to run the test images in\n  -o, --output string            
Output format for results. Valid values: text, json, xunit (default \"text\")\n  -l, --selector string          label selector to determine which tests are run\n  -s, --service-account string   Service account to use for tests (default \"default\")\n  -x, --skip-cleanup             
Disable resource cleanup after tests are run\n  -b, --storage-image string     Storage image to be used by the Scorecard pod 
(default \"docker.io/library/busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af\")\n  -t, --test-output string       
Test output directory. (default \"test-output\")\n  -u, --untar-image string      
 Untar image to be used by the Scorecard pod (default \"registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7\")\n  -w, --wait-time duration       seconds to wait for tests to complete. 
Example: 35s (default 30s)\n\n
Global Flags:\n      --plugins strings   plugin keys to be used for this subcommand execution\n      --verbose           Enable verbose logging\n\ntime=\"2022-11-21T02:46:04Z\" level=fatal msg=\"error running tests context deadline exceeded\"\n"
time="2022-11-21T02:46:04Z" level=info msg="check completed: ScorecardBasicSpecCheck" err="failed to run operator-sdk scorecard: exit status 1" result=ERROR

Now we have more logs (thank you team!) and see that the timeout happens here: Untar image to be used by the Scorecard pod

operator-sdk/internal/cmd/operator-sdk/scorecard/cmd.go

Lines 96 to 98 in 5d541d0

    
           	"Untar image to be used by the Scorecard pod") 
        
           scorecardCmd.Flags().StringVarP(&c.testOutput, "test-output", "t", "test-output", 
        
           	"Test output directory.")

@asmacdo @theishshah do you have any clues?

cc: @acornett21

tkrishtop · 2022-11-22T10:44:56Z

Upd. In half of such situations, the scorecard-test pod is stuck in the namespace, failing to terminate. This last bad thing should be easy to fix, whatever the reason for the initial issue.

oc get all -n preflight-testing

NAME                      READY   STATUS        RESTARTS   AGE
pod/scorecard-test-tjbf   0/1     Terminating   0          14m

openshift-bot · 2023-02-21T01:01:08Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

tkrishtop · 2023-02-21T08:10:48Z

/remove-lifecycle stale

openshift-bot · 2023-03-23T08:30:53Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

porridge · 2023-03-30T07:11:08Z

/remove-lifecycle stale

acornett21 · 2023-03-30T16:39:02Z

@tkrishtop Recently operator-sdk switched to using their own images for this cmd. So instead of using an image from docker.io and an image from registry.access.redhat.com both images come from quay. This change is in 1.28.0 you can see the PR here. Have you upgraded to 1.28.0 yet?

tkrishtop · 2023-03-30T17:04:52Z

Hi @acornett21, thank you for the information! Typically, we run the scorecard-sdk from preflight, which currently uses version 1.26.0. However, I do have the option to run the standalone scorecard-sdk, so perhaps I'll activate it for our daily runs using version 1.28.0.

openshift-bot · 2023-06-29T01:00:53Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2023-07-29T08:30:37Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

porridge · 2023-08-01T05:23:36Z

/remove-lifecycle rotten

openshift-bot · 2023-10-30T09:00:31Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

porridge · 2023-10-30T09:26:37Z

/remove-lifecycle stale

openshift-bot · 2024-01-29T01:00:21Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

porridge · 2024-01-29T05:35:54Z

/remove-lifecycle stale

openshift-bot · 2024-04-28T09:00:13Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

porridge · 2024-04-29T05:16:44Z

/remove-lifecycle stale

openshift-bot · 2024-07-28T09:00:37Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

porridge · 2024-08-05T06:30:05Z

/remove-lifecycle stale

openshift-bot · 2024-11-03T09:00:28Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

porridge · 2024-11-04T07:32:54Z

/remove-lifecycle stale

asmacdo assigned theishshah Dec 13, 2021

asmacdo added this to the Backlog milestone Dec 13, 2021

asmacdo added the scorecard Issue relates to the scorecard subcomponent label Dec 13, 2021

asmacdo modified the milestones: Backlog, v1.17.0 Dec 13, 2021

tkrishtop mentioned this issue Dec 20, 2021

File scorecard_BasicSpecCheck.json is empty if operator-sdk produces timeout redhat-openshift-ecosystem/openshift-preflight#355

Closed

theishshah modified the milestones: v1.17.0, v1.18.0 Feb 2, 2022

theishshah added the needs discussion label Feb 2, 2022

asmacdo modified the milestones: v1.18.0, Backlog Feb 2, 2022

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 5, 2022

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 5, 2022

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 4, 2022

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 4, 2022

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 21, 2023

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 23, 2023

porridge mentioned this issue Mar 30, 2023

Basic scorecard test not working #5232

Open

ebaron mentioned this issue Apr 14, 2023

test(scorecard): add option to install cert-manager on OpenShift cryostatio/cryostat-operator#550

Merged

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 29, 2023

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 29, 2023

openshift-ci bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 1, 2023

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 30, 2023

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 30, 2023

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 29, 2024

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 29, 2024

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 28, 2024

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 29, 2024

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 28, 2024

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 5, 2024

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 3, 2024

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Operator-sdk scorecard basic spec check occasionally produces timeout with no logs #5452

Operator-sdk scorecard basic spec check occasionally produces timeout with no logs #5452

tkrishtop commented Dec 13, 2021

porridge commented Apr 6, 2022

openshift-bot commented Jul 5, 2022

tkrishtop commented Jul 5, 2022

openshift-bot commented Oct 4, 2022

porridge commented Oct 4, 2022 via email

tkrishtop commented Nov 22, 2022 •

edited

Loading

tkrishtop commented Nov 22, 2022

openshift-bot commented Feb 21, 2023

tkrishtop commented Feb 21, 2023 •

edited

Loading

openshift-bot commented Mar 23, 2023

porridge commented Mar 30, 2023

acornett21 commented Mar 30, 2023

tkrishtop commented Mar 30, 2023

openshift-bot commented Jun 29, 2023

openshift-bot commented Jul 29, 2023

porridge commented Aug 1, 2023

openshift-bot commented Oct 30, 2023

porridge commented Oct 30, 2023

openshift-bot commented Jan 29, 2024

porridge commented Jan 29, 2024 via email

openshift-bot commented Apr 28, 2024

porridge commented Apr 29, 2024

openshift-bot commented Jul 28, 2024

porridge commented Aug 5, 2024

openshift-bot commented Nov 3, 2024

porridge commented Nov 4, 2024

Operator-sdk scorecard basic spec check occasionally produces timeout with no logs #5452

Operator-sdk scorecard basic spec check occasionally produces timeout with no logs #5452

Comments

tkrishtop commented Dec 13, 2021

Bug Report

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

Possible Solution

porridge commented Apr 6, 2022

openshift-bot commented Jul 5, 2022

tkrishtop commented Jul 5, 2022

openshift-bot commented Oct 4, 2022

porridge commented Oct 4, 2022 via email

tkrishtop commented Nov 22, 2022 • edited Loading

tkrishtop commented Nov 22, 2022

openshift-bot commented Feb 21, 2023

tkrishtop commented Feb 21, 2023 • edited Loading

openshift-bot commented Mar 23, 2023

porridge commented Mar 30, 2023

acornett21 commented Mar 30, 2023

tkrishtop commented Mar 30, 2023

openshift-bot commented Jun 29, 2023

openshift-bot commented Jul 29, 2023

porridge commented Aug 1, 2023

openshift-bot commented Oct 30, 2023

porridge commented Oct 30, 2023

openshift-bot commented Jan 29, 2024

porridge commented Jan 29, 2024 via email

openshift-bot commented Apr 28, 2024

porridge commented Apr 29, 2024

openshift-bot commented Jul 28, 2024

porridge commented Aug 5, 2024

openshift-bot commented Nov 3, 2024

porridge commented Nov 4, 2024

tkrishtop commented Nov 22, 2022 •

edited

Loading

tkrishtop commented Feb 21, 2023 •

edited

Loading