[flake]FederatedResourceQuota auto-provision testing

[RainbowMango]

Which jobs are flaking:

FederatedResourceQuota auto-provision testing
https://github.com/karmada-io/karmada/runs/6517485253?check_suite_focus=true

Which test(s) are flaking:

 [FAILED] [310.565 seconds]
FederatedResourceQuota auto-provision testing
/home/runner/work/karmada/karmada/test/e2e/federatedresourcequota_test.go:23
  delete a federatedResourceQuota
  /home/runner/work/karmada/karmada/test/e2e/federatedresourcequota_test.go:44
    [It] federatedResourceQuota should be removed from member clusters
    /home/runner/work/karmada/karmada/test/e2e/federatedresourcequota_test.go:50

  Begin Captured StdOut/StdErr Output >>
    I0520 01:32:59.907410   40980 resourcequota.go:28] Waiting for resourceQuota(karmadatest-dwx/frq-rms) present on cluster(member1)
    I0520 01:33:04.936870   40980 resourcequota.go:28] Waiting for resourceQuota(karmadatest-dwx/frq-rms) present on cluster(member2)
    I0520 01:33:04.951461   40980 resourcequota.go:28] Waiting for resourceQuota(karmadatest-dwx/frq-rms) present on cluster(member3)
    I0520 01:33:05.184389   40980 resourcequota.go:50] Waiting for resourceQuota(karmadatest-dwx/frq-rms) disappear on cluster(member1)
    I0520 01:33:10.347692   40980 resourcequota.go:50] Waiting for resourceQuota(karmadatest-dwx/frq-rms) disappear on cluster(member2)
    I0520 01:33:10.369059   40980 resourcequota.go:50] Waiting for resourceQuota(karmadatest-dwx/frq-rms) disappear on cluster(member3)
  << End Captured StdOut/StdErr Output

  Begin Captured GinkgoWriter Output >>
    STEP: Creating FederatedResourceQuota(karmadatest-dwx/frq-rms) 05/20/22 01:32:59.81
    STEP: Check if resourceQuota(karmadatest-dwx/frq-rms) present on member clusters 05/20/22 01:32:59.906
    STEP: Removing FederatedResourceQuota(karmadatest-dwx/frq-rms) 05/20/22 01:33:04.957
    STEP: Check if resourceQuota(karmadatest-dwx/frq-rms) disappear on member clusters 05/20/22 01:33:05.183
  << End Captured GinkgoWriter Output

  Timed out after 300.001s.
  Expected
      <bool>: false
  to equal
      <bool>: true
  In [It] at: /home/runner/work/karmada/karmada/test/e2e/framework/resourcequota.go:54

  Full Stack Trace
    github.com/karmada-io/karmada/test/e2e/framework.WaitResourceQuotaDisappearOnCluster({0xc0005f16b9, 0x7}, {0xc00070c6c1, 0xc0007c7380}, {0xc0005aacc9, 0x7})
    	/home/runner/work/karmada/karmada/test/e2e/framework/resourcequota.go:54 +0x4a2
    github.com/karmada-io/karmada/test/e2e/framework.WaitResourceQuotaDisappearOnClusters.func1()
    	/home/runner/work/karmada/karmada/test/e2e/framework/resourcequota.go:40 +0xbd
    github.com/karmada-io/karmada/test/e2e/framework.WaitResourceQuotaDisappearOnClusters({0xc0001417c0, 0x3, 0x4}, {0xc00070c6c1, 0xf}, {0xc0005aacc9, 0x7})
    	/home/runner/work/karmada/karmada/test/e2e/framework/resourcequota.go:38 +0x19b
    github.com/karmada-io/karmada/test/e2e.glob..func5.3.2()
    	/home/runner/work/karmada/karmada/test/e2e/federatedresourcequota_test.go:52 +0x13d

Reason for failure:

Anything else we need to know:

Download the full log from : https://github.com/karmada-io/karmada/suites/6584849366/logs?attempt=1
View the full log online: https://github.com/karmada-io/karmada/commit/8ac5c676115d0f76e162dff9f005d35c3aca9786/checks/6517485253/logs

[XiShanYongYe-Chang]

Currently, the logs of the member3 cluster are not collected. Therefore, We can't determine if the resources on the member3 cluster are being handled correctly. So we need to collect logs on the pull mode cluster.

[XiShanYongYe-Chang]

During the test, we should output more detailed error info to help us determine whether the abnormal running of the clusterClient causes the fault:

karmada/test/e2e/framework/resourcequota.go

Lines 52 to 53 in 7e6c38a

	_, err := clusterClient.CoreV1().ResourceQuotas(namespace).Get(context.TODO(), name, metav1.GetOptions{})
	return apierrors.IsNotFound(err)

[XiShanYongYe-Chang]

/assign

[RainbowMango]

Another failure:
https://github.com/karmada-io/karmada/runs/6573754591?check_suite_focus=true

[XiShanYongYe-Chang]

Another failure: https://github.com/karmada-io/karmada/runs/6573754591?check_suite_focus=true

Timeout for waiting unjoin cluster.

[XiShanYongYe-Chang]

Another same failure for unjoin cluster failed: https://github.com/karmada-io/karmada/runs/6586927906?check_suite_focus=true

The failure reason is different from the original issue.

    I0525 07:00:08.044723   43590 unjoin.go:291] Waiting for the cluster object member-e2e-8j8 to be deleted
...
    E0525 07:00:08.067729   43590 unjoin.go:166] Failed to delete cluster object. cluster name: member-e2e-8j8, error: timed out waiting for the condition

  Begin Captured GinkgoWriter Output >>
    STEP: Creating FederatedResourceQuota(karmadatest-44v/frq-wrg) 05/25/22 06:57:19.932
    STEP: Creating cluster: member-e2e-8j8 05/25/22 06:57:20.202
    STEP: Joinning cluster: member-e2e-8j8 05/25/22 06:59:00.363
    STEP: waiting federatedResourceQuota(karmadatest-44v/frq-wrg) present on cluster: member-e2e-8j8 05/25/22 06:59:02.784
    STEP: Unjoinning cluster: member-e2e-8j8 05/25/22 06:59:07.955
    STEP: Deleting clusters: member-e2e-8j8 05/25/22 07:00:08.07
    STEP: Removing FederatedResourceQuota(karmadatest-44v/frq-wrg) 05/25/22 07:00:09.88
  << End Captured GinkgoWriter Output

  Unexpected error:
      <*errors.errorString | 0xc0003e6d70>: {
          s: "timed out waiting for the condition",
      }
      timed out waiting for the condition
  occurred
  In [AfterEach] at: /home/runner/work/karmada/karmada/test/e2e/federatedresourcequota_test.go:96

  Full Stack Trace
    github.com/karmada-io/karmada/test/e2e.glob..func5.4.4.1()
    	/home/runner/work/karmada/karmada/test/e2e/federatedresourcequota_test.go:96 +0x226
    github.com/karmada-io/karmada/test/e2e.glob..func5.4.4()
    	/home/runner/work/karmada/karmada/test/e2e/federatedresourcequota_test.go:83 +0xf0

[XiShanYongYe-Chang]

Open a new issue #1882 to record the timeout failure for waiting to unjoin the cluster.

[XiShanYongYe-Chang]

This problem is a random occurrence. I added the related log info and collected the karmada-agent logs. Close this issue first.

/close

[karmada-bot]

@XiShanYongYe-Chang: Closing this issue.

In response to this:

This problem is a random occurrence. I added the related log info and collected the karmada-agent logs. Close this issue first.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.