NO-ISSUE: Add filtered controller logs when timeout during installation.

[bkopilov]

During CI runs we got timeout failures without any root cause for failures. The default timeout is 3600 seconds.

We are going to increase total masters and operators installed and probably May hit on timeout and the current timers are not enough.

Added support for filtering last messages from assisted controller log when timeout occured. In case controller logs exists on timeout we extract last error/ warning and report them as part of the raise exception

We expect the see additional info when bubbled up timeout exception and get more details from the raised exception
.

[bkopilov]

@adriengentil , @mburman5 , @AlonaKaplan , @eifrach , @danmanor

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bkopilov
Once this PR has been reviewed and has the lgtm label, please assign avishayt for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

Hi @bkopilov. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[danmanor]

@bkopilov Do we have a way to reproduce the issue and check it works ?

[bkopilov]

@bkopilov Do we have a way to reproduce the issue and check it works ?

Yes , i changed the timeout for 10 minutes .
We have flow -> on timeout exception or any other exception raised ->
Try to get the assisted-controller.log :

• if no catch exception and return []
• if yes extract / filter. in case of error exception should be caught and set to []

[bkopilov]

@bkopilov Do we have a way to reproduce the issue and check it works ?

Yes , i changed the timeout for 10 minutes . We have flow -> on timeout exception or any other exception raised -> Try to get the assisted-controller.log :

• if no catch exception and return []
• if yes extract / filter. in case of error exception should be caught and set to []

Example from Run:

      raise Exception(e.__class__.__name__, e.__dict__)

E Exception: ('TimeoutExpired', {'_timeout_seconds': 60, '_what': "Monitored ['builtin'] operators to be in of the statuses ['available']", 'notes': ["Failed to deploy the following operators ['console', 'cvo']"], 'filter_message': '['time="2024-12-04T10:36:39Z" level=error msg="Failed to get list of nodes from k8s client" func="github.com/openshift/assisted-installer/src/assisted_installer_controller.(*controller).waitAndUpdateNodesStatus" file="/remote-source/app/src/assisted_installer_controller/assisted_installer_controller.go:256" error="the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)" request_id=a1b43260-4693-49f8-9c1f-ec8b1cd08044\n']'})

src/assisted_test_infra/test_infra/helper_classes/cluster.py:706: Exception

[danmanor]

@bkopilov when the PR is ready, lets run the tests

[bkopilov]

@bkopilov when the PR is ready, lets run the tests

Its ready , i will start running full CI with it

[bkopilov]

Hi , Example of a test when the installation failed:
https://ci-jenkins-csb-kniqe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-assisted-installer-virt-tf/21297/

[eifrach]

/retitle NO-ISSUE: Add filtered controller logs when timeout during installation.

[openshift-ci-robot]

@bkopilov: This pull request explicitly references no jira issue.

In response to this:

During CI runs we got timeout failures without any root cause for failures. The default timeout is 3600 seconds.

We are going to increase total masters and operators installed and probably May hit on timeout and the current timers are not enough.

Added support for filtering last messages from assisted controller log when timeout occured. In case controller logs exists on timeout we extract last error/ warning and report them as part of the raise exception

We expect the see additional info when bubbled up timeout exception and get more details from the raised exception
.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bkopilov]

Tested in CI when there is a timeout:
https://ci-jenkins-csb-kniqe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-assisted-installer-virt-tf/21346//testReport/junit/test_lvms_multi/TestLVMSMulti/test_lvms_multi_with_mce_only_52000_20_3_36000_12_/

Exception: ('TimeoutExpired', {'_timeout_seconds': 3600, '_what': "Nodes to be in of the statuses ['installed']", 'filter_message': '[\'time="2024-12-12T14:43:11Z" level=error msg="Failed to get logs from kube-api, reading from file" func=github.com/openshift/assisted-installer/src/common.GetControllerPodLogs file="/remote-source/app/src/common/common.go:116" error="etcdserver: request timed out"\\n\', \'time="2024-12-12T14:42:18Z" level=error msg="Failed to get list of nodes from k8s client" func="github.com/openshift/assisted-installer/src/assisted_installer_controller.(*controller).waitAndUpdateNodesStatus" file="/remote-source/app/src/assisted_installer_controller/assisted_installer_controller.go:256" error="the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)" request_id=56ae929a-e118-454d-860f-08f9ab09f1b9\\n\', \'time="2024-12-12T14:42:17Z" level=error msg="Failed to get list of CSRs." func=github.com/openshift/assisted-installer/src/k8s_client.k8sClient.ListCsrs file="/remote-source/app/src/k8s_client/k8s_client.go:367" error="the server was unable to return a response in the time allotted, but may still be processing the request (get certificatesigningrequests.certificates.k8s.io)"\\n\']'})

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.