E2E test cases are faling

[Amulyam24]

/kind bug
/area provider/ibmcloud

What steps did you take and what happened:
Two failures are encountered

1. Expected machine count doesn't match the created count, machine creation is failing

INFO: Waiting for control plane to be ready
INFO: Waiting for control plane create-workload-cluster-gkyo84/capi-ibmcloud-e2e-7lh1jd-control-plane to be ready (implies underlying nodes to be ready as well)
�[1mSTEP�[0m: Waiting for the control plane to be ready
INFO: Waiting for the machine deployments to be provisioned
�[1mSTEP�[0m: Waiting for the workload nodes to exist
[AfterEach] Workload cluster creation
  /home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:75
�[1mSTEP�[0m: Dumping logs from the "capi-ibmcloud-e2e-7lh1jd" workload cluster
�[1mSTEP�[0m: Dumping all the Cluster API resources in the "create-workload-cluster-gkyo84" namespace
�[1mSTEP�[0m: Deleting all clusters in the create-workload-cluster-gkyo84 namespace
�[1mSTEP�[0m: Deleting cluster capi-ibmcloud-e2e-7lh1jd
INFO: Waiting for the Cluster create-workload-cluster-gkyo84/capi-ibmcloud-e2e-7lh1jd to be deleted
�[1mSTEP�[0m: Waiting for cluster capi-ibmcloud-e2e-7lh1jd to be deleted
�[1mSTEP�[0m: Deleting namespace used for hosting the "create-workload-cluster" test spec
INFO: Deleting namespace create-workload-cluster-gkyo84

�[91m�[1m• Failure [2787.429 seconds]�[0m
Workload cluster creation
�[90m/home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:40�[0m
  Creating a single control-plane cluster
  �[90m/home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:90�[0m
    �[91m�[1mShould create a cluster with 1 worker node and can be scaled [It]�[0m
    �[90m/home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:91�[0m

    �[91mTimed out after 1800.001s.
    Expected
        <int>: 2
    to equal
        <int>: 3�[0m

    /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/[email protected]/framework/machinedeployment_helpers.go:121

2. Unable to connect to the created cluster

INFO: Installing a CNI plugin to the workload cluster
Unable to connect to the server: dial tcp 158.176.141.158:6443: i/o timeout

[AfterEach] Workload cluster creation
  /home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:75
�[1mSTEP�[0m: Dumping logs from the "capi-ibmcloud-e2e-vfi9lm" workload cluster
�[1mSTEP�[0m: Dumping all the Cluster API resources in the "create-workload-cluster-yivmk8" namespace
�[1mSTEP�[0m: Deleting all clusters in the create-workload-cluster-yivmk8 namespace
�[1mSTEP�[0m: Deleting cluster capi-ibmcloud-e2e-vfi9lm
INFO: Waiting for the Cluster create-workload-cluster-yivmk8/capi-ibmcloud-e2e-vfi9lm to be deleted
�[1mSTEP�[0m: Waiting for cluster capi-ibmcloud-e2e-vfi9lm to be deleted
�[1mSTEP�[0m: Deleting namespace used for hosting the "create-workload-cluster" test spec
INFO: Deleting namespace create-workload-cluster-yivmk8

�[91m�[1m• Failure [401.406 seconds]�[0m
Workload cluster creation
�[90m/home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:40�[0m
  Creating a highly available control-plane cluster
  �[90m/home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:136�[0m
    �[91m�[1mShould create a cluster with 3 control-plane nodes and 1 worker node [It]�[0m
    �[90m/home/prow/go/src/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/test/e2e/e2e_test.go:137�[0m

    �[91mUnexpected error:
        <*errors.withStack | 0xc000496930>: {
            error: <*exec.ExitError | 0xc000064240>{
                ProcessState: {
                    pid: 21167,
                    status: 256,
                    rusage: {
                        Utime: {Sec: 0, Usec: 44210},
                        Stime: {Sec: 0, Usec: 22121},
                        Maxrss: 90400,
                        Ixrss: 0,
                        Idrss: 0,
                        Isrss: 0,
                        Minflt: 6204,
                        Majflt: 2,
                        Nswap: 0,
                        Inblock: 216,
                        Oublock: 0,
                        Msgsnd: 0,
                        Msgrcv: 0,
                        Nsignals: 0,
                        Nvcsw: 280,
                        Nivcsw: 25,
                    },
                },
                Stderr: nil,
            },
            stack: [0x15500f5, 0x155061d, 0x16e7bac, 0x1ab5322, 0x1b115d4, 0x14eec9a, 0x14ee665, 0x14edd5b, 0x14f3ae9, 0x14f34c7, 0x1513ca5, 0x15139c5, 0x1513205, 0x15154b2, 0x151e2a5, 0x151e0be, 0x1b0ff09, 0x515582, 0x46b181],
        }
        exit status 1
    occurred�[0m

    /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/[email protected]/framework/clusterctl/clusterctl_helpers.go:300

What did you expect to happen:
E2E test cases should pass

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[Amulyam24]

For the machine creation failing, this error shows up in the logs

E0405 08:43:23.054917       1 ibmpowervsmachine_controller.go:196] controllers/IBMPowerVSMachine "msg"="unable to create instance" "error"="failed to Create PVM Instance :&{0   } (*models.Error) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface" "cluster"="capi-ibmcloud-e2e-7lh1jd" "ibmpowervsmachine"={"Namespace":"create-workload-cluster-gkyo84","Name":"capi-ibmcloud-e2e-7lh1jd-md-0-75fnh"} 

[mkumatag]

For the machine creation failing, this error shows up in the logs

E0405 08:43:23.054917       1 ibmpowervsmachine_controller.go:196] controllers/IBMPowerVSMachine "msg"="unable to create instance" "error"="failed to Create PVM Instance :&{0   } (*models.Error) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface" "cluster"="capi-ibmcloud-e2e-7lh1jd" "ibmpowervsmachine"={"Namespace":"create-workload-cluster-gkyo84","Name":"capi-ibmcloud-e2e-7lh1jd-md-0-75fnh"} 

wondering if we can enable -v=5 or higher number for the CI to dump all the API request and response as well to see what's going on?!

[Amulyam24]

@mkumatag, FYI
Tried a sample deployment on lon04, in additional to the errors seen on the UI -

The host group s922 was not found. Power virtual server instance ibm-powervs-md-0-n2wdg creation has failed.
The requested resource allocations are currently not available in this region. Power virtual server instance ibm-powervs-md-0-g24vq creation has failed.

The API response in logs -

POST /pcloud/v1/cloud-instances/97ff60d4-5b60-4a3d-bb28-34aedc603bf3/pvm-instances HTTP/1.1
Host: lon.power-iaas.cloud.ibm.com
User-Agent: Go-http-client/1.1
Content-Length: 2357
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
Crn: crn:v1:bluemix:public:power-iaas:lon04:a/65b64c1f1c29460e8c2e4bbfbd893c2c:97ff60d4-5b60-4a3d-bb28-34aedc603bf3::
Accept-Encoding: gzip

{"imageID":"4c866e41-ec04-4fdd-bfec-696af00636d9","keyPairName":"amulya-key","memory":4,"networkIDs":null,"networks":[{"networkID":"afd97669-fb9c-4c1c-9104-c40232bfd91d"}],"procType":"shared","processors":0.25,"serverName":"ibm-powervs-md-0-g24vq","sysType":"s922","userData":"IyMgdGVtcGxhdGU6IGppbmphCiNjbG91ZC1jb25maWcKCndyaXRlX2ZpbGVzOgotICAgcGF0aDogL3J1bi9rdWJlYWRtL2t1YmVhZG0tam9pbi1jb25maWcueWFtbAogICAgb3duZXI6IHJvb3Q6cm9vdAogICAgcGVybWlzc2lvbnM6ICcwNjQwJwogICAgY29udGVudDogfAogICAgICAtLS0KICAgICAgYXBpVmVyc2lvbjoga3ViZWFkbS5rOHMuaW8vdjFiZXRhMwogICAgICBkaXNjb3Zlcnk6CiAgICAgICAgYm9vdHN0cmFwVG9rZW46CiAgICAgICAgICBhcGlTZXJ2ZXJFbmRwb2ludDogMTkyLjE2OC4xNTAuMzU6NjQ0MwogICAgICAgICAgY2FDZXJ0SGFzaGVzOgogICAgICAgICAgLSBzaGEyNTY6YTBmMzhkYWM4OWE2NDg3NjkxY2FhYjBlMmFkZTFjMDEyYTU5ZTU4Nzk3M2I5ZWJjODgwZGE3ZTY0MzhhNjY0ZAogICAgICAgICAgdG9rZW46IG14bTBkai55djdpYWk2MDB2bXB6ZjJwCiAgICAgIGtpbmQ6IEpvaW5Db25maWd1cmF0aW9uCiAgICAgIG5vZGVSZWdpc3RyYXRpb246CiAgICAgICAgY3JpU29ja2V0OiAvdmFyL3J1bi9jb250YWluZXJkL2NvbnRhaW5lcmQuc29jawogICAgICAgIGt1YmVsZXRFeHRyYUFyZ3M6CiAgICAgICAgICBjbG91ZC1wcm92aWRlcjogZXh0ZXJuYWwKICAgICAgICAgIGV2aWN0aW9uLWhhcmQ6IG5vZGVmcy5hdmFpbGFibGU8MCUsbm9kZWZzLmlub2Rlc0ZyZWU8MCUsaW1hZ2Vmcy5hdmFpbGFibGU8MCUKICAgICAgICAgIHByb3ZpZGVyLWlkOiBpYm1wb3dlcnZzOi8vaWJtLXBvd2VydnMvJ3t7IHYxLmxvY2FsX2hvc3RuYW1lIH19JwogICAgICAgIG5hbWU6ICd7eyB2MS5sb2NhbF9ob3N0bmFtZSB9fScKICAgICAgICB0YWludHM6IG51bGwKICAgICAgCi0gICBwYXRoOiAvcnVuL2NsdXN0ZXItYXBpL3BsYWNlaG9sZGVyCiAgICBvd25lcjogcm9vdDpyb290CiAgICBwZXJtaXNzaW9uczogJzA2NDAnCiAgICBjb250ZW50OiAiVGhpcyBwbGFjZWhvbGRlciBmaWxlIGlzIHVzZWQgdG8gY3JlYXRlIHRoZSAvcnVuL2NsdXN0ZXItYXBpIHN1YiBkaXJlY3RvcnkgaW4gYSB3YXkgdGhhdCBpcyBjb21wYXRpYmxlIHdpdGggYm90aCBMaW51eCBhbmQgV2luZG93cyAobWtkaXIgLXAgL3J1bi9jbHVzdGVyLWFwaSBkb2VzIG5vdCB3b3JrIHdpdGggV2luZG93cykiCnJ1bmNtZDoKICAtICJob3N0bmFtZSBcInt7IHYxLmxvY2FsX2hvc3RuYW1lIH19XCIiCiAgLSAiZWNobyBcIjo6MSAgICAgICAgIGlwdjYtbG9jYWxob3N0IGlwdjYtbG9vcGJhY2tcIiA+L2V0Yy9ob3N0cyIKICAtICJlY2hvIFwiMTI3LjAuMC4xICAgbG9jYWxob3N0XCIgPj4vZXRjL2hvc3RzIgogIC0gImVjaG8gXCIxMjcuMC4wLjEgICB7eyB2MS5sb2NhbF9ob3N0bmFtZSB9fVwiID4+L2V0Yy9ob3N0cyIKICAtICJlY2hvIFwie3sgdjEubG9jYWxfaG9zdG5hbWUgfX1cIiA+L2V0Yy9ob3N0bmFtZSIKICAtIGt1YmVhZG0gam9pbiAtLWNvbmZpZyAvcnVuL2t1YmVhZG0va3ViZWFkbS1qb2luLWNvbmZpZy55YW1sICAmJiBlY2hvIHN1Y2Nlc3MgPiAvcnVuL2NsdXN0ZXItYXBpL2Jvb3RzdHJhcC1zdWNjZXNzLmNvbXBsZXRlCg==","volumeIDs":null}

HTTP/1.1 504 Gateway Time-out
Transfer-Encoding: chunked
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Cf-Ray: 6f81f7da399c0bd4-DFW
Connection: keep-alive
Content-Type: text/html; charset=UTF-8
Date: Thu, 07 Apr 2022 10:13:31 GMT
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Referrer-Policy: same-origin
Server: cloudflare
Set-Cookie: cf_ob_info=504:6f81f7da399c0bd4:DFW; path=/; expires=Thu, 07-Apr-22 10:14:01 GMT
Set-Cookie: cf_use_ob=443; path=/; expires=Thu, 07-Apr-22 10:14:01 GMT
X-Frame-Options: SAMEORIGIN


E0407 10:13:31.423780       9 ibmpowervsmachine_controller.go:196] controllers/IBMPowerVSMachine "msg"="unable to create instance" "error"="failed to Create PVM Instance :&{0   } (*models.Error) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface" "cluster"="ibm-powervs" "ibmpowervsmachine"={"Namespace":"default","Name":"ibm-powervs-md-0-g24vq"} 

For the machine creation failing, this error shows up in the logs

E0405 08:43:23.054917       1 ibmpowervsmachine_controller.go:196] controllers/IBMPowerVSMachine "msg"="unable to create instance" "error"="failed to Create PVM Instance :&{0   } (*models.Error) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface" "cluster"="capi-ibmcloud-e2e-7lh1jd" "ibmpowervsmachine"={"Namespace":"create-workload-cluster-gkyo84","Name":"capi-ibmcloud-e2e-7lh1jd-md-0-75fnh"} 

I guess this error is a distraction and occurs due to incompatibility with the response being returned in html format

[mkumatag]

E0405 08:43:23.054917       1 ibmpowervsmachine_controller.go:196] controllers/IBMPowerVSMachine "msg"="unable to create instance" "error"="failed to Create PVM Instance :&{0   } (*models.Error) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface" "cluster"="capi-ibmcloud-e2e-7lh1jd" "ibmpowervsmachine"={"Namespace":"create-workload-cluster-gkyo84","Name":"capi-ibmcloud-e2e-7lh1jd-md-0-75fnh"} 

I guess this error is a distraction and occurs due to incompatibility with the response being returned in html format

This sort of errors needs to be handled in the sdk I feel @yussufsh

[mkumatag]

@Amulyam24 lets find a way to enable the debug logs for the e2e tests we run periodically, I badly want to see all the API reqs and responses for every failures we are hitting.

[mkumatag]

/assign @Amulyam24