Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bastionhosts_test.go tests give false results when running tests in parallel #1753

Closed
karuppiah7890 opened this issue Oct 2, 2021 · 3 comments · Fixed by #1754
Closed

bastionhosts_test.go tests give false results when running tests in parallel #1753

karuppiah7890 opened this issue Oct 2, 2021 · 3 comments · Fixed by #1754
Assignees
@karuppiah7890

Comments

@karuppiah7890
Copy link
Contributor

For example, if you run the below command to run TestReconcileBastionHosts test on commit df75af2 which is the latest commit on main branch as of this writing

go test -run ^TestReconcileBastionHosts$ sigs.k8s.io/cluster-api-provider-azure/azure/services/bastionhosts

You will get an output like this

$ go test -run ^TestReconcileBastionHosts$ sigs.k8s.io/cluster-api-provider-azure/azure/services/bastionhosts
ok  	sigs.k8s.io/cluster-api-provider-azure/azure/services/bastionhosts	0.424s

Now remove the t.Parallel in the TestReconcileBastionHosts test's sub tests, that is this line -

cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go

Line 248 in df75af2

t.Parallel()

Now run the same command again to run TestReconcileBastionHosts test

go test -run ^TestReconcileBastionHosts$ sigs.k8s.io/cluster-api-provider-azure/azure/services/bastionhosts

And you will get an output like this

$ go test -run ^TestReconcileBastionHosts$ sigs.k8s.io/cluster-api-provider-azure/azure/services/bastionhosts
I1002 10:22:26.339869   30265 azurebastion.go:34]  "msg"="getting azure bastion public IP"  "publicIP"="my-publicip"
I1002 10:22:26.340267   30265 azurebastion.go:34]  "msg"="getting azure bastion public IP"  "publicIP"="my-publicip"
I1002 10:22:26.340450   30265 azurebastion.go:34]  "msg"="getting azure bastion public IP"  "publicIP"="my-publicip"
I1002 10:22:26.340638   30265 azurebastion.go:34]  "msg"="getting azure bastion public IP"  "publicIP"="my-publicip"
I1002 10:22:26.340977   30265 azurebastion.go:34]  "msg"="getting azure bastion public IP"  "publicIP"="my-publicip"
I1002 10:22:26.341177   30265 azurebastion.go:34]  "msg"="getting azure bastion public IP"  "publicIP"="my-publicip"
I1002 10:22:26.341352   30265 azurebastion.go:34]  "msg"="getting azure bastion public IP"  "publicIP"="my-publicip"
I1002 10:22:26.341483   30265 azurebastion.go:40]  "msg"="getting azure bastion subnet"  "subnet"={"name":"my-subnet","securityGroup":{},"routeTable":{},"natGateway":{"ip":{"name":""}}}
I1002 10:22:26.341503   30265 azurebastion.go:46]  "msg"="creating bastion host"  "bastion"="my-bastion"
--- FAIL: TestReconcileBastionHosts (0.00s)
    --- FAIL: TestReconcileBastionHosts/fail_to_get_subnets (0.00s)
        azurebastion.go:35: Unexpected call to *mock_publicips.MockClient.Get([context.TODO.WithValue(type tele.CorrIDKey, val <not Stringer>).WithValue(type trace.traceContextKeyType, val <not Stringer>) my-rg my-publicip]) at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/azurebastion.go:35 because: there are no expected calls of the method "Get" for that receiver
        controller.go:266: missing call(s) to *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:74
        controller.go:266: aborting test due to missing call(s)
    --- FAIL: TestReconcileBastionHosts/fail_to_get_publicip (0.00s)
        azurebastion.go:35: Unexpected call to *mock_publicips.MockClient.Get([context.TODO.WithValue(type tele.CorrIDKey, val <not Stringer>).WithValue(type trace.traceContextKeyType, val <not Stringer>) my-rg my-publicip]) at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/azurebastion.go:35 because: 
            Expected call at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:99 doesn't have a prerequisite call satisfied:
            *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:98
            should be called before:
            *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:99
        controller.go:266: missing call(s) to *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:98
        controller.go:266: missing call(s) to *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:99
        controller.go:266: aborting test due to missing call(s)
    --- FAIL: TestReconcileBastionHosts/create_publicip_fails (0.00s)
        azurebastion.go:35: Unexpected call to *mock_publicips.MockClient.Get([context.TODO.WithValue(type tele.CorrIDKey, val <not Stringer>).WithValue(type trace.traceContextKeyType, val <not Stringer>) my-rg my-publicip]) at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/azurebastion.go:35 because: 
            Expected call at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:125 doesn't have a prerequisite call satisfied:
            *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:124
            should be called before:
            *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:125
        controller.go:266: missing call(s) to *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:124
        controller.go:266: missing call(s) to *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:125
        controller.go:266: missing call(s) to *mock_publicips.MockClient.CreateOrUpdate(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-publicip, is assignable to PublicIPAddress) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:126
        controller.go:266: aborting test due to missing call(s)
    --- FAIL: TestReconcileBastionHosts/fails_to_get_a_created_publicip (0.00s)
        bastionhosts_test.go:268: 
            Expected
                <*errors.withStack | 0xc000114b58>: {
                    error: <*errors.withMessage | 0xc00012afc0>{
                        cause: <*errors.withStack | 0xc000114b28>{
                            error: <*errors.withMessage | 0xc00012afa0>{
                                cause: <autorest.DetailedError>{
                                    Original: nil,
                                    PackageType: "",
                                    Method: "",
                                    StatusCode: <int>404,
                                    Message: "Not found",
                                    ServiceError: nil,
                                    Response: {
                                        Status: "",
                                        StatusCode: 404,
                                        Proto: "",
                                        ProtoMajor: 0,
                                        ProtoMinor: 0,
                                        Header: nil,
                                        Body: nil,
                                        ContentLength: 0,
                                        TransferEncoding: nil,
                                        Close: false,
                                        Uncompressed: false,
                                        Trailer: nil,
                                        Request: nil,
                                        TLS: nil,
                                    },
                                },
                                msg: "failed to get public IP for azure bastion",
                            },
                            stack: [0x1e5afa6, 0x1e5b4fc, 0x1e5be33, 0x1108be2, 0x106a101],
                        },
                        msg: "error creating Azure Bastion",
                    },
                    stack: [0x1e5b554, 0x1e5be33, 0x1108be2, 0x106a101],
                }
            to match error
                <string>: failed to get created publicIP: #: Internal Server Error: StatusCode=500
        controller.go:266: missing call(s) to *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:150
        controller.go:266: missing call(s) to *mock_publicips.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:153
        controller.go:266: missing call(s) to *mock_publicips.MockClient.CreateOrUpdate(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-publicip, is assignable to PublicIPAddress) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:152
        controller.go:266: aborting test due to missing call(s)
    --- FAIL: TestReconcileBastionHosts/bastion_successfully_created_with_created_public_ip (0.00s)
        azurebastion.go:35: Unexpected call to *mock_publicips.MockClient.Get([context.TODO.WithValue(type tele.CorrIDKey, val <not Stringer>).WithValue(type trace.traceContextKeyType, val <not Stringer>) my-rg my-publicip]) at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/azurebastion.go:35 because: 
            Expected call at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:179 doesn't have a prerequisite call satisfied:
            *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:178
            should be called before:
            *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:179
            Expected call at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:181 doesn't have a prerequisite call satisfied:
            *mock_publicips.MockClient.CreateOrUpdate(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-publicip, is assignable to PublicIPAddress) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:180
            should be called before:
            *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:181
        controller.go:266: missing call(s) to *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:178
        controller.go:266: missing call(s) to *mock_publicips.MockClient.CreateOrUpdate(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-publicip, is assignable to PublicIPAddress) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:180
        controller.go:266: missing call(s) to *mock_bastionhosts.Mockclient.CreateOrUpdate(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-bastion, is assignable to BastionHost) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:182
        controller.go:266: missing call(s) to *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:179
        controller.go:266: missing call(s) to *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:181
        controller.go:266: aborting test due to missing call(s)
    --- FAIL: TestReconcileBastionHosts/bastion_successfully_created (0.00s)
        azurebastion.go:35: Unexpected call to *mock_publicips.MockClient.Get([context.TODO.WithValue(type tele.CorrIDKey, val <not Stringer>).WithValue(type trace.traceContextKeyType, val <not Stringer>) my-rg my-publicip]) at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/azurebastion.go:35 because: 
            Expected call at /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:209 doesn't have a prerequisite call satisfied:
            *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:208
            should be called before:
            *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:209
        controller.go:266: missing call(s) to *mock_subnets.MockClient.Get(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-vnet, is equal to my-subnet) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:208
        controller.go:266: missing call(s) to *mock_publicips.MockClient.Get(expected a context.Context, but got *context.valueCtx, is equal to my-rg, is equal to my-publicip) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:209
        controller.go:266: missing call(s) to *mock_bastionhosts.Mockclient.CreateOrUpdate(expected a context.Context, but got <nil>, is equal to my-rg, is equal to my-bastion, is assignable to BastionHost) /Users/karuppiahn/projects/github.com/kubernetes-sigs/cluster-api-provider-azure/azure/services/bastionhosts/bastionhosts_test.go:210
        controller.go:266: aborting test due to missing call(s)
FAIL
FAIL	sigs.k8s.io/cluster-api-provider-azure/azure/services/bastionhosts	0.518s
FAIL

Which shows that there are failures. And I believe these are valid failures as I can see from the corresponding source code that CreateOrUpdate method of publicips client is never called, so the failures saying missing call(s) to *mock_publicips.MockClient.CreateOrUpdate... are true

We need to investigate this issue, as this kind of denotes that we cannot trust the tests because they can give false results - in the above case - a false result where failing tests are shown as passing when running in parallel
@karuppiah7890
Copy link
Contributor Author

I have fixed the bastionhosts_test.go tests as part of #1745 here https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/1745/files#diff-3bccc34c3fa316cdb2570f0d05aea4fc252815a7e76a0e571f881afe5d6bffe6 as I had to fix compile errors when I renamed publicips client method CreateOrUpdate to CreateOrUpdateAsync and also changed it's mocks

@karuppiah7890
Copy link
Contributor Author

/assign

@karuppiah7890
Copy link
Contributor Author

I think I'll raise a separate PR for this to fix this issue separately, instead of coupling the fix with #1745

@karuppiah7890 karuppiah7890 mentioned this issue Oct 4, 2021
Merged
3 tasks
karuppiah7890 added a commit to karuppiah7890/issues-info that referenced this issue Oct 4, 2021
@karuppiah7890
add kubernetes-sigs/cluster-api-provider-azure#1753 issue
0348832 
Signed-off-by: Karuppiah Natarajan <[email protected]>
This was referenced Oct 5, 2021
Closed
Closed
karuppiah7890 added a commit to karuppiah7890/issues-info that referenced this issue Oct 15, 2021
@karuppiah7890
update kubernetes-sigs/cluster-api-provider-azure#1753
97474c5 
Signed-off-by: Karuppiah Natarajan <[email protected]>
karuppiah7890 added a commit to karuppiah7890/issues-info that referenced this issue Oct 15, 2021
@karuppiah7890
update kubernetes-sigs/cluster-api-provider-azure#1753
2e14805 
Signed-off-by: Karuppiah Natarajan <[email protected]>
Signed-off-by: Karuppiah Natarajan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@karuppiah7890 karuppiah7890

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix tests in bastion_hosts_test.go file karuppiah7890/cluster-api-provider-azure
1 participant
@karuppiah7890