[WIP] 🌱 capd: inspect container when container bootstrap fails #4416

sbueringer · 2021-03-31T14:07:15Z

What this PR does / why we need it:
Adds more debug information in case the container creation does not work. For context, please see #4405 (comment)

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Improves observability to debug #4405

sbueringer · 2021-03-31T14:08:07Z

Let's try to hit the issue and find out if the docker inspect works and helps.
/test pull-cluster-api-e2e-full-main

fabriziopandini

I think we should move this within the PR for adding more info to the logs, so we can cross check the problem in logs as well

fabriziopandini · 2021-03-31T14:17:57Z

test/infrastructure/docker/docker/machine.go

@@ -431,3 +435,12 @@ func (m *Machine) machineImage(version *string) string {

 	return fmt.Sprintf("%s:%s", defaultImageName, versionString)
 }
+
+func logInspectContainer(ctx context.Context, log logr.Logger, name string) {


nit
If you passing ctx you can use log := ctrl.LoggerFrom(ctx) (no need to pass the logger)

Good point. Is the pattern in general to just pass the context and always get the logger from there? (in cases where we have to pass the context anyway, which is probably almost everywhere)

sbueringer · 2021-03-31T14:22:14Z

I think we should move this within the PR for adding more info to the logs, so we can cross check the problem in logs as well

How do you mean this?

The logger itself should have and print the name and namespace of the machine afaik. But I think you're referring to something else.

Of course open to all suggestions I'm not familiar with the code at all :)

sbueringer · 2021-03-31T15:29:05Z

That wasn't the error I was looking for :) (will be fixed in #4415)
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-03-31T15:30:37Z

/test pull-cluster-api-e2e-full-main

k8s-ci-robot · 2021-03-31T16:15:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign vincepri after the PR has been reviewed.
You can assign the PR to them by writing /assign @vincepri in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbueringer · 2021-03-31T16:17:15Z

I hit the issue, unfortunately there was nothing conclusive in the docker inspect output: https://gist.github.com/sbueringer/581a793084fd129d97816b8529ecdc81

Cherry-picked #4414 for now, maybe that helps

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-03-31T16:44:31Z

Just realized that I can also just find the root cause on this PR with a few test runs. I'll play around a bit, let's discuss afterwards which of the changes we consider merging into the main branch.

For now: added docker logs

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-03-31T18:21:24Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-03-31T19:11:41Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T05:43:02Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T06:51:19Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T07:43:29Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T08:40:39Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T09:45:08Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T11:08:09Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T11:59:53Z

/test pull-cluster-api-e2e-full-main
(I think the error occurs only at certain times, possibly when there is high load in the CI cluster through noisy neighbour effects)

sbueringer · 2021-04-01T12:56:55Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T14:07:45Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T14:26:10Z

/test pull-cluster-api-e2e-main

sbueringer · 2021-04-01T15:10:50Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T16:11:40Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T16:52:45Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T18:19:20Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T18:42:18Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-01T20:14:38Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-02T05:23:37Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-02T07:26:07Z

Now with a new kind image which uses /bin/echo instead of echo when piping into head -n 1:

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-02T10:23:46Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-02T12:39:40Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-02T15:04:53Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-02T15:45:49Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2021-04-02T17:28:36Z

/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-full-main

k8s-ci-robot · 2021-04-02T19:44:53Z

@sbueringer: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-cluster-api-e2e-full-main	`0d74d6d`	link	`/test pull-cluster-api-e2e-full-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sbueringer · 2021-04-06T07:44:20Z

I'll close this PR as it served it's purpose (and found an issue in kind). I'll open a new (clean) one for the addition of docker inspect/logs.
/close

k8s-ci-robot · 2021-04-06T07:44:27Z

@sbueringer: Closed this PR.

In response to this:

I'll close this PR as it served it's purpose (and found an issue in kind). I'll open a new (clean) one for the addition of docker inspect/logs.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 31, 2021

k8s-ci-robot requested review from detiber and justinsb March 31, 2021 14:09

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Mar 31, 2021

fabriziopandini reviewed Mar 31, 2021

View reviewed changes

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 31, 2021

sbueringer changed the title ~~[WIP] 🐛 capd: inspect container when container bootstrap fails~~ [WIP] 🌱 capd: inspect container when container bootstrap fails Apr 1, 2021

sbueringer mentioned this pull request Apr 1, 2021

Multiple e2e tests are flaky because of error container is not running #4405

Closed

sbueringer mentioned this pull request Apr 2, 2021

🌱 gather more Docker and containerd data/logs in ci-e2e.sh #4414

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2021

sbueringer added 5 commits April 2, 2021 17:32

capd: inspect container when container bootstrap fails

69322c2

add docker log

cb32558

run tests 20x, use kindest verbose image

e5ff01e

small cleanup

c572b96

upgrade kind to 0.10.0

0d74d6d

sbueringer force-pushed the pr-capd-inspect-on-container-creation-error branch from f0e7a5a to 0d74d6d Compare April 2, 2021 15:36

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2021

k8s-ci-robot closed this Apr 6, 2021

sbueringer mentioned this pull request Apr 6, 2021

🌱 capd: print debug infos on container creation error #4432

Merged

[WIP] 🌱 capd: inspect container when container bootstrap fails #4416

[WIP] 🌱 capd: inspect container when container bootstrap fails #4416

Conversation

sbueringer commented Mar 31, 2021 • edited Loading

sbueringer commented Mar 31, 2021

fabriziopandini left a comment

Choose a reason for hiding this comment

fabriziopandini Mar 31, 2021

Choose a reason for hiding this comment

sbueringer Mar 31, 2021

Choose a reason for hiding this comment

sbueringer commented Mar 31, 2021 • edited Loading

sbueringer commented Mar 31, 2021 • edited Loading

sbueringer commented Mar 31, 2021

k8s-ci-robot commented Mar 31, 2021

sbueringer commented Mar 31, 2021

sbueringer commented Mar 31, 2021

sbueringer commented Mar 31, 2021

sbueringer commented Mar 31, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021 • edited Loading

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 1, 2021

sbueringer commented Apr 2, 2021

sbueringer commented Apr 2, 2021 • edited Loading

sbueringer commented Apr 2, 2021

sbueringer commented Apr 2, 2021

sbueringer commented Apr 2, 2021

sbueringer commented Apr 2, 2021

sbueringer commented Apr 2, 2021

k8s-ci-robot commented Apr 2, 2021

sbueringer commented Apr 6, 2021

k8s-ci-robot commented Apr 6, 2021

sbueringer commented Mar 31, 2021 •

edited

Loading

sbueringer commented Mar 31, 2021 •

edited

Loading

sbueringer commented Mar 31, 2021 •

edited

Loading

sbueringer commented Apr 1, 2021 •

edited

Loading

sbueringer commented Apr 2, 2021 •

edited

Loading