Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1763700: kubelet: add dependency on network-online.target #1206

Conversation

rphillips
Copy link
Contributor

- What I did
This fixes a race when the kubelet enumerates the hostname at startup. Wait for the network-online.target to be ready before starting the Kubelet.

ref: https://bugzilla.redhat.com/show_bug.cgi?id=1763700

- How to verify it

- Description for the changelog

This fixes a race when the kubelet enumerates the hostname at startup.

ref: https://bugzilla.redhat.com/show_bug.cgi?id=1763700
@openshift-ci-robot
Copy link
Contributor

@rphillips: This pull request references Bugzilla bug 1763700, which is invalid:

  • expected the bug to target the "4.3.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1763700: kubelet: add dependency on network-online.target

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 22, 2019
@rphillips
Copy link
Contributor Author

/bugzilla refresh

@cgwalters
Copy link
Member

Also remember the installer has a copy of kubelet.service that needs updating too. See e.g. openshift/installer#2459

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Oct 22, 2019
@openshift-ci-robot
Copy link
Contributor

@rphillips: This pull request references Bugzilla bug 1763700, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@cgwalters
Copy link
Member

Given Kube dependency on the hostname (something that'd be good to fix eventually but)...this makes sense. Did this really never come up before though? Do other people's kubelet.service units do this too?

Copy link
Member

@ashcrow ashcrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 22, 2019
@rphillips
Copy link
Contributor Author

rphillips commented Oct 22, 2019

The cloud provider in the kubelet will gather the hostname from the metadata service (ie, aws, etc). Namely, this PR fixes the issue with metal or other non-cloud based platforms.

@cgwalters typhoon waits for the resolve.conf to get set [1], but I don't love the implementation.

  1. https://github.com/poseidon/typhoon/blob/24fc440d837afceb189986c5375b68d8ce84ce50/bare-metal/container-linux/kubernetes/cl/worker.yaml.tmpl#L17

@cgwalters
Copy link
Member

The cloud provider in the kubelet will gather the hostname from the metadata service (ie, aws, etc).

😱 There's another codebase doing that?

@rphillips
Copy link
Contributor Author

@cgwalters same code base as the kubelet, but each cloud provider does the hostname detection slightly different [source].

@jlebon
Copy link
Member

jlebon commented Oct 22, 2019

/approve

@rphillips
Copy link
Contributor Author

/retest

@kikisdeliveryservice
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 22, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ashcrow, jlebon, kikisdeliveryservice, miabbott, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [ashcrow,kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rphillips
Copy link
Contributor Author

level=error msg="Error: Error creating S3 bucket: timeout while waiting for state to become 'success' (timeout: 5m0s)"
level=error
level=error msg="  on ../tmp/openshift-install-518815423/bootstrap/main.tf line 5, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg="   5: resource \"aws_s3_bucket\" \"ignition\" {"

@rphillips
Copy link
Contributor Author

/retest

@openshift-ci-robot
Copy link
Contributor

@rphillips: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-aws-scaleup-rhel7 bce770d link /test e2e-aws-scaleup-rhel7

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit db3893e into openshift:master Oct 23, 2019
@openshift-ci-robot
Copy link
Contributor

@rphillips: All pull requests linked via external trackers have merged. Bugzilla bug 1763700 has been moved to the MODIFIED state.

In response to this:

Bug 1763700: kubelet: add dependency on network-online.target

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@runcom
Copy link
Member

runcom commented Oct 23, 2019

/cherrypick release-4.2

@openshift-cherrypick-robot

@runcom: new pull request created: #1212

In response to this:

/cherrypick release-4.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sdodson added a commit to sdodson/installer that referenced this pull request Nov 19, 2019
Recent changes[1] mean that critical services wait for
NetworkManager-wait-online.service and the timeout has been increased[2] to
300 seconds which results in significant delays in booting instances
while we wait for the second interface.

1 - openshift/machine-config-operator#1206
2 - https://bugzilla.redhat.com/show_bug.cgi?id=1741296
sdodson added a commit to sdodson/installer that referenced this pull request Dec 4, 2019
Recent changes[1] mean that critical services wait for
NetworkManager-wait-online.service and the timeout has been increased[2] to
300 seconds which results in significant delays in booting instances
while we wait for the second interface.

1 - openshift/machine-config-operator#1206
2 - https://bugzilla.redhat.com/show_bug.cgi?id=1741296
hardys pushed a commit to hardys/machine-config-operator that referenced this pull request Mar 9, 2020
This was added in openshift#1206 to the base unit, but not the baremetal
copy of this file
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/machine-config-operator that referenced this pull request Mar 9, 2020
This was added in openshift#1206 to the base unit, but not the baremetal
copy of this file
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/machine-config-operator that referenced this pull request Mar 9, 2020
This was added in openshift#1206 to the base unit, but not the baremetal
copy of this file
yboaron pushed a commit to yboaron/machine-config-operator that referenced this pull request Mar 22, 2020
This was added in openshift#1206 to the base unit, but not the baremetal
copy of this file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants