Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix panic as pod is nil #1006

Merged
merged 1 commit into from
May 24, 2019
Merged

Fix panic as pod is nil #1006

merged 1 commit into from
May 24, 2019

Conversation

ScorpioCPH
Copy link
Member

@ScorpioCPH ScorpioCPH commented May 23, 2019 •
edited by jlewi
Loading

@richardsliu @gaocegege After testing v1 in our cluster, at some time when there are lots of tfjobs in cluster, and many pods/services with running/error/complete state, will panic after restart tf-operator.

This change is Reviewable

@ScorpioCPH
Copy link
Member Author

/test

@coveralls
Copy link

Coverage Status

Coverage remained the same at 76.744% when pulling 13e1a12 on ScorpioCPH:fix-panic-error into f41a5e5 on kubeflow:master.

@richardsliu
Copy link
Contributor

Is this related to #965? Can you also explain more on why this is related to lots of tfjobs in the cluster?

@richardsliu
Copy link
Contributor

/retest

1 similar comment
@richardsliu
Copy link
Contributor

/retest

gaocegege
gaocegege approved these changes May 24, 2019
View reviewed changes
Copy link
Member

@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

gaocegege
gaocegege reviewed May 24, 2019
View reviewed changes
Copy link
Member

@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@gaocegege
Copy link
Member

/hold for @richardsliu 's comments

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gaocegege
Copy link
Member

/hold

@richardsliu
Copy link
Contributor

/retest

@ScorpioCPH
Copy link
Member Author

@richardsliu not very sure about #965 but i think this is not the same issue.

In k8s informer mechanism, DeleteFunc is called when an existing resource is deleted and it may be return a type cache.DeletedFinalStateUnknown.
Ref: https://github.com/kubernetes/client-go/blob/master/tools/cache/controller.go#L173

@k8s-ci-robot k8s-ci-robot merged commit 671cdab into kubeflow:master May 24, 2019
@johnugeorge johnugeorge mentioned this pull request May 27, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gaocegege gaocegege gaocegege approved these changes

@ChanYiLin ChanYiLin Awaiting requested review from ChanYiLin

@jimexist jimexist Awaiting requested review from jimexist

Assignees

@gaocegege gaocegege

Labels
approved lgtm size/XS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ScorpioCPH @coveralls @richardsliu @gaocegege @k8s-ci-robot