volcano scheduler panic when setNodeState

[zishen]

What happened:
When add new pod，volcano-scheduler panic.

2022/01/14 10:11:10 maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS
W0114 10:11:10.692724       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0114 10:11:11.182069       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 354 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0xaaaae2c9aec0, 0xaaaae3af9450})
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x98
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0})
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x88
panic({0xaaaae2c9aec0, 0xaaaae3af9450})
	/opt/buildtools/golang/go-1.17.5/src/runtime/panic.go:1038 +0x21c
volcano.sh/volcano/pkg/scheduler/api.(*NodeInfo).setNodeState(0x400084a4d0, 0x0)
	/usr1/go/src/volcano.sh/volcano/pkg/scheduler/api/node_info.go:236 +0x114
volcano.sh/volcano/pkg/scheduler/api.NewNodeInfo(0x0)
	/usr1/go/src/volcano.sh/volcano/pkg/scheduler/api/node_info.go:114 +0x35c
volcano.sh/volcano/pkg/scheduler/cache.(*SchedulerCache).addTask(0x400049a000, 0x400084a420)
	/usr1/go/src/volcano.sh/volcano/pkg/scheduler/cache/event_handlers.go:66 +0x68
volcano.sh/volcano/pkg/scheduler/cache.(*SchedulerCache).addPod(0x400049a000, 0x40014abb58)
	/usr1/go/src/volcano.sh/volcano/pkg/scheduler/cache/event_handlers.go:92 +0x34
volcano.sh/volcano/pkg/scheduler/cache.(*SchedulerCache).AddPod(0x400049a000, {0xaaaae2e82f20, 0x40014abb58})
	/usr1/go/src/volcano.sh/volcano/pkg/scheduler/cache/event_handlers.go:198 +0xe0
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/cache/controller.go:227
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnAdd({0x4000666bd0, {0xaaaae2ee08e0, 0x4000666be8}}, {0xaaaae2e82f20, 0x40014abb58})
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/cache/controller.go:260 +0x64
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/cache/shared_informer.go:777 +0xb4
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x4000726f50)
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x64
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000a69f50, {0xaaaae2eb2680, 0x4000aa2000}, 0x1, 0x4000aa0000)
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x94
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4000726f50, 0x3b9aca00, 0x0, 0x1, 0x4000aa0000)
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x88
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0x400048c880)
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x80
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0x40002c01b0, 0x4000a8e000)
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x54
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/usr1/go/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x5c
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xaaaae2867124]

And the code is

func (ni *NodeInfo) setNodeState(node *v1.Node) {
	// If node is nil, the node is un-initialized in cache
	if node == nil {
		ni.State = NodeState{
			Phase:  NotReady,
			Reason: "UnInitialized",
		}
		klog.Warningf("set the node %s status to %s for the reason UnInitialized.", node.Name, NotReady.String())
		return
	}
....
}

in node_info.go

What you expected to happen:

not panic

How to reproduce it (as minimally and precisely as possible):

volcano v1.4.0 add v1.5.0-bata pacth:https://github.com/volcano-sh/volcano/pull/1685/files

Anything else we need to know?:

Environment:

• Volcano Version:v1.4.0+patch#1685

• Kubernetes version (use kubectl version):1.17.3

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):euleros

• Kernel (e.g. uname -a):

• Install tools:

• Others:

[zishen]

@Lion-Wei

[Thor-wl]

fix: #1952

[zishen]

fix: #1952

This is log panic when using nil node.

Have a look at line 236

[Thor-wl]

/assign @Thor-wl

[zishen]

/assign @Thor-wl

Thanks a lot.

[Thor-wl]

/close