v0.4 runtime panic

[kinglion811]

kubernetes version is :1.11
kube-batch version is:v0.5
when I start kube-batch and schedule tf job，After running for a while ，kube-batch will panic:
panic information is

Resource is not sufficient to do operation: <cpu 52000.00, memory 261334462464.00, GPU 0.00> sub <cpu 12000.00, memory 60000000000.00, GPU 2000.00> [recovered]
	panic: Resource is not sufficient to do operation: <cpu 52000.00, memory 261334462464.00, GPU 0.00> sub <cpu 12000.00, memory 60000000000.00, GPU 2000.00>

Causing panic are：

[kinglion811]

　@k82cn

[k82cn]

We merged #860 few days ago, which maybe helpful :)

[kinglion811]

@k82cn that can not slove my problem,this issue is mainly node's Idle And the main reason is the inconsistent resources of gpu，I merge the code,the problem reappear

[k82cn]

Thanks for your confirmation :)
We also meet similar issue this morning; in our cause, device plugin did not report gpu info in time when kubelet restart. We're working on the PR. Is that similar to your scenario, panic when kubelet with device plugin restart?

[k82cn]

/kind bug
/priority important-soon
/sig scheduling

[kinglion811]

@k82cn maybe,I will confirm your information

[kinglion811]

@k82cn
how is the progress of this issue？

[k82cn]

@asifdxtreme , would you help to cherry pick volcano-retired#26 into kube-batch :)

[kinglion811]

Observed a panic: &errors.errorString{s:"Resource is not sufficient to do operation: <cpu 56000.00, memory 270086234112.00, hugepages-1Gi 0.00, hugepages-2Mi 0.00, nvidia.com/gpu 5000.00> sub <cpu 54000.00, memory 268435456000.00, nvidia.com/gpu 8000.00>"} (Resource is not sufficient to do operation: <cpu 56000.00, memory 270086234112.00, hugepages-1Gi 0.00, hugepages-2Mi 0.00, nvidia.com/gpu 5000.00> sub <cpu 54000.00, memory 268435456000.00, nvidia.com/gpu 8000.00>)
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/workspace/src/github.com/kubernetes-sigs/kube-batch/pkg/scheduler/api/resource_info.go:158
/workspace/src/github.com/kubernetes-sigs/kube-batch/pkg/scheduler/api/node_info.go:182
/workspace/src/github.com/kubernetes-sigs/kube-batch/pkg/scheduler/cache/event_handlers.go:82
/workspace/src/github.com/kubernetes-sigs/kube-batch/pkg/scheduler/cache/event_handlers.go:93
/workspace/src/github.com/kubernetes-sigs/kube-batch/pkg/scheduler/cache/event_handlers.go:192
/workspace/src/github.com/kubernetes-sigs/kube-batch/pkg/scheduler/cache/cache.go:262
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/client-go/tools/cache/controller.go:195
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/client-go/tools/cache/controller.go:227
:0
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/client-go/tools/cache/shared_informer.go:554
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
/workspace/src/github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71

This is mainly caused by gpu lost，
If the gpu plugin reports gpu lost, it will cause the resource view to be inconsistent.

[kinglion811]

@k82cn