job in two queue will reclaim each other's tasks in dead loop

[lowang-bh]

Description

With a cluster has 11C CPU, Queue-a has a deserved=5C and capability=10C, same as queue-b.
First create job-a with replicas=5, and minAvailable=2. job a will take 10C.
Then create a job-b same as job-a, it will reclaim and evict job-a's two tasks. But now queue-a used is less than deserved, it will also reclaim from queue-b, and so on.

Steps to reproduce the issue

with scheduler cm

apiVersion: v1
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, reclaim, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
        enablePreemptable: false
      - name: conformance
    - plugins:
      - name: overcommit
      - name: drf
        enablePreemptable: false
      - name: predicates
      - name: capacity
      - name: nodeorder
      - name: binpack
kind: ConfigMap

1. apply queue-a, queue-b with yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: queue-a
spec:
  reclaimable: true
  deserved:
    cpu: 5
    memory: 2Gi
  capability:          
    cpu: 10
    memory: 5Gi
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: queue-b
spec:
  reclaimable: true
  deserved:
    cpu: 5
    memory: 2Gi
  capability:          
    cpu: 10
    memory: 5Gi

3. apply job-a.yaml

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: job-a
spec:
  schedulerName: volcano
  queue: queue-a
  tasks:
    - replicas: 5
      minAvailable: 2
      name: "master"
      template:
        metadata:
          annotations:
            volcano.sh/preemptable: "true"
        spec:
          containers:
            - image: nginx:1.14.2
              name: nginx
              resources:
                requests:
                  cpu: "2"
                  memory: "50Mi"
          restartPolicy: OnFailure

5. apply job-b.yaml

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: job-b
spec:
  schedulerName: volcano
  queue: queue-b
  tasks:
    - replicas: 5
      minAvailable: 2
      name: "worker"
      template:
        metadata:
          annotations:
            volcano.sh/preemptable: "true"
        spec:
          containers:
            - image: nginx:1.14.2
              name: nginx
              resources:
                requests:
                  cpu: "2"
                  memory: "50Mi"
          restartPolicy: OnFailure

Describe the results you received and expected

After upgrade image to #3696, it keeps stable.

What version of Volcano are you using?

master

Any other relevant information

master branch at 95d5a92

[Monokaix]

Good catch.