Simultaneous start of processes

[Mas73r]

Describe the bug
~ 1-4 times per hour simultaneous start of processes, even with the option "concurrency": "forbid".

Example of job:

{
  "name": "cron_actions",
  "displayname": "",
  "timezone": "",
  "schedule": "* * * * * *",
  "owner": "",
  "owner_email": "",
  "success_count": 26575,
  "error_count": 1,
  "last_success": "2020-04-28T12:25:34.486081724Z",
  "last_error": "2020-04-21T10:28:17.63857519Z",
  "disabled": false,
  "tags": {
    "datacenter": "msk:1",
    "server_role": "tl:1"
  },
  "metadata": null,
  "retries": 0,
  "dependent_jobs": null,
  "parent_job": "",
  "processors": {},
  "concurrency": "forbid",
  "executor": "shell",
  "executor_config": {
    "command": "/bin/bash -c '/usr/bin/sudo -u www-data php /var/www/project/run.php 2>&1 | /usr/bin/logger -p local5.info -t cron_actions'"
  },
  "status": "success",
  "next": "2020-04-28T10:01:27Z"
}

To Reproduce
Server #1:

http-addr: 127.0.0.1:8080
server: true
bootstrap-expect: 2
raft-multiplier: 1
data-dir: /var/lib/dkron/data
log-level: info
tags:
   server_role: tl
   role: tl
   datacenter: msk
encrypt: 4LjI72PW7azA4/vm77wmYA==
join:
   - 10.77.255.28
   - 10.77.255.56

Server #2:

http-addr: 127.0.0.1:8080
server: true
bootstrap-expect: 2
raft-multiplier: 1
data-dir: /var/lib/dkron/data
log-level: info
tags:
   server_role: tl
   role: tl
   datacenter: msk
encrypt: 4LjI72PW7azA4/vm77wmYA==
join:
   - 10.77.255.61
   - 10.77.255.56

Server #3

http-addr: 127.0.0.1:8080
server: true
raft-multiplier: 1
bootstrap-expect: 2
data-dir: /var/lib/dkron/data
log-level: debug
tags:
   server_role: arbiter
   role: ap
   datacenter: msk
encrypt: 4LjI72PW7azA4/vm77wmYA==
join:
   - 10.77.255.61
   - 10.77.255.28

Expected behavior
There will be no task launches on two servers at the same time

** Specifications:**

• OS: debian 9,
• Version 2.2.2

[andreygolev]

I guess I have same behaviour, but in my case I'm scheduling jobs with 2 seconds interval. Sometimes they're scheduled twice with subsecond difference between each run.
Looks like there's lack of some blocking action to ensure that scheduled job will have "running" flag right after scheduling.

[vcastellm]

I think this is expected, you are specifying two tags with cardinality, you need to specify just one tag in the job, do you have more than one datacenter? if not, try with just:

"tags": {
    "server_role": "tl:1"
  }

Let me know if that work, I will review the filters that are taking place with this config though.

[vcastellm]

check with #749

[Mas73r]

I think this is expected, you are specifying two tags with cardinality, you need to specify just one tag in the job, do you have more than one datacenter? if not, try with just:

"tags": {
    "server_role": "tl:1"
  }

Let me know if that work, I will review the filters that are taking place with this config though.

If use

tags:
   server_role: tl

in config file -- all work normal