Very rare race condition between move_and_rebuild and validate_env

[sadsfae]

• QUADS version quads-1.1.2-20200122.noarch
• Python version: 3.7
• Operating System: Fedora 29

Describe the bug
There appears to be a very rare race condition between when validate_env.py runs and when move_and_rebuild_host.py executes in terms of releasing an environment.

To Reproduce

• validate_env.py runs out of cron prior to when move_and_rebuild_host.py is able to mark target systems for build in Foreman, reboot them via badfish/ipmitool or begin it's provisioning
• validate_env.py runs after a cloud is defined to start (example: 13:00 UTC in cloud09 on 2020-01-27 for us) but before move_and_rebuild_host.py can start it's provisioning.

According to QUADS cloud09 is active, but no provisioning actions have happened yet to make it fail the validation checks.

End Result

The end result is that an environment could release an all clear, ready-to-use notification to the tenant but it's still not actually passed the full validation workflow.

This is our working theory as to how we had an environment immediately send out a "ready to use" notification but it still had not undergone full validation yet.

Expected behavior
An environment should never be released when it's not fully validated and checked.