Isolated runs choke on rsync's .~tmp~ directories

[domq]

ISSUE TYPE

• Bug Report

SUMMARY

In some circumstances, running an Ansible playbook as an isolated job causes a spurious failure (despite all individual Ansible tasks succeeding), because a stray .~tmp~ directory pops up in an unexpected place.

ENVIRONMENT

• AWX version: AWX 10.0.0
• AWX install method: openshift
• Ansible version: 2.9.5

STEPS TO REPRODUCE

• Set AWX_RESOURCE_PROFILING_ENABLED = True
• Keep running isolated jobs until the problem shows up

EXPECTED RESULTS

All such jobs should succeed or fail depending solely on the underlying Ansible tasks' outcomes

ACTUAL RESULTS

Here is an example stack trace (also found in #6280):

Traceback (most recent call last): 
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/tasks.py", line 1468, in run
    ident=str(self.instance.pk)) 
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/isolated/manager.py", line 422, in run   status, rc = self.check() 
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/isolated/manager.py", line 241, in check
     self.consume_events() 
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/isolated/manager.py", line 293, in consume_events
    open(os.path.join(events_path, event), 'r')
IsADirectoryError: [Errno 21] Is a directory: '/tmp/awx_754_hs8ylxru/artifacts/754/job_events/.~tmp~' 

[domq]

Analysis:

• awx/main/isolated/manager.py periodically runs the awx/playbooks/check_isolated.yml playbook, which relies on the synchronize module
• Said module forces the --delay-updates rsync flag, which causes rsync to create these .~tmp~ directories

Suggested fix: add .~tmp~ to the rsync_exclude list

[domq]

Upon inspecting this further, I believe I have the root cause wrong; now I am suspecting an rsync deadlock — According to the logs of my awx-task container on Kubernetes, the error pops up exactly 10 minutes after rsync starts

I shall close this issue and reopen a new one if I manage to get to the bottom of this.

[ryanpetrello]

Thanks @domq - the traceback you reported does look legitimate, though, so I merged the PR I opened.

[domq]

I wrote up the proper root cause analysis of my issue in #6692.