awx hangs after 48 hours

[blaisemichel]

ISSUE TYPE

• Bug Report

SUMMARY

every 48 hours awx hang and all job are hangs in pending state

ENVIRONMENT

• AWX version: 9.2.0
• AWX install method:docker on linux
• Ansble version: 2.9.2
• Operating System: centos 7
• Web Browser:chrome

STEPS TO REPRODUCE

launch big job with elapsed time 20:00 and wait 48 hours

EXPECTED RESULTS

I expect awx run without hang

ACTUAL RESULTS

awx hangs after 48 hours

ADDITIONAL INFORMATION

[blaisemichel]

awx.zip
awx1.zip
awx2.zip
awx3.zip

rename awx1.zip awx.z01
rename awx2.zip awx.z02
rename awx3.zip awx.z03

[blaisemichel]

sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar1.zip
sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar2.zip
sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar3.zip
sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar4.zip
sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.zip

sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar1.zip rename sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.z01

sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar2.zip rename sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.z02

sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar3.zip rename sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.z03

sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar4.zip rename sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.z04

[ryanpetrello]

@blaisemichel I can't get your .zip archives to unpack.

% unzip awx.zip
Archive:  awx.zip
warning [awx.zip]:  zipfile claims to be last disk of a multi-part archive;
  attempting to process anyway, assuming all parts have been concatenated
  together in order.  Expect "errors" and warnings...true multi-part support
  doesn't exist yet (coming soon).
file #1:  bad zipfile offset (local header sig):  4

I think what you're encountering is a duplicate of #5617

If you don't mind, and if you can reproduce this again, check out these comments (starting here), and see if you can reproduce the same thing:

#5617 (comment)

If you can reproduce a similar hang, it would be very helpful and much appreciated it you would chime in with details on that issue as well.

Thanks!