Jobs output is cut due to kubernetes container log rotation

[kzinas-adv]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

After fix ansible/receptor#683 job will not fail, but it outputs to AWX only first ant last log from rotation.

k8s 1.25.4-do.0
AWX 21.10.1
receptor 1.3.0
Fix enabled

root@mmx-pool07-mar4q:/var/log/pods/default_automation-job-25971-7s9hc_cb76abc8-65ee-4191-bb02-4a28980b12d8/worker# ls -ltr
total 13988
-rw-r----- 1 root root 10670670 Dec 23 15:06 0.log.20221223-150601
-rw-r--r-- 1 root root   765905 Dec 23 15:06 0.log.20221223-145810.gz
-rw-r----- 1 root root  2879976 Dec 23 15:08 0.log
root@mmx-pool07-mar4q:/var/log/pods/default_automation-job-25971-7s9hc_cb76abc8-65ee-4191-bb02-4a28980b12d8/worke

Time: 22:04:32 -> 00:11:31
Playbook run: 0:01:40.151 -> 2:08:39.984

Wednesday 21 December 2022  22:04:32 +0000 (0:00:29.791)       0:01:40.151 ****

TASK [create : Create var volume] ***************************************
ok: [db02.host.ge -> localhost]
ok: [db01.host.ge -> localhost]

changed: [localhost] => (item=www.random.org)
Thursday 22 December 2022  00:11:31 +0000 (0:00:03.724)       2:08:39.984 *****

Time: 01:39:55 -> 01:59:38
Playbook run: 0:08:04.813 -> 0:27:48.142

Thursday 22 December 2022  01:39:55 +0000 (0:00:00.186)       0:08:04.813 ***** 

TASK [generic-backend : Copy files to /opt/ab-reports] *************************
ok: [srv02.host.ge] => (item=/runner/project/private/nats/nats.ca.pem)
ok: [srv02.host.ge] => (item=/runner/project/private/nats/nats.client.pem)
ok: [srv02.host.ge] => (item=/runner/project/private/nats/nats.client.key)
Thursday 22 December 2022  01:59:38 +0000 (0:00:00.498)       0:27:48.142 ***** 

AWX version

21.10.1

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

2.12.5

Operating system

Ubuntu

Web browser

No response

Steps to reproduce

On K8S Run any longer job with enough output to reach 21Mb (I assume default k8s container log size is 10MB).
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ says it is 10MB
--container-log-max-size string     Default: 10Mi
You should get first 10MB in https://awx.com/api/v2/jobs/99999/stdout/ and then last 1MB.

Expected results

Get all output.

Actual results

Cut output, only from first logrotate file and last one.

Additional information

No response

[gundalow]

@kzinas-adv Thanks for reporting this, given the upvotes on this issue seems like others are hitting this problem as well.

[gundalow]

@fosterseth Could you please take a look as a matter of priority.

[fosterseth]

@kzinas-adv are you still experiencing issues around this?

[kzinas-adv]

Nop with awx 22.0.0 everything went fine.

[kzinas-adv]

Nop it still wrong, today we got failed task , but did not found any issue in awx playbook output and it was cut:

ok: [bdv-prd-es-proxy-vl01.trtr.ge]

Friday 21 April 2023 00:19:00 +0000 (0:00:03.164) 0:15:46.023 **********

ok: [bdv-prd-es-proxy-vl01.trtr.ge -> localhost]

Friday 21 April 2023 01:25:01 +0000 (0:00:28.379) 1:21:47.222 **********

Friday 21 April 2023 01:25:02 +0000 (0:00:00.504) 1:21:47.727 **********

Please reopen issue. It is with awx 22.0.0