[17.1.0] Job remains pending indefinitely when awx_task_hostname and awx_web_hostname are modified in inventory

[jean-christophe-manciot]

@ryanpetrello

ISSUE TYPE

• Bug Report

SUMMARY

A git source control project is created, then the related job remains pending indefinitely and no cloning is ever made.

ENVIRONMENT

• AWX version: 17.1.0
• AWX install method: docker compose on linux
• Ansible version: 3.2.0
• Operating System: Ubuntu 21.04
• docker-ce: 5:20.10.5~3-0~ubuntu-groovy
• docker-compose: 1.29.0

STEPS TO REPRODUCE

0.1. Run inside Python3 virtual env
0.2. Install latest ansible, wheel, docker-compose and redis with pip3

1. Checkout 17.1.0 & create new branch
2. Modify installer/inventory with:

• awx_official=true
• docker_compose_dir=/some/folder
• host_port=<some_available_port>
• logging_driver=journald
• pg_database=awx
• pg_hostname=<some_private_ip_address>
• pg_password=some_password
• pg_port=<<some_pg_used_port>
• pg_sslmode=require
• pg_username=awx
• postgres_data_dir=''
• project_data_dir=/awx/projects
• secret_key=some_password
• use_docker_compose=true

2. Restart PostgreSQL 12 AWX Cluster
3. Build as root with:

export awx_version="17.1.0"
cd installer
ansible-playbook -i inventory install.yml

4. Login as admin
5. Create organization
6. Create source control credential
7. Create project with:

Source Control Type: git
Source Control URL: user@<ip_address>:/path/to/git/repository
Options:
  - Update Revision on Launch
  - Allow Branch Override

EXPECTED RESULTS

The git repository should be cloned into the Playbook Directory within <project_data_dir>.

ACTUAL RESULTS

• The related job is pending.
• The project details shows a "Project Base Path" (/var/lib/awx/projects) mismatch with <project_data_dir>
• /var/lib/awx/projects is also empty inside awx_task container
• The Playbook Directory shown in the project details is not created in <project_data_dir>

A manual git clone on the controller is successful:

/tmp/awx# git clone user@<ip_address>:/path/to/git/repository
Cloning into '.'...
...
remote: Enumerating objects: 95, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (81/81), done.
remote: Total 95 (delta 54), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (95/95), 30.72 KiB | 6.14 MiB/s, done.
Resolving deltas: 100% (54/54), done.

Also, following some advices in a similar old thread inside awx_task:

bash-4.4# awx-manage run_dispatcher --status

bash-4.4# awx-manage shell_plus
# Shell Plus Model Imports
from awx.conf.models import Setting
from awx.main.models.activity_stream import ActivityStream
from awx.main.models.ad_hoc_commands import AdHocCommand
from awx.main.models.credential import Credential, CredentialInputSource, CredentialType
from awx.main.models.events import AdHocCommandEvent, InventoryUpdateEvent, JobEvent, ProjectUpdateEvent, SystemJobEvent
from awx.main.models.ha import Instance, InstanceGroup, InventoryInstanceGroupMembership, OrganizationInstanceGroupMembership, TowerScheduleState, UnifiedJobTemplateInstanceGroupMembership
from awx.main.models.inventory import CustomInventoryScript, Host, Inventory, InventorySource, InventoryUpdate, SmartInventoryMembership
from django.contrib.auth.models import Group, Permission, User
from awx.main.models.jobs import Job, JobHostSummary, JobLaunchConfig, JobTemplate, SystemJob, SystemJobTemplate
from awx.main.models.label import Label
from awx.main.models.notifications import Notification, NotificationTemplate
from awx.main.models.oauth import OAuth2AccessToken, OAuth2Application
from awx.main.models.organization import Organization, OrganizationGalaxyCredentialMembership, Profile, Team, UserSessionMembership
from awx.main.models.projects import Project, ProjectUpdate
from awx.main.models.rbac import Role, RoleAncestorEntry
from awx.main.models.schedules import Schedule
from awx.main.models.unified_jobs import UnifiedJob, UnifiedJobDeprecatedStdout, UnifiedJobTemplate
from awx.main.models.workflow import WorkflowApproval, WorkflowApprovalTemplate, WorkflowJob, WorkflowJobNode, WorkflowJobTemplate, WorkflowJobTemplateNode
from awx.sso.models import UserEnterpriseAuth
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from django.contrib.sites.models import Site
from oauth2_provider.models import Grant, RefreshToken
from social_django.models import Association, Code, Nonce, Partial, UserSocialAuth
from taggit.models import Tag, TaggedItem
# Shell Plus Django Imports
from django.core.cache import cache
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import transaction
from django.db.models import Avg, Case, Count, F, Max, Min, Prefetch, Q, Sum, When, Exists, OuterRef, Subquery
from django.utils import timezone
from django.urls import reverse
Python 3.6.8 (default, Mar 19 2021, 05:13:41) 
[GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> def check_lock():
...     import time
...     from awx.main.utils.pglock import advisory_lock
...     time.sleep(1)
...     with advisory_lock('task_manager_lock', wait=False) as acquired:
...         return acquired
... 
>>> [check_lock() for i in range(20)]
[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]
>>> exit

[shanemcd]

Hello. We removed the installer you're using in AWX 18, and do not support past releases of AWX. Can you try to reproduce this on the latest release?

[jean-christophe-manciot]

I discovered the source of the issue in AWX 17.1.0: changing awx_task_hostname and awx_web_hostname in installer/inventory.
Using default values for both variables (awx and awxweb respectively) is a workaround.

[derifgig]

I has the same symptoms too.
Issue reproduced on 19.0 (deployment by manual with minikube)

docker@minikube:~$ docker exec 3a71d144c201 awx-manage version
19.0.0

docker@minikube:~$ docker inspect 3a71d144c201
[
    {
        "Id": "3a71d144c20193ccfae5e6d5f31b58586cd9286b9029e998b443312d77c77a75",
        "Created": "2021-04-26T14:14:58.830381764Z",
        "Path": "/usr/bin/tini",
        "Args": [
            "--",
            "/bin/sh",
            "-c",
            "/usr/bin/launch_awx.sh"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 7177,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-04-26T14:15:06.853577243Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        

docker@minikube:~$ docker logs 3a71d144c201
<--- skipped --->
2021-04-26 15:35:02,991 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:02,991 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:03,994 INFO spawned: 'wsbroadcast' with pid 2425
2021-04-26 15:35:03,994 INFO spawned: 'wsbroadcast' with pid 2425
2021-04-26 15:35:04,996 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:04,996 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:07,130 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2021-04-26 15:35:12,818 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:12,818 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:13,821 INFO spawned: 'wsbroadcast' with pid 2429
2021-04-26 15:35:13,821 INFO spawned: 'wsbroadcast' with pid 2429
2021-04-26 15:35:14,823 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:14,823 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:16,409 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2021-04-26 15:35:23,131 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:23,131 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:24,134 INFO spawned: 'wsbroadcast' with pid 2433
2021-04-26 15:35:24,134 INFO spawned: 'wsbroadcast' with pid 2433
2021-04-26 15:35:25,136 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:25,136 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:26,844 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2021-04-26 15:35:33,469 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:33,469 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:34,472 INFO spawned: 'wsbroadcast' with pid 2437
2021-04-26 15:35:34,472 INFO spawned: 'wsbroadcast' with pid 2437
2021-04-26 15:35:35,474 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:35,474 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:37,219 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2021-04-26 15:35:42,775 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:42,775 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:43,783 INFO spawned: 'wsbroadcast' with pid 2441
2021-04-26 15:35:43,783 INFO spawned: 'wsbroadcast' with pid 2441
2021-04-26 15:35:44,785 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:44,785 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:46,530 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2021-04-26 15:35:52,140 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:52,140 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:35:53,144 INFO spawned: 'wsbroadcast' with pid 2445
2021-04-26 15:35:53,144 INFO spawned: 'wsbroadcast' with pid 2445
2021-04-26 15:35:54,145 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:54,145 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:35:55,837 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2021-04-26 15:36:01,508 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:36:01,508 INFO exited: wsbroadcast (exit status 0; expected)
RESULT 2
OKREADY
2021-04-26 15:36:01,512 INFO spawned: 'wsbroadcast' with pid 2449
2021-04-26 15:36:01,512 INFO spawned: 'wsbroadcast' with pid 2449
2021-04-26 15:36:02,516 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:36:02,516 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:36:04,447 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2021-04-26 15:36:11,161 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:36:11,161 INFO exited: wsbroadcast (exit status 0; expected)
2021-04-26 15:36:12,165 INFO spawned: 'wsbroadcast' with pid 2453
2021-04-26 15:36:12,165 INFO spawned: 'wsbroadcast' with pid 2453
2021-04-26 15:36:13,167 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:36:13,167 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-26 15:36:14,843 ERROR    [-] awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...