AWX UI returns "500 A server error has occurred", appears to be a RabbitMQ issue #3897

ea-consulting · 2019-05-15T04:04:23Z

ISSUE TYPE

Bug Report

SUMMARY

I am able to log into AWX and complete several actions within the UI without issues; however, I am also getting, for example, Call to /api/v2/.../... failed. DELETE returned status: 500. A server error has occurred. and Failed to save settings. Returned status: 500 A server error has occurred. error messages when completing several actions within the UI.

ENVIRONMENT

AWX version: 4.0.0 (https://github.com/ansible/awx)
AWX install method: Docker on Linux (Docker version 18.09.6, build 481bc77156)
Ansible version: 2.7.10
Operating System: CentOS Linux 7 (AWS EC2 AMI)
Web Browser: Google Chrome Version 74.0.3729.157

STEPS TO REPRODUCE

Some of several examples...

I attempted to delete the "Demo Project" within the UI and the Call to /api/v2/projects/4/ failed. DELETE returned status: 500. A server error has occurred. error message is returned after confirming the delete.
I attempted to delete the "Default" organization within the UI and the Call to /api/v2/organizations/1/ failed. DELETE returned status: 500 A server error has occurred. error message is returned after confirming the delete.
I attempted to update the "Base URL of Tower Host" under the Settings section within the UI and the Failed to save settings. Returned status: 500 A server error has occurred. error message is returned after clicking save.

EXPECTED RESULTS

Things to work when the action is submitted within the UI.

ACTUAL RESULTS

No action is completed, 500. A server error has occurred. error messages returned.

ADDITIONAL INFORMATION

Running Docker hosts for AWS, along with Nginx and Portainer:

[root@ip-10-0-0-91 my_nginx_build]# docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                                              NAMES
f62dc2c98bd3        my-nginx                      "nginx -g 'daemon of…"   39 seconds ago      Up 38 seconds       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:9000->9000/tcp   my-nginx
96a97b7fabf5        ansible/awx_task:latest       "/tini -- /bin/sh -c…"   About an hour ago   Up 23 minutes       8052/tcp                                                           awx_task
c3a20ccfc1ce        ansible/awx_web:latest        "/tini -- /bin/sh -c…"   About an hour ago   Up About an hour    0.0.0.0:8080->8052/tcp                                             awx_web
fdb255650c5d        ansible/awx_rabbitmq:3.6.14   "/bin/sh -c /launch.…"   About an hour ago   Up About an hour    4369/tcp, 5671-5672/tcp, 15672/tcp, 25672/tcp                      awx_rabbitmq
5f43482136d6        portainer/portainer           "/portainer"             3 hours ago         Up 3 hours          9000/tcp                                                           portainer
0fe14879c1de        postgres:9.6                  "docker-entrypoint.s…"   4 days ago          Up 3 days           5432/tcp                                                           awx_postgres
3700cae1151f        memcached:alpine              "docker-entrypoint.s…"   4 days ago          Up 3 days           11211/tcp                                                          awx_memcached

Nginx settings running with "my-nginx" container:

[root@ip-10-0-0-91 installer]# cat ~/my_nginx_build/nginx.conf
user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    server {
        listen                 80 default_server;
        listen                 443 ssl default_server;
        server_name            _;

        ssl_certificate        /etc/ssl/nginx.crt;
        ssl_certificate_key    /etc/ssl/nginx.key;

        return                 444;
    }

    server {
        listen      80;
        server_name awx.example.info;

        rewrite     ^  https://$host$request_uri? permanent;
    }

    server {
        listen              443;
        server_name         awx.example.info;

        ssl                 on;
        ssl_certificate     /etc/ssl/example.info.chained.crt;
        ssl_certificate_key /etc/ssl/example.info.key;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;

        location / {
            proxy_set_header    Host $host;
            proxy_pass          http://awx_web:8052;
            proxy_http_version  1.1;
            proxy_set_header    Upgrade $http_upgrade;
            proxy_set_header      Connection "upgrade";
        }
    }

    server {
        listen              9000;
        server_name         portainer.example.info;

        ssl                 on;
        ssl_certificate     /etc/ssl/example.info.chained.crt;
        ssl_certificate_key /etc/ssl/example.info.key;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;

        location / {
            proxy_set_header    Host $host;
            proxy_pass          http://portainer:9000;
            proxy_http_version  1.1;
            proxy_set_header    Upgrade $http_upgrade;
            proxy_set_header      Connection "upgrade";
        }
    }
}

Docker logs for "awx_web" comntainer:

[root@ip-10-0-0-91 installer]# docker logs --tail 10 awx_web
2019-05-15 03:44:01,715 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:06,731 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:11,740 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:16,753 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:21,767 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:26,783 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:31,792 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:36,819 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:44:41,834 ERROR    Error trying to receive messages: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.

Docker logs for "awx_task" container:

  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/connection.py", line 501, in on_inbound_method
    method_sig, payload, content,
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/abstract_channel.py", line 128, in dispatch_method
    listener(*args)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/connection.py", line 623, in _on_close
    (class_id, method_id), ConnectionError)
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2019-05-15 03:46:10,121 INFO exited: channels-worker (exit status 1; not expected)

Docker logs for "awx_rabbitmq" container:

[root@ip-10-0-0-91 installer]# docker logs --tail 10 awx_rabbitmq

=INFO REPORT==== 15-May-2019::03:46:37 ===
accepting AMQP connection <0.21630.1> (172.18.0.5:39446 -> 172.18.0.6:5672)

=ERROR REPORT==== 15-May-2019::03:46:37 ===
Error on AMQP connection <0.21630.1> (172.18.0.5:39446 -> 172.18.0.6:5672, state: starting):
AMQPLAIN login refused: user 'guest' can only connect via localhost

=INFO REPORT==== 15-May-2019::03:46:37 ===
closing AMQP connection <0.21630.1> (172.18.0.5:39446 -> 172.18.0.6:5672)

The text was updated successfully, but these errors were encountered:

ea-consulting · 2019-05-24T23:32:48Z

Any help would be greatly appreciated! Thanks!

dipeshthakur · 2019-05-28T08:50:51Z

Even , i'm facing similar issue . any help would be appreciate

Thanks

konkolorado · 2019-05-28T14:44:32Z

Looks like the loopback_users setting didn't make it into your RabbitMQ configuration. Can you check that it's set to none or false?
https://www.rabbitmq.com/access-control.html#loopback-users

wwolkers · 2019-07-01T13:59:17Z

I had the same problem and I can confirm that the loopback_users setting was the problem, it was not in the rabbitmq config file.

ea-consulting · 2019-07-02T04:43:33Z

I will take a look at the loopback_users setting in the next day or two and update the issue with the outcome. Thanks.

AmmeonSupport · 2019-07-09T11:43:51Z

Hi, for me the loopback_users are as false.
However in the logs shows the guest credentials are invalid. Is there an issues with special characters ? Checking the config files the password match in both rabbitmq.conf and credentials.py

snerge · 2019-07-11T11:55:23Z

I run 4.0.0 without any issue but I do get the same problem when I upgrade from 4.0.0 to 5.0.0 or 6.0.0 in case that can help pin point the source of the issue.

snerge · 2019-07-11T12:06:26Z

Hi, for me the loopback_users are as false.
However in the logs shows the guest credentials are invalid. Is there an issues with special characters ? Checking the config files the password match in both rabbitmq.conf and credentials.py

I have just removed special characters from my rabbitmq_password and now I'm able to upgrade to 5.0.0 and I was able to upgrade to 6.0.0 after.

axeljacobs · 2019-11-26T17:30:33Z

I just installed v 9.0.1 I had the same issue. I indeed had an exclamation point in the rabbitmq_password and in pg_password. I did remove the exclamation point '!' from both password an re ran the instalation playbook. Now it's running without issue.

AechLyons · 2019-12-12T13:55:06Z

I did have an exclamation mark in the default admin account for awx and I removed it and reran the yaml file......same error

de1ayer · 2020-03-17T14:12:46Z

Same error when I'm trying to cancel running job

Call to /api/v2/jobs/57/ failed. returned status: 500. A server error has occurred.

No special symbols in any passwords

de1ayer · 2020-03-17T15:10:38Z

Update: after upgrading rabbitmq-server to 3.6.16 (by default was version 3.3.5) there are no more errors, jobs are cancelling smoothly.
I've found this workaround at issue MrMEEE/awx-build#152 (comment)

ryanpetrello · 2020-03-17T15:30:16Z

@de1ayer (and others) for what it's worth, we're unlikely to address this given:

#5443
https://groups.google.com/forum/#!topic/awx-project/lRnm2vB1oEQ

awxbot added state:needs_triage type:bug labels May 15, 2019

ryanpetrello removed the state:needs_triage label Jun 28, 2019

awithrow9 added the help wanted label Jun 28, 2019

ryanpetrello closed this as completed Mar 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWX UI returns "500 A server error has occurred", appears to be a RabbitMQ issue #3897

AWX UI returns "500 A server error has occurred", appears to be a RabbitMQ issue #3897

ea-consulting commented May 15, 2019

ea-consulting commented May 24, 2019

dipeshthakur commented May 28, 2019

konkolorado commented May 28, 2019

wwolkers commented Jul 1, 2019

ea-consulting commented Jul 2, 2019

AmmeonSupport commented Jul 9, 2019

snerge commented Jul 11, 2019

snerge commented Jul 11, 2019 •

edited

Loading

axeljacobs commented Nov 26, 2019

AechLyons commented Dec 12, 2019

de1ayer commented Mar 17, 2020 •

edited

Loading

de1ayer commented Mar 17, 2020

ryanpetrello commented Mar 17, 2020

AWX UI returns "500 A server error has occurred", appears to be a RabbitMQ issue #3897

AWX UI returns "500 A server error has occurred", appears to be a RabbitMQ issue #3897

Comments

ea-consulting commented May 15, 2019

ISSUE TYPE

SUMMARY

ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ADDITIONAL INFORMATION

ea-consulting commented May 24, 2019

dipeshthakur commented May 28, 2019

konkolorado commented May 28, 2019

wwolkers commented Jul 1, 2019

ea-consulting commented Jul 2, 2019

AmmeonSupport commented Jul 9, 2019

snerge commented Jul 11, 2019

snerge commented Jul 11, 2019 • edited Loading

axeljacobs commented Nov 26, 2019

AechLyons commented Dec 12, 2019

de1ayer commented Mar 17, 2020 • edited Loading

de1ayer commented Mar 17, 2020

ryanpetrello commented Mar 17, 2020

snerge commented Jul 11, 2019 •

edited

Loading

de1ayer commented Mar 17, 2020 •

edited

Loading