Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR on install with docker #767

Closed
matthiasdoering opened this issue Dec 4, 2017 · 6 comments
Closed

ERROR on install with docker #767

matthiasdoering opened this issue Dec 4, 2017 · 6 comments

Comments

@matthiasdoering
Copy link

matthiasdoering commented Dec 4, 2017

ISSUE TYPE
  • Bug Report
COMPONENT NAME
  • Installer
SUMMARY

I get the whole error again and again after install out of the installer directory with ansible-playbook -i inventory install.yml

ENVIRONMENT
  • AWX version:
    git log
    commit cf0cc2e
    Author: Matthew Jones [email protected]
    Date: Mon Dec 4 07:56:34 2017 -0500

    Add system requirements to install docs

  • AWX install method: docker on linux

  • Ansible version: 2.4.2.0

  • Operating System: Ubuntu Xenial

  • Web Browser:

STEPS TO REPRODUCE
[2017-12-04 22:44:41,184: ERROR/MainProcess] Control command error: error(104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/pidbox.py", line 42, in on_message
    self.node.handle_message(body, message)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message
    return self.dispatch(**body)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch
    ticket=ticket)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply
    serializer=self.mailbox.serializer)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply
    **opts
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py", line 1734, in _basic_publish
    (0, exchange, routing_key, mandatory, immediate), msg
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/transport.py", line 258, in write
    self._write(s)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer
[2017-12-04 22:44:41,197: DEBUG/MainProcess] Closed channel #3
[2017-12-04 22:44:41,197: DEBUG/MainProcess] using channel_id: 3
[2017-12-04 22:44:41,198: DEBUG/MainProcess] Channel open
2017-12-04 22:44:42,209 ERROR    awx.main.scheduler Failed to retrieve active tasks from celery
2017-12-04 22:44:42,209 ERROR    awx.main.scheduler Failed to retrieve active tasks from celery
[2017-12-04 22:44:42,209: ERROR/ForkPoolWorker-200] Failed to retrieve active tasks from celery
EXPECTED RESULTS

When I tail the log i won't see such erros

ACTUAL RESULTS

error loop

ADDITIONAL INFORMATION

/

@worsco
Copy link

worsco commented Dec 5, 2017

+1
Issue persists in 1.0.1.283, docker container deployment.
I'm exporting logs to loggly and this error gets logged every 6 minutes, or so.

{"cluster_host_id": "awx", "level": "ERROR", "@timestamp": "2017-12-05T14:13:27.539Z", "host": "awx", "logger_name": "awx.main.scheduler", "message": "Failed to retrieve active tasks from celery", "type": "loggly"}
{"cluster_host_id": "awx", "level": "ERROR", "@timestamp": "2017-12-05T14:06:46.955Z", "host": "awx", "logger_name": "awx.main.scheduler", "message": "Failed to retrieve active tasks from celery", "type": "loggly"}
{"cluster_host_id": "awx", "level": "ERROR", "@timestamp": "2017-12-05T14:00:07.226Z", "host": "awx", "logger_name": "awx.main.scheduler", "message": "Failed to retrieve active tasks from celery", "type": "loggly"}
{"cluster_host_id": "awx", "level": "ERROR", "@timestamp": "2017-12-05T13:53:28.431Z", "host": "awx", "logger_name": "awx.main.scheduler", "message": "Failed to retrieve active tasks from celery", "type": "loggly"}

@worsco
Copy link

worsco commented Dec 6, 2017

Issue persists in 1.0.1.290, docker container deployment. Logs error about every 6 minutes 30 seconds.

@Aletor93
Copy link

Aletor93 commented Dec 7, 2017

Hi,

I followed given steps by @matburt & @knechtionscoding (#116) but I'm still having the same issue :

1.Stop each container in descending order:
•awx_task
•awx_web
•memcached
•rabbitmq
•postgres
2.Remove /tmp/pgdocker/
3.Re-run install.yml with no other changes.

ENVIRONMENT

  • OS : RHEL 7.3
  • Docker : 1.12.6-32
  • Python Docker : 1.10.6-1
  • Ansible version : 2.4.1.0-1
  • AWX commit : 7fc896e
  • Web browser : IE11 & FF40

SUMMARY
As the author, I'm getting errors like "Control command error: error(104, 'Connection reset by peer')".
Web UI is blank (nothing showed) but I can see favicon so it's partially working.
I should not use root to install AWX, I'll retry with a different user tomorrow.
I follow this guide http://khmel.org/?p=1245 with some customisation due to our proxy server.

LOGS

Using /etc/ansible/ansible.cfg as config file
[DEPRECATION WARNING]: The sudo command line option has been deprecated in 
favor of the "become" command line arguments. This feature will be removed in 
version 2.6. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
127.0.0.1 | SUCCESS => {
    "changed": true, 
    "db": "awx"
}
Operations to perform:
  Apply all migrations: auth, conf, contenttypes, django_celery_results, main, sessions, sites, social_django, sso, taggit
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying taggit.0001_initial... OK
  Applying taggit.0002_auto_20150616_2121... OK
  Applying main.0001_initial... OK
  Applying main.0002_squashed_v300_release... OK
  Applying main.0003_squashed_v300_v303_updates... OK
  Applying main.0004_squashed_v310_release... OK
  Applying conf.0001_initial... OK
  Applying conf.0002_v310_copy_tower_settings... OK
  Applying conf.0003_v310_JSONField_changes... OK
  Applying conf.0004_v320_reencrypt... OK
  Applying django_celery_results.0001_initial... OK
  Applying main.0005_squashed_v310_v313_updates... OK
  Applying main.0005a_squashed_v310_v313_updates... OK
  Applying main.0005b_squashed_v310_v313_updates... OK
  Applying main.0006_v320_release... OK
2017-12-07 15:47:27,829 DEBUG    awx.main.migrations Removing all Rackspace InventorySource from database.
2017-12-07 15:47:28,315 DEBUG    awx.main.migrations Removing all Azure Credentials from database.
2017-12-07 15:47:28,800 DEBUG    awx.main.migrations Removing all Azure InventorySource from database.
2017-12-07 15:47:29,286 DEBUG    awx.main.migrations Removing all InventorySource that have no link to an Inventory from database.
  Applying main.0007_v320_data_migrations... OK
  Applying main.0008_v320_drop_v1_credential_fields... OK
  Applying main.0009_v330_multi_credential... OK
  Applying sessions.0001_initial... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
  Applying social_django.0001_initial... OK
  Applying social_django.0002_add_related_name... OK
  Applying social_django.0003_alter_email_max_length... OK
  Applying social_django.0004_auto_20160423_0400... OK
  Applying social_django.0005_auto_20160727_2333... OK
  Applying social_django.0006_partial... OK
  Applying social_django.0007_code_timestamp... OK
  Applying social_django.0008_partial_timestamp... OK
  Applying sso.0001_initial... OK
  Applying sso.0002_expand_provider_options... OK
Default organization added.
Demo Credential, Inventory, and Job Template added.
Successfully registered instance awx
(changed: True)
Creating instance group tower
Added instance awx to tower
(changed: True)
2017-12-07 15:48:08,896 CRIT Supervisor running as root (no user in config file)
2017-12-07 15:48:08,903 WARN For [program:awx-celeryd-beat], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
Unlinking stale socket /tmp/supervisor.sock
2017-12-07 15:48:09,215 INFO RPC interface 'supervisor' initialized
2017-12-07 15:48:09,215 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2017-12-07 15:48:09,222 INFO supervisord started with pid 92
2017-12-07 15:48:10,225 INFO spawned: 'celery' with pid 95
2017-12-07 15:48:10,227 INFO spawned: 'channels-worker' with pid 96
2017-12-07 15:48:10,228 INFO spawned: 'callback-receiver' with pid 97
2017-12-07 15:48:10,229 INFO spawned: 'awx-celeryd-beat' with pid 98
2017-12-07 15:48:11,230 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-12-07 15:48:11,230 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-12-07 15:48:11,230 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-12-07 15:48:11,230 INFO success: awx-celeryd-beat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-12-07 15:48:12,959 INFO     awx.main.tasks Syncing Schedules
2017-12-07 15:48:12,959 INFO     awx.main.tasks Syncing Schedules
[2017-12-07 15:48:12,961: DEBUG/MainProcess] Setting default socket timeout to 30
[2017-12-07 15:48:12,961: INFO/MainProcess] beat: Starting...
2017-12-07 15:48:12,996 - INFO - runworker - Using single-threaded worker.
2017-12-07 15:48:12,996 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2017-12-07 15:48:12,996 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the -u option.
User information: uid=0 euid=0 gid=0 egid=0
  uid=uid, euid=euid, gid=gid, egid=egid,
[2017-12-07 15:48:13,147: DEBUG/MainProcess] Current schedule:







[2017-12-07 15:48:13,147: DEBUG/MainProcess] beat: Ticking with max interval->1.00 minute
[2017-12-07 15:48:13,163: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2017 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@bdd1e58a1ebe', 'platform': 'Erlang/OTP 19.2.1', 'version': '3.6.14'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US']
[2017-12-07 15:48:13,165: INFO/MainProcess] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2017-12-07 15:48:13,190: DEBUG/MainProcess] using channel_id: 1
[2017-12-07 15:48:13,190: DEBUG/MainProcess] Channel open
[2017-12-07 15:48:13,193: DEBUG/MainProcess] beat: Synchronizing schedule...
[2017-12-07 15:48:13,205: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-12-07 15:48:13,207: DEBUG/MainProcess] | Worker: Building graph...
[2017-12-07 15:48:13,208: DEBUG/MainProcess] | Worker: New boot order: {Beat, Timer, Hub, Pool, Autoscaler, StateDB, Consumer}
[2017-12-07 15:48:13,211: DEBUG/MainProcess] awx.main.scheduler.tasks.run_task_manager sent. id->8b7ccc91-cc9e-46ae-afff-3d8697a07fdb
[2017-12-07 15:48:13,212: DEBUG/MainProcess] beat: Waking up in 19.93 seconds.
[2017-12-07 15:48:13,254: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-12-07 15:48:13,254: DEBUG/MainProcess] | Consumer: Building graph...
[2017-12-07 15:48:13,269: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Heart, Agent, Tasks, Control, event loop}
[2017-12-07 15:48:13,284: DEBUG/MainProcess] | Worker: Starting Hub
[2017-12-07 15:48:13,284: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:13,284: DEBUG/MainProcess] | Worker: Starting Pool
[2017-12-07 15:48:13,285: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:13,285: DEBUG/MainProcess] | Worker: Starting Autoscaler
[2017-12-07 15:48:13,285: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:13,285: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-12-07 15:48:13,285: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-12-07 15:48:13,300: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2017 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@bdd1e58a1ebe', 'platform': 'Erlang/OTP 19.2.1', 'version': '3.6.14'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US']
[2017-12-07 15:48:13,301: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672/awx
[2017-12-07 15:48:13,301: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:13,301: DEBUG/MainProcess] | Consumer: Starting Events
[2017-12-07 15:48:13,315: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2017 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@bdd1e58a1ebe', 'platform': 'Erlang/OTP 19.2.1', 'version': '3.6.14'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US']
[2017-12-07 15:48:13,317: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:13,317: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-12-07 15:48:13,317: INFO/MainProcess] mingle: searching for neighbors
[2017-12-07 15:48:13,317: DEBUG/MainProcess] using channel_id: 1
[2017-12-07 15:48:13,318: DEBUG/MainProcess] Channel open
[2017-12-07 15:48:13,340: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2017 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@bdd1e58a1ebe', 'platform': 'Erlang/OTP 19.2.1', 'version': '3.6.14'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US']
[2017-12-07 15:48:13,340: DEBUG/MainProcess] using channel_id: 1
[2017-12-07 15:48:13,341: DEBUG/MainProcess] Channel open
[2017-12-07 15:48:14,346: INFO/MainProcess] mingle: all alone
[2017-12-07 15:48:14,347: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:14,347: DEBUG/MainProcess] | Consumer: Starting Gossip
[2017-12-07 15:48:14,347: DEBUG/MainProcess] using channel_id: 2
[2017-12-07 15:48:14,347: DEBUG/MainProcess] Channel open
[2017-12-07 15:48:14,352: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:14,352: DEBUG/MainProcess] | Consumer: Starting Heart
[2017-12-07 15:48:14,353: DEBUG/MainProcess] using channel_id: 1
[2017-12-07 15:48:14,353: DEBUG/MainProcess] Channel open
[2017-12-07 15:48:14,354: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:14,354: DEBUG/MainProcess] | Consumer: Starting Tasks
[2017-12-07 15:48:14,391: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:14,391: DEBUG/MainProcess] | Consumer: Starting Control
[2017-12-07 15:48:14,391: DEBUG/MainProcess] using channel_id: 3
[2017-12-07 15:48:14,392: DEBUG/MainProcess] Channel open
[2017-12-07 15:48:14,396: DEBUG/MainProcess] ^-- substep ok
[2017-12-07 15:48:14,396: DEBUG/MainProcess] | Consumer: Starting event loop
[2017-12-07 15:48:14,396: DEBUG/MainProcess] | Worker: Hub.register Autoscaler...
[2017-12-07 15:48:14,396: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2017-12-07 15:48:14,399: INFO/MainProcess] Received task: awx.main.tasks.update_inventory_computed_fields[d7b022b9-38ab-4cd7-abcb-a0de5ddf01e8]  
[2017-12-07 15:48:14,400: INFO/MainProcess] Scaling up 1 processes.
[2017-12-07 15:48:14,502: DEBUG/MainProcess] TaskPool: Apply  (args:('awx.main.tasks.update_inventory_computed_fields', 'd7b022b9-38ab-4cd7-abcb-a0de5ddf01e8', {'origin': 'gen57@awx', 'lang': 'py', 'task': 'awx.main.tasks.update_inventory_computed_fields', 'group': None, 'root_id': 'd7b022b9-38ab-4cd7-abcb-a0de5ddf01e8', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': 'd7b022b9-38ab-4cd7-abcb-a0de5ddf01e8', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(1, True)', 'eta': None, 'parent_id': None, u'reply_to': '75e336a2-afb7-3233-b982-0acf56a5da6f', 'id': 'd7b022b9-38ab-4cd7-abcb-a0de5ddf01e8', 'kwargsrepr': '{}'}, u'[[1, true], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2017-12-07 15:48:14,504: DEBUG/ForkPoolWorker-1] Closed channel #1
[2017-12-07 15:48:14,508: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8b7ccc91-cc9e-46ae-afff-3d8697a07fdb]   expires:[2017-12-07 15:48:33.171242+00:00]
[2017-12-07 15:48:14,508: INFO/MainProcess] Scaling up 1 processes.
[2017-12-07 15:48:14,602: DEBUG/MainProcess] TaskPool: Apply  (args:('awx.main.scheduler.tasks.run_task_manager', '8b7ccc91-cc9e-46ae-afff-3d8697a07fdb', {'origin': 'gen98@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': '8b7ccc91-cc9e-46ae-afff-3d8697a07fdb', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2017-12-07T15:48:33.171242+00:00', u'correlation_id': '8b7ccc91-cc9e-46ae-afff-3d8697a07fdb', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': 'c09686be-962f-3f9d-8c78-e5f5a6d37236', 'id': '8b7ccc91-cc9e-46ae-afff-3d8697a07fdb', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2017-12-07 15:48:14,604: INFO/MainProcess] celery@localhost ready.
[2017-12-07 15:48:14,604: DEBUG/MainProcess] basic.qos: prefetch_count->12
[2017-12-07 15:48:14,707: DEBUG/MainProcess] Task accepted: awx.main.tasks.update_inventory_computed_fields[d7b022b9-38ab-4cd7-abcb-a0de5ddf01e8] pid:140
[2017-12-07 15:48:14,707: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[8b7ccc91-cc9e-46ae-afff-3d8697a07fdb] pid:139
2017-12-07 15:48:14,722 DEBUG    awx.main.scheduler Running Tower task manager.
2017-12-07 15:48:14,722 DEBUG    awx.main.scheduler Running Tower task manager.
[2017-12-07 15:48:14,722: DEBUG/ForkPoolWorker-1] Running Tower task manager.
2017-12-07 15:48:14,729 DEBUG    awx.main.scheduler Starting Scheduler
2017-12-07 15:48:14,729 DEBUG    awx.main.scheduler Starting Scheduler
[2017-12-07 15:48:14,729: DEBUG/ForkPoolWorker-1] Starting Scheduler
2017-12-07 15:48:14,731 DEBUG    awx.main.scheduler Failing inconsistent running jobs.
2017-12-07 15:48:14,731 DEBUG    awx.main.scheduler Failing inconsistent running jobs.
[2017-12-07 15:48:14,731: DEBUG/ForkPoolWorker-1] Failing inconsistent running jobs.
2017-12-07 15:48:14,733 DEBUG    awx.main.models.inventory Going to update inventory computed fields
[2017-12-07 15:48:14,733: DEBUG/ForkPoolWorker-2] Going to update inventory computed fields
[2017-12-07 15:48:14,756: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2017 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@bdd1e58a1ebe', 'platform': 'Erlang/OTP 19.2.1', 'version': '3.6.14'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US']
[2017-12-07 15:48:14,757: DEBUG/ForkPoolWorker-1] using channel_id: 1
[2017-12-07 15:48:14,758: DEBUG/ForkPoolWorker-1] Channel open
[2017-12-07 15:48:14,775: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2017 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@bdd1e58a1ebe', 'platform': 'Erlang/OTP 19.2.1', 'version': '3.6.14'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US']
[2017-12-07 15:48:14,776: DEBUG/ForkPoolWorker-1] using channel_id: 1
[2017-12-07 15:48:14,777: DEBUG/ForkPoolWorker-1] Channel open
[2017-12-07 15:48:14,779: DEBUG/MainProcess] pidbox received method active() [reply_to:{u'routing_key': u'6eedccd3-afda-376f-ba15-aa306ff2623f', u'exchange': u'reply.celery.pidbox'} ticket:a85543ef-c005-478f-98d2-dd51df7449ba]
[2017-12-07 15:48:14,780: ERROR/MainProcess] Control command error: error(104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/pidbox.py", line 42, in on_message
    self.node.handle_message(body, message)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message
    return self.dispatch(**body)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch
    ticket=ticket)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply
    serializer=self.mailbox.serializer)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply
    **opts
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 194, in _publish
    [maybe_declare(entity) for entity in declare]
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 102, in maybe_declare
    return maybe_declare(entity, self.channel, retry, **retry_policy)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/common.py", line 129, in maybe_declare
    return _maybe_declare(entity, declared, ident, channel, orig)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/common.py", line 135, in _maybe_declare
    entity.declare(channel=channel)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/entity.py", line 185, in declare
    nowait=nowait, passive=passive,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py", line 616, in exchange_declare
    wait=None if nowait else spec.Exchange.DeclareOk,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/transport.py", line 258, in write
    self._write(s)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer
[2017-12-07 15:48:14,838: DEBUG/MainProcess] Closed channel #3
[2017-12-07 15:48:14,838: DEBUG/MainProcess] using channel_id: 3
[2017-12-07 15:48:14,839: DEBUG/MainProcess] Channel open
2017-12-07 15:48:14,855 DEBUG    awx.main.models.inventory Finished updating inventory computed fields
[2017-12-07 15:48:14,855: DEBUG/ForkPoolWorker-2] Finished updating inventory computed fields
[2017-12-07 15:48:14,865: INFO/ForkPoolWorker-2] Task awx.main.tasks.update_inventory_computed_fields[d7b022b9-38ab-4cd7-abcb-a0de5ddf01e8] succeeded in 0.158185196s: None
2017-12-07 15:48:15,783 ERROR    awx.main.scheduler Failed to retrieve active tasks from celery
2017-12-07 15:48:15,783 ERROR    awx.main.scheduler Failed to retrieve active tasks from celery
[2017-12-07 15:48:15,783: ERROR/ForkPoolWorker-1] Failed to retrieve active tasks from celery
[2017-12-07 15:48:15,824: INFO/ForkPoolWorker-1] Task awx.main.scheduler.tasks.run_task_manager[8b7ccc91-cc9e-46ae-afff-3d8697a07fdb] succeeded in 1.11806831s: None

Regards,

@Aletor93
Copy link

Aletor93 commented Dec 8, 2017

I tried to remove all docker's images, volumes and rerun the whole playbook from scratch with a dedicated user (not root, but I had to run the playbook with sudo) and I'm having the exact same issue.

Regards,

@silverskysoft
Copy link

This seems like the issue reported in #689. Have you gotten a chance to ensure the minimum requirements as discussed there?

@matthiasdoering
Copy link
Author

Yes, let us link this issue to #689

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants