Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

job-status-consumer: better catch exceptions #423

Open
tiborsimko opened this issue Nov 22, 2021 · 0 comments
Open

job-status-consumer: better catch exceptions #423

tiborsimko opened this issue Nov 22, 2021 · 0 comments

Comments

@tiborsimko
Copy link
Member

Seen when testing many pMSSM workflows:

$ kubectl logs deployment/reana-workflow-controller job-status-consumer
...
2021-11-22 11:18:15,651 | root | MainThread | INFO |  [x] Received workflow_uuid: cbeca699-418c-4084-911f-23ba1a62c6b0 status: RunStatus.running
2021-11-22 11:18:15,691 | root | MainThread | INFO |  [x] Received workflow_uuid: b808972f-9071-4826-b9a6-b7c4353e55d8 status: RunStatus.running
2021-11-22 11:18:15,725 | root | MainThread | INFO |  [x] Received workflow_uuid: 4a5fd15c-8c1b-4a8d-8cfc-f3593c74f68a status: RunStatus.running
2021-11-22 11:18:15,760 | root | MainThread | INFO |  [x] Received workflow_uuid: dcf8d5bf-0d90-4a24-8891-a5e874c39700 status: RunStatus.running
2021-11-22 11:18:15,798 | root | MainThread | INFO |  [x] Received workflow_uuid: 017eb1f1-f732-471a-9637-afff257a5ba1 status: RunStatus.failed
2021-11-22 11:18:17,313 | root | MainThread | INFO |  [x] Received workflow_uuid: ad980370-ca20-4d5b-a5d9-eff33b9c7eaf status: RunStatus.running
2021-11-22 11:18:17,357 | root | MainThread | INFO |  [x] Received workflow_uuid: d6c956dd-5812-4173-a822-a6d5fffcd4f8 status: RunStatus.finished
2021-11-22 11:18:21,448 | root | MainThread | ERROR | Unexpected error while processing workflow: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/reana_workflow_controller/consumer.py", line 82, in on_message
    message.ack()
  File "/usr/local/lib/python3.8/site-packages/kombu/message.py", line 126, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "/usr/local/lib/python3.8/site-packages/amqp/channel.py", line 1393, in basic_ack
    return self.send_method(
  File "/usr/local/lib/python3.8/site-packages/amqp/abstract_channel.py", line 59, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/usr/local/lib/python3.8/site-packages/amqp/method_framing.py", line 189, in write_frame
    write(view[:offset])
  File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 305, in write
    self._write(s)
ConnectionResetError: [Errno 104] Connection reset by peer

Can this exception be caught and acted upon?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant