Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unhandled exception in socket server rejected WebSocket connection: HTTP 404 #1024

Closed
jayhuynh opened this issue Jun 18, 2023 · 7 comments · Fixed by #1025
Closed

Unhandled exception in socket server rejected WebSocket connection: HTTP 404 #1024

jayhuynh opened this issue Jun 18, 2023 · 7 comments · Fixed by #1025

Comments

@jayhuynh
Copy link

Issue Description

When I try to deploy my task with heroku + mturk sandbox it keeps thowring Unhandled exception in socket server rejected WebSocket connection: HTTP 404

Already turn on debug flag but can't see much information

Expected Behavior

I worked fine a week or two ago. This is very simple task so I would expect it should be fine this time\

Current Behavior

Keep throwing WebSocket error 404 when sending alive after heroku Scaling dynos step

Steps to Reproduce

  1. python3 deploy.py

Screenshots

bash-5.1# python3 ../deploy.py 
/usr/lib/python3.9/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
[06/18/23 14:44:35] INFO     Prometheus server launched at process 113                                                                                                                                                                                     metrics.py:185

up to date, audited 783 packages in 12s

148 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> [email protected] dev
> webpack --mode development


🌼 daisyUI components 2.52.0  https://daisyui.com
  ✔︎ Including:  base, components, 29 themes, utilities
  ❤︎ Support daisyUI:  https://opencollective.com/daisyui 
  
asset build/bundle.js 1.4 MiB [compared for emit] (name: main)
runtime modules 1.07 KiB 6 modules
modules by path ./ 1.19 MiB
  modules by path ./node_modules/ 1.13 MiB 12 modules
  modules by path ./src/ 63.2 KiB
    modules by path ./src/css/*.css 55.4 KiB 2 modules
    ./src/main.js 230 bytes [built] [code generated]
    ./src/app.jsx 4.9 KiB [built] [code generated]
    ./src/components/core_components.jsx 2.7 KiB [built] [code generated]
modules by path ../../packages/ 156 KiB
  ../../packages/mephisto-task/build/bundle.js 134 KiB [built] [code generated]
  ../../packages/annotated/annotator-tracker/build/bundle.js 22.1 KiB [built] [code generated]
webpack 5.87.0 compiled successfully in 34243 ms
Locating heroku...
Getting heroku
/usr/local/bin/heroku
 ›   Warning: Our terms of service have changed: https://dashboard.heroku.com/terms-of-service
 ›   Warning: HEROKU_API_KEY is set
[06/18/23 14:46:00] INFO     Creating a task run under task name: s3-voting                                                                                                                                                                               operator.py:262
[06/18/23 14:46:01] DEBUG    type of assignment data: <class 'list'>                                                                                                                                                                                  task_launcher.py:86
Building server files...

up to date, audited 84 packages in 10s

7 packages are looking for funding
  run `npm fund` for details

3 high severity vulnerabilities

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
Locating heroku...
/usr/local/bin/heroku
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
 ›   Warning: HEROKU_API_KEY is set
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Heroku: Starting server Jay Huynh...
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Creating root-s3-voting-22-4a8629019e21... done
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Enabling http-session-affinity for root-s3-voting-22-4a8629019e21... done
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Scaling dynos... done, now running web at 1:Basic
[06/18/23 14:47:55] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:56] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:57] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:58] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:59] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:00] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:00] ERROR    Unhandled exception in socket server rejected WebSocket connection: HTTP 404, InvalidStatusCode(404, Headers([('Content-Length', '563'), ('Cache-Control', 'no-cache, no-store'), ('Content-Type', 'text/html;      websocket_channel.py:165
                             charset=utf-8'), ('Date', '2023-06-18 14:45:09.991494749 +0000 UTC'), ('Server', 'heroku-router')]))                                                                                                                                        
                             Traceback (most recent call last):                                                                                                                                                                                                          
                               File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 132, in run_socket                                                                                                                                  
                                 async with websockets.connect(                                                                                                                                                                                                          
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 642, in __aenter__                                                                                                                                              
                                 return await self                                                                                                                                                                                                                       
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__                                                                                                                                  
                                 return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)                                                                                                                                                                 
                               File "/usr/lib/python3.9/asyncio/tasks.py", line 479, in wait_for                                                                                                                                                                         
                                 return fut.result()                                                                                                                                                                                                                     
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 666, in __await_impl__                                                                                                                                          
                                 await protocol.handshake(                                                                                                                                                                                                               
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 332, in handshake                                                                                                                                               
                                 raise InvalidStatusCode(status_code, response_headers)                                                                                                                                                                                  
                             websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 404                                                                                                                                                     
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:01] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:02] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:03] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:04] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:05] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
Exception in thread socket-thread-wss://root-s3-voting-22-4a8629019e21.herokuapp.com/:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 185, in async_socket_wrap
    event_loop.run_until_complete(close_websocket())
  File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 183, in close_websocket
    await self._socket_task
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 168, in run_socket
    raise e
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 132, in run_socket
    async with websockets.connect(
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 642, in __aenter__
    return await self
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__
    return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 479, in wait_for
    return fut.result()
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 666, in __await_impl__
    await protocol.handshake(
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 332, in handshake
    raise InvalidStatusCode(status_code, response_headers)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 404
[06/18/23 14:48:05] ERROR    Encountered error while launching run, shutting down                                                                                                                                                                         operator.py:302
                             Traceback (most recent call last):                                                                                                                                                                                                          
                               File "/mephisto/mephisto/operations/operator.py", line 300, in launch_task_run_or_die                                                                                                                                                     
                                 live_run.client_io.launch_channels()                                                                                                                                                                                                    
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 224, in launch_channels                                                                                                                                                   
                                 self._register_channel(channel)                                                                                                                                                                                                         
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 200, in _register_channel                                                                                                                                                 
                                 raise ConnectionRefusedError(  # noqa F821 we only support py3                                                                                                                                                                          
                             ConnectionRefusedError: Was not able to establish a connection with the server, please try to run again. If that fails,please launch with mephisto.log_level=debug and watch for clear errors. If this doesn't help, feel                   
                             free to open an issue with your debug logs on the Mephisto github.                                                                                                                                                                          
Heroku: Deleting server: root-s3-voting-22-4a8629019e21
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Destroying root-s3-voting-22-4a8629019e21 (including all add-ons)... done
[06/18/23 14:48:17] ERROR    Ran into error while launching run:                                                                                                                                                                                          operator.py:522
                             Traceback (most recent call last):                                                                                                                                                                                                          
                               File "/mephisto/mephisto/operations/operator.py", line 518, in launch_task_run                                                                                                                                                            
                                 return self.launch_task_run_or_die(                                                                                                                                                                                                     
                               File "/mephisto/mephisto/operations/operator.py", line 312, in launch_task_run_or_die                                                                                                                                                     
                                 raise e                                                                                                                                                                                                                                 
                               File "/mephisto/mephisto/operations/operator.py", line 300, in launch_task_run_or_die                                                                                                                                                     
                                 live_run.client_io.launch_channels()                                                                                                                                                                                                    
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 224, in launch_channels                                                                                                                                                   
                                 self._register_channel(channel)                                                                                                                                                                                                         
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 200, in _register_channel                                                                                                                                                 
                                 raise ConnectionRefusedError(  # noqa F821 we only support py3                                                                                                                                                                          
                             ConnectionRefusedError: Was not able to establish a connection with the server, please try to run again. If that fails,please launch with mephisto.log_level=debug and watch for clear errors. If this doesn't help, feel                   
                             free to open an issue with your debug logs on the Mephisto github.                                                                                                                                                                          
                    INFO     operator shutting down                                                                                                                                                                                                       operator.py:413
[06/18/23 14:48:27] DEBUG    Prometheus server appears to be running at 9090                                                                                                                                                                               metrics.py:164
                    INFO     Prometheus server shut down

Environment

Additional Information

Possible Solution (Optional)

  • Maybe Heroku update there API then we might need to update our backend
@jayhuynh
Copy link
Author

Hi @JackUrb . Hope you are doing well.

Our team have been doing very great with Mephisto for a while but we just encounter this error that blocking us from deploying the task.

Hope you might have some ideas how to fix this issue as I have spent a few days on this but no luck at all

@jayhuynh
Copy link
Author

I also tested with freshly latest version of mephisto. Still throwing error 404

@JackUrb JackUrb mentioned this issue Jun 20, 2023
Closed
@JackUrb
Copy link
Contributor

JackUrb commented Jun 20, 2023

Hi @jayhuynh, this is certainly an issue! We haven't used Heroku extensively any time recently, so it's certainly possible something has changed. One thing you may do - if you could try to open the heroku logs on heroku when the server launches. You may want to add an input statement near the "Encountered error while launching run, shutting down" throw to ensure the heroku server stays up.

As Heroku's free model was removed, we've been looking to onboard alternatives in #1010 (fly.io) and also AWS App Runner.

@JackUrb
Copy link
Contributor

JackUrb commented Jun 20, 2023

Figured out the issue - heroku has started to add random hashes to the ends of the app url, so we need to now query for those. Will put up a fix shortly.

@jayhuynh
Copy link
Author

Thank you so much Jack. I'm excited to see your change 😄

@JackUrb JackUrb mentioned this issue Jun 20, 2023
Merged
@JackUrb
Copy link
Contributor

JackUrb commented Jun 20, 2023

@jayhuynh try the heroku-domain-query branch. If it works, I'll merge it in.

@jayhuynh
Copy link
Author

I work for me as well. Well done @JackUrb 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update heroku_architect.py facebookresearch/Mephisto
2 participants
@JackUrb @jayhuynh