Hasura WebSocket closed before connection is established error while trying to create subscription

[vmironichev]

Hello, we are using Hasura GraphQL engine for querying and subscriptions using web sockets, some times we are seeing errors with invalid timestamp pointing to some date from 1970 (random day/month/time) while trying to establish subscription. Hasura client (Web UI) can do a lot of reconnect attempts until it subscribes for the data (please see attached screenshot). Can you please advice, is it configuration/infrastructure issue or a bug? Thanks!

Hasura version 1.1.1 running in EKS
Postgres 11

[vmironichev]

Hello any update on this? I was able to reproduce this on your demo site as well (https://realtime-poll.demo.hasura.app/console)

[whitesunset]

Any update, please?

[marcfalk]

Experiencing the same issue currently. Running 1.3.1.

[viruhemanth]

Same issue. ver 1.3.2

[flofeld]

I got the same issue and so far I have been working around it by switching the browser.

[whitesunset]

I got the same issue and so far I have been working around it by switching the browser.

Switching the browser is not a solution, actually. Any updates? I'm establishing connection with Apollo client, by the way. Anyone else?

[MikeBosw]

Confirmed, I'm seeing this issue when I use Brave, although not when I use Chrome. My setup is a local docker-compose with port 8083(host):8080(client).

[soorajshankar]

Hello @vmironichev ,

Thanks for submitting the issue!

I was trying to reproduce this issue with the console on chrome, firefox & Brave. I could not reproduce this issue with the following steps

1. Installed Hasura with Hasura docker-compose setup.
2. Created a test table (users with a simple id as PK and name as Text)
3. Opened GraphiQL and started a subscription which lists all users
4. I tried disconnecting and reconnecting the subscription multiple times to check if I am getting a connection error or not.
5. I tried this with 1.3.2 also with one older version 1.1.1
6. I have also tried to connect to the subscription with a web application that uses apollo JS client in it.

Can any of you share the steps where you faced this issue?

[gabrielalmeida]

I was seeing the same issue because our application was replacing Hasura Cloud endpoint https protocol with wss on production, but when developing locally, I realized that docker-compose by default will not handle SSL.

Therefore, people self-hosting Hasura and not handling SSL should be using ws instead of wss to "fix" this.

[tirumaraiselvan]

It will really help if ppl can give details of their environments like Hasura version, Docker version, browser and OS since this seems to be environment specific.

FWIW, I tried the realtime app as given in this comment and also Brave browser as given in this comment but not able to get this error.

[philihp]

Double check the hostname, this is the error you'll get if you try to connect to a hostname that doesn't exist.

[slashvortal]

Same here.

• Hasura v1.3.2.
• @apollo/client v3.2.5
• graphql v15.3.0
• subscriptions-transport-ws 0.9.18
• https/wss

client > at first attempt > WebSocket is closed before the connection is established. (status: Finished)
client > second attempt (reconnect) > works perfectly (status: 101)

[morriq]

I have the same issue.
It stopped working on my local environment and on production the same day.

It works in private mode - both local and production

First I though is client issue: apollographql/subscriptions-transport-ws#377 and tried to resolve it with

SubscriptionClient.prototype.createMaxConnectTimeGenerator = function() {
  return {
    duration() {
      return WS_TIMEOUT // mine is at 60000
    },
    reset() {
      // noop
    },
  }
}

but it did not have any affect

Hasura v1.3.1(prod)1.3.2(local).
@apollo/client v3.2.5
graphql v15.3.0
subscriptions-transport-ws 0.9.18
http(locally)/wss

[rvigneshw]

Any solutions? I'm also getting the same issue
ApolloLink.js:12 Uncaught TypeError: Cannot read property 'length' of undefined at isTerminating (ApolloLink.js:12)

[nasatome]

both with "@apollo/client": "^3.3.3" in nextjs, and with the hasura console 1.3.3 and 1.3.4.beta.1, and 1.3.4.beta 2 with Brave Version 1.17.73 Chromium: 87.0.4280.67 (Official Build) (64-bit) on Linux PopOs 20.04, randomly in a subscription , I have the problem of:

On Firefox 84.0b4 (64-bit) works fine all time. (i tried one time in Google Chrome Version 87.0.4280.66 (Official Build) (64-bit) and its works)

if... i do a docker-compose down && docker-compose up -d and reload tab; works temporally in Brave browser

my docker-compose.yml file:

services:
  postgres:
    image: postgres:12
    restart: always
    ports:
      - "127.0.0.1:5432:5432"
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_DB: ${POSTGRES_DB}
  graphql-engine:
    image: hasura/graphql-engine:v1.3.4-beta.2
    ports:
      - "127.0.0.1:8080:8080"
    depends_on:
      - "postgres"
    restart: always
    environment:
      HASURA_GRAPHQL_DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: ${HASURA_GRAPHQL_ENABLE_CONSOLE} # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: ${HASURA_GRAPHQL_DEV_MODE}
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
      HASURA_GRAPHQL_ADMIN_SECRET: ${HASURA_GRAPHQL_ADMIN_SECRET}
      HASURA_GRAPHQL_JWT_SECRET: ${HASURA_GRAPHQL_JWT_SECRET}

volumes:
  db_data:

.env file for docker-compose

POSTGRES_USER=aaaaaaaaa
POSTGRES_PASSWORD=aaaaaaaaaaaaaaaaa
POSTGRES_DB=aaaaaa
HASURA_GRAPHQL_ENABLE_CONSOLE=true
## enable debugging mode. It is recommended to disable this in production
HASURA_GRAPHQL_DEV_MODE=true
HASURA_GRAPHQL_ADMIN_SECRET=aaaaaaaaaaaaaaaaaaaaaaaaaa

#jwt
## same JWT_SECRET in other env
HASURA_GRAPHQL_JWT_SECRET={"type":"HS512","key":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}

will it be a brave or hasura problem?

[seanonthenet]

Also have this issue. Running on localhost for Hasura Cloud.

declaring ApolloClient like this:

import { ApolloClient, HttpLink, split } from "@apollo/client";
import { WebSocketLink } from "@apollo/client/link/ws";
import { getMainDefinition } from "@apollo/client/utilities";
import { cache } from "./cache";

export const apolloClientCreator = (token) => {
  let headers = token
    ? {
        "content-type": "application/json",
        Authorization: `Bearer ${token}`,
      }
    : {
        "content-type": "application/json",
      };

  const httpLink = new HttpLink({
    uri: process.env.REACT_APP_GRAPHQL_HTTPS_ENDPOINT,
    headers,
  });

  const wsLink = new WebSocketLink({
    uri: process.env.REACT_APP_GRAPHQL_WS_ENDPOINT,
    options: {
      reconnect: true,
      connectionParams: { headers },,
    },
  });

  const splitLink = split(
    ({ query }) => {
      const definition = getMainDefinition(query);
      return definition.kind === "OperationDefinition" && definition.operation === "subscription";
    },
    wsLink,
    httpLink
  );

  return new ApolloClient({
    link: splitLink,
    cache: cache(),
  });
};

[nasatome]

@Spark3000 In what context are you? (browser,SO,etc). The error appears in all browsers? or Only Brave Browser?

[ctrepka]

I am also experiencing hang time on websockets in Brave. Developing on localhost with a free tier of Hasura Cloud, frontend developed with React, Apollo Client, and Auth0 SPA. The connection is eventually established, and the actual query takes about 30ms to complete per Hasura Logs. Once the connection is made, subsequent queries are relatively quick.

I've tested in Firefox and Chrome and have not experienced the same amount of network hangtime or websocket errors as in Brave. Not 100% positive whether this is something server side, or if Brave browser is fudging up my websocket connections before they reach the server, but it appears to be the same issue posted here.

[seanonthenet]

@Spark3000 In what context are you? (browser,SO,etc). The error appears in all browsers? or Only Brave Browser?

Only tested on Chrome latest on Mac OS 10.15. I’ve tried multiple configs in setting up the ApolloClient and can’t get rid of it. I’m using Create-React-App with a standard no-vpn public DNS network setup. Leads me to think it’s a server side thing.

[GavinRay97]

Hey folks, I am unable to consistently reproduce this.

I had that warning about the socket connection being closed once, but it didn't seem to impact the data being loaded and then I couldn't seem to reproduce it again.

Although, one Apollo setting may be necessary to prevent this warning from happening:

lazy?: boolean : use to set lazy mode - connects only when first subscription created, and delay the socket initialization

We had another user facing this issue report that this resolved it for them.

See: https://github.com/apollographql/subscriptions-transport-ws#constructorurl-options-websocketimpl

So here I have enabled this in my WS Link constructor options:

const wsLink = new WebSocketLink({
  uri: HASURA_WEBSOCKET_ENDPOINT,
  options: {
    reconnect: true,
    lazy: true, // <--- HERE
    connectionParams: { headers }
  }
})

Please have a look at the code in my repro attempt here (copied from @Spark3000's code):
https://codesandbox.io/s/blissful-cookies-vt0h0?file=/src/utils/apollo-client-creator.ts

And then the main app page:
https://codesandbox.io/s/blissful-cookies-vt0h0?file=/src/App.tsx

[seanonthenet]

Thanks @GavinRay97 !
I don't know why I didn't see the lazy property in any docs. Thanks for the help. Solved for me. Appreciate it.

[phishy]

I get this a ton when using Brave. If I restart the browser if goes away for a while, until it eventually comes back.

[davidhq]

This is still happening quite frequently on Brave, I filled an issue: brave/brave-browser#15410

maybe those affected (@phishy) could comment / confirm there as well? thank you

Would be great if they finally looked into this... I think I'm switching from Brave for time being because of this issue.

[davidhq]

cc/ @MikeBosw @ctrepka → so, as mentioned, would be good if more people commented on the issue I filled on Brave browser repo so it may get more visibility... maybe confirm that the problem is still happening for you as well.

[seanonthenet]

Is anyone seeing this error on a client other than Apollo?

[davidhq]

Yes, I'm mostly seeing on websites I'm developing. It happens rarely enough that I don't see it on other sites but I'm 99.9% sure it's issue in Brave since it doesn't happen in Chrome and it always works again from incognito mode or after browser restart and my code doesn't change in between. Another confirmation are reports here and at some other places. It's sporadic but very annoying and almost definitely has to do with Brave / Chromium (?)... any other problems on top are possibly separate inside webapps... but most of them use some websocket library and with good coding such issues where basic connection is a problem shouldn't happen.

[bel7aG]

SOLVED:

Set inactivityTimeout in a good timeout

const wsLink = new WebSocketLink({
  uri: 'wss://bel7ag.netlify.com/graphql',

  options: {
    reconnect: true,
    lazy: true,
    inactivityTimeout: 30000, // < --
  },
})

;(() => {
  ;(wsLink as any).subscriptionClient.maxConnectTimeGenerator.duration = () =>
    (wsLink as any).subscriptionClient.maxConnectTimeGenerator.max
})()

Versions:
"subscriptions-transport-ws": "^0.10.0",
"@apollo/client": "^3.4.0-rc.20",

NOTICE:
Make sure that the uri is missing "/" at the end (eg. wss://bel7ag.netlify.com/graphql) .

[ptakpatryk]

Hasura v2.0.0-alpha.10
@apollo/client v3.4.10

Frontend is written in VueJS. Tried everything above in Firefox and Chrome (ws:// instead of was://, lazy: true, inactivityTimeout...).
Getting the same error in chrome and firefox.

Error: Network error: cannot start as connection_init failed with : Could not verify JWT: JWTIssuedAtFuture

I also thought it's related to the timezone on my Hasura, but tried to tweak it - no luck.
The weirdest part is that it sometimes works nicely (even for few hours) and sometimes doesn't at all.

[jasonprado]

I am seeing this intermittently in my self-hosted Hasura server using Apollo in Chrome. It worked in Safari at the same time. POST-style GraphQL queries also succeeded from the same browser. The same browser failed in the same way against a staging server on a different hostname. Deleting cookies and local storage in the browser had no effect.

Quitting Chrome and restarting it made the issue stop reproducing.

Chrome macOS: 95.0.4638.54 (Official Build) (arm64)

Client libs:

"@apollo/client": "^3.4.16",
"subscriptions-transport-ws": "^0.9.19",

Hasura:

hasura/graphql-engine:v2.0.8

[CodingAleCR]

Don't if this will help anyone but here it goes:

When working localhost or with certain free hosting services you don't get a SSL certificate, and with that your endpoints will need to change from https to http and from wss to ws. At least that did the trick for me.

Hope this helps someone.

[tony-landis]

Have this same issue, Hasura is running in kubernetes NodePort.

HTTPS works just fine. WSS doesnt work even in the hasura console.

v2.9.0

[rahulagarwal13]

Thanks for reporting this and providing more info. We are looking into this and will keep you posted.

[tirumaraiselvan]

Hey folks, there are multiple scenarios where such an error can be thrown. For instance:

1. Not using the appropriate ws protocol: Hasura WebSocket closed before connection is established error while trying to create subscription #4509 (comment)
2. Not using the lazy option in Apollo: Hasura WebSocket closed before connection is established error while trying to create subscription #4509 (comment)
3. Not using inactivityTimeout: Hasura WebSocket closed before connection is established error while trying to create subscription #4509 (comment)

I am closing this ticket because of the above discussions and resolutions. If you have a different scenario where this error is thrown, then feel free to open a new ticket with details on reproducing it.

[EphremDemlew]

Same issue here, Hasura WebSocket closing before the connection is established, but this problem seemed to be only on Brave browser.

[trinhminhhieu]

Tôi gặp vấn đề tương tự vì ứng dụng của chúng tôi đang thay thế httpsgiao thức điểm cuối Hasura Cloud bằng giao thức wssđang được sản xuất nhưng khi phát triển cục bộ, tôi nhận ra rằng docker-compose theo mặc định sẽ không xử lý SSL .

Vì vậy, những người tự hosting Hasura và không xử lý SSL nên sử dụng wsthay vì wssđể "sửa" lỗi này.

Thanks it work! ,wss not working with https://domain.com/v1/graphql .I using ws http://ip domain:8080/v1/graphql it working!