[jetbrains] gateway hangs to connect sometimes

[akosyakov]

Bug description

I got it once in prev envs after trying to use in 1 week. My token was invalid, since prev env were recreated. But instead of connecting, GW gitpod plugin hanged for some time without any UI and only then tried to reauthenticate. We should invalidate token and do auth again immediately.

You can see 30 secs delay in logs:

``` 2022-09-12 10:44:27,258 [ 22694] WARN - i.g.g.a.GitpodServerConnectionImpl - ak-tune-jb-build.preview.gitpod-dev.com: connection error: org.eclipse.jetty.websocket.api.UpgradeException: 401 Unauthorized at org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.onComplete(WebSocketUpgradeRequest.java:529) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210) at org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener.forwardFailureComplete(AuthenticationProtocolHandler.java:297) at org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener.onComplete(AuthenticationProtocolHandler.java:164) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:481) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:461) at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:424) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:365) at org.eclipse.jetty.http.HttpParser.handleContentMessage(HttpParser.java:597) at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1722) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1551) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:204) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:144) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: Authentication challenge without WWW-Authenticate header ... 29 more 2022-09-12 10:44:57,273 [ 52709] WARN - i.g.g.a.GitpodServerConnectionImpl - ak-tune-jb-build.preview.gitpod-dev.com: failed to stop websocket container: java.lang.RuntimeException: java.lang.InterruptedException at org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:650) at org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:618) at org.eclipse.jetty.io.SelectorManager.doStop(SelectorManager.java:289) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStop(AbstractConnectorHttpClientTransport.java:70) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.client.HttpClient.doStop(HttpClient.java:279) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.websocket.client.WebSocketClient.doStop(WebSocketClient.java:429) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.websocket.jsr356.ClientContainer.doStop(ClientContainer.java:314) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at io.gitpod.gitpodprotocol.api.GitpodServerLauncher.lambda$listen$0(GitpodServerLauncher.java:92) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at io.gitpod.gitpodprotocol.api.GitpodServerLauncher$1.onError(GitpodServerLauncher.java:114) at org.eclipse.jetty.websocket.jsr356.endpoints.JsrEndpointEventDriver.onError(JsrEndpointEventDriver.java:153) at org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.handleException(WebSocketUpgradeRequest.java:545) at org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.onComplete(WebSocketUpgradeRequest.java:529) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210) at org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener.forwardFailureComplete(AuthenticationProtocolHandler.java:297) at org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener.onComplete(AuthenticationProtocolHandler.java:164) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218) at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:481) at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:461) at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:424) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:365) at org.eclipse.jetty.http.HttpParser.handleContentMessage(HttpParser.java:597) at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1722) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1551) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:204) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:144) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.InterruptedException at java.base/java.lang.Object.wait(Native Method) at java.base/java.lang.Thread.join(Thread.java:1298) at org.eclipse.jetty.util.thread.QueuedThreadPool.joinThreads(QueuedThreadPool.java:320) at org.eclipse.jetty.util.thread.QueuedThreadPool.doStop(QueuedThreadPool.java:259) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:148) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:642) ... 60 more 2022-09-12 10:44:57,299 [ 52735] WARN - #k.c.CoroutineScope - ak-tune-jb-build.preview.gitpod-dev.com: invalid token, authorizing again and reconnecting: io.gitpod.jetbrains.gateway.GitpodConnectionService$InvalidTokenException: failed web socket handshake (401) at io.gitpod.jetbrains.gateway.GitpodConnectionService$obtainClient$1$job$1$connect$1.invokeSuspend(GitpodConnectionService.kt:71) at io.gitpod.jetbrains.gateway.GitpodConnectionService$obtainClient$1$job$1$connect$1.invoke(GitpodConnectionService.kt) at io.gitpod.jetbrains.gateway.GitpodConnectionService$obtainClient$1$job$1$connect$1.invoke(GitpodConnectionService.kt) at io.gitpod.jetbrains.gateway.GitpodConnectionService$obtainClient$1$job$1.invokeSuspend(GitpodConnectionService.kt:95) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) 2022-09-12 10:44:59,777 [ 55213] INFO - #k.c.CoroutineScope - ak-tune-jb-build.preview.gitpod-dev.com: connected ```

Steps to reproduce

• Auth and then close GW.
• Delete token in prev envs.
• Try to connect again.

Workspace affected

No response

Expected behavior

If token expired then invalidation should happen immediately on reconnection.

Example repository

No response

Anything else?

No response