"No connection to Gradle server" in versions 2.6+

[h0use]

Since 2.6 (and, I'm guessing, as a result of the gRPC refactor?) my VS Code shows "no connection to Gradle server" whenever it starts, in spite of the log showing "Server started, listening on XXX".

This is on VS Code 1.45.1 on Mac OS Catalina, running Zulu JDK 11.

Thanks so much for your work on this plugin! It's really appreciated :)

[badsyntax]

Odd. Can you send me a screenshot of the output of the "Gradle Tasks Server" task in the terminal panel?
Can you also send a screenshot of the output of "Gradle Tasks" in the output panel?

Can you also confirm if the extension is actually broken for you, or if you're just seeing that message?

[h0use]

Hope these are helpful! Please let me know anything else I can do

With v2.5

With v3.0.2 (or anything 2.6 and above)

[vinzlee97]

Same here. I’m on Windows 10 and the extension sometimes manage or doesn’t manage to connect to the Gradle tasks server. Whenever it fails to connect, I press on Restart Server and the terminal window pops up only to say that the server has been listened on certain port, but in the end, the Gradle Tasks explorer pane never shows up.

[badsyntax]

@h0use thanks for sending these.

@vinzlee97 in your case can you also please add any more info, like a screenshot of the output for "Gradle Tasks". Do you get the same "failed to connect" error?

Sadly I can't replicate this problem on neither MacOS Catalina nor Windows 10.

Seems like a network thing, do you have any local firewalls or proxing enabled for localhost?

[vinzlee97]

In my case, as I said earlier, the extension sometimes connects successfully to the Gradle Tasks server, and sometimes it doesn't. Here is the failure case screencast:

And here's the successful case screencast for the same project:

I think it's not the network issue, but if that's the case, can you please tell me the debugging steps? Sorry for the laggy footage, and thanks.

[badsyntax]

@vinzlee97 thanks for this. can you try disabling all extensions, then only enable the Gradle Tasks extension, and see if this issue still occurs?

You can disable all extensions with the command palette:

Can you also send a list of your installed extensions? You can do this with the vscode command line executable:

code --list-extensions

Also, when the issue occurs, can you try run "Refresh Gradle Tasks" from the command palette and see if that loads the treeview? (although i don't think this will help, but give it a try)

[badsyntax]

@vinzlee97 can you also let me know what that green icon on the left of the statusbar is? Are you developing on remotely WSL? Can you let me know any other "non-standard" things you are doing? I'll need to replicate your setup to replicate the bug. If you can give me a good description of your exact setup, then I might be able to replicate. Thanks.

[badsyntax]

@h0use It's also appears you're using the remote development feature. Can you also describe your exact setup to help me replicate your scenario.
Can you also try disabling all other extensions as mentioned above? Thanks.

[badsyntax]

Ok I have a theory for this. It seems like it's a race condition due to the time it takes to bootstrap the Java Server process and when the client tries to connect.

Series of events:

1. Server Task is started (but process is not yet started)
2. Server Task tells client to connect
3. Client attempts to connect but fails. In most cases the client will successfully connect if the machine is able to bootstrap the server process in a short amount of time.

I was able to replicate this once, when my machine was under heavy load, and with the manual retries disabled, which supports this theory.

I think i'm using the wrong vscode API here:

https://github.com/badsyntax/vscode-gradle/blob/a1330db1a977795054fbc8e597a8efbc30f5f0d8/extension/src/server.ts#L32-L36

I need to be using onDidStartTaskProcess instead.

I was using the onDidStartTask API due to internal vscode issues when setting the Server Task presentation options, which I've since removed with PR #405

This is the smelly code to handle client retries:

https://github.com/badsyntax/vscode-gradle/blob/a1330db1a977795054fbc8e597a8efbc30f5f0d8/extension/src/client.ts#L433-L440

I was always suspicious of why i needed to do this, as the grpc client should be retrying itself. This code is fishy and again supports my theory above. It's a patch over a deeper problem. I believe I've fixed the deeper problem with how i construct the server task.

So in simple terms, the fix to only tell the client to connect once the underlying server process has been successfully bootstrapped.

I will try get a potential fix out for this today.

[vinzlee97]

Yeah, it has to do with Java Extension Pack by Microsoft (in my case).

So I reopened the same project 10 times for each case of extension disabling scenario, and here are the results:

It turned out when I disabled all extensions except Gradle Extension Pack, everything worked fine (10/10 successful connection attempts).

The next experiment I tried was to disable all extensions except Gradle Extension Pack and Java Extension Pack, and there was 2 failed attempts at the first and second try, then it connected successfully in the next attempts (8/10 successful connection attempts).

And finally, I tried to enable all extensions except Java Extension Pack, which surprised me that all attempts resulted in successful connection (10/10 successful connection attempts).

Here is my list of extensions:

aaron-bond.better-comments
alphabotsec.vscode-eclipse-keybindings
auchenberg.vscode-browser-preview
bierner.markdown-emoji
CoenraadS.bracket-pair-colorizer-2
Dart-Code.dart-code
Dart-Code.flutter
dkundel.vscode-new-file
eamodio.gitlens
FelixAngelov.bloc
Flutterando.flutter-mobx
formulahendry.code-runner
Gruntfuggly.todo-tree
jeroen-meijer.pubspec-assist
localizely.flutter-intl
mathiasfrohlich.Kotlin
ms-dotnettools.csharp
ms-python.python
ms-vscode-remote.remote-wsl
ms-vscode.cpptools
ms-vscode.Go
msjsdiag.debugger-for-chrome
naco-siren.gradle-language
Nash.awesome-flutter-snippets
pnp.polacode
quicktype.quicktype
redhat.java
richardwillis.vscode-gradle
richardwillis.vscode-gradle-extension-pack
scala-lang.scala
silverlakesoftware.searchdocsets-vscode
VisualStudioExptTeam.vscodeintellicode
vscjava.vscode-java-debug
vscjava.vscode-java-dependency
vscjava.vscode-java-pack
vscjava.vscode-java-test
vscjava.vscode-maven

[badsyntax]

I know the Java language server is quite heavy and perhaps it's just a machine load issue. (In that it takes a while to start 2 different Java servers.)

I've made a couple changes that can help with this issue.

Can you test this fix using Gradle Tasks v3.0.4? Can you test with all extensions enabled? Many thanks.

[vinzlee97]

Unfortunately, I got this in v3.0.4.

The Gradle: Connecting indicator keeps running though.
UPDATE: At some point long enough, the indicator then stops.

[badsyntax]

Arg. I just spotted this too. Sorry. I assume you're using Git Bash on windows? I'll get a fix out asap. Refs #420

[vinzlee97]

Yeah, it's Git Bash.

[badsyntax]

Can you try with version 3.0.5?

[badsyntax]

I'm actually still seeing some issues here, the client never connects occasionally.

I'm also seeing this in my CI tests occasionally. I will need more time to debug this, it's not obvious why the gRPC client is not always connecting. Sorry for this.

[vinzlee97]

Unfortunately, it doesn't work for me either.
It went on connecting until the connection failure message pops up.

This is the final output window:

[badsyntax]

Thanks for the screencasts, they're super useful.

The root cause is still a race condition. I need to find a way to only let the client connect to server once the gRPC server has started.

We can see the problem in your first screencast:

1. The Java process is started (Executing task "\.tasks-server.bat\")
2. The client tries to connect (Gradle: Connecting...)
3. Then we see the server is started ([main] INFO com.github.badsyntax.gradletasks.GradleTasksServer - Server started, listening on)

The correct order events should be:

1. The Java process is started (Executing task "\.tasks-server.bat\")
2. The server is started ([main] INFO com.github.badsyntax.gradletasks.GradleTasksServer - Server started, listening on)
3. The client tries to connect (Gradle: Connecting...)

This issue only occurs on slowish machines, where it takes a while for the gRPC server to start.

The documentation for the Java gRPC server also states I should only connect once it's started:

https://github.com/grpc/grpc-java/blob/0d6546719a34323545610fc4442c5265fa4b272f/api/src/main/java/io/grpc/Server.java#L41-L50

I had incorrectly assumed the gRPC client would keep retrying to connect, and connect when the server starts, but it seems it needs the server to be started first.

Now I better understand this issues, I can try find a fix. Thanks for your help and patience!

[vinzlee97]

You’re welcome. I’m glad they’re useful.

And thanks for the great extension! I look forward for the fix to the issue.

[badsyntax]

I've made a change that seems to resolve this issue. The extension will now do the following:

1. Start the server process
2. Wait for the gRPC server to be ready by waiting for the socket to be open
3. Only once the socket is open, let the client try to connect

The extension will keep trying to connect to the gRPC socket for a maximum of 2 minutes, after-which it will bail out with an error. I think 2 minutes is more than enough time, even on very slow machines.

With this approach, I'm no longer able to replicate the issue, and tests are passing consistently in CI. So this change looks good.

Can you test with version 3.0.6?

[vinzlee97]

I tested in a separate project with all extensions enabled 10 times. All attempted connections were successful. 👍

[badsyntax]

Great news, thanks @vinzlee97!

@h0use can you confirm the latest version fixes things for you?

[vinzlee97]

Actually, today I opened the same separate project in VSCode after turning on my machine. The "No connection to Gradle server" popup message appeared again at some point. It seemed because of the first open workload that made it happen, but after I reopened VSCode, the connection remained successful.

I guess in this case, it's my slowish machine which is the cause then.

[badsyntax]

Ok, I will add more logging around why the client is struggling to connect, increase the connect deadline, and add a mechanism for you to manually recover in this scenario.

[badsyntax]

I've released some changes with version 3.0.7

I've bumped the client connect deadline to 20 seconds, the client state will be logged to the output panel, and you'll be able to try re-connect:

[vinzlee97]

I'll test it in the next machine boot.

[vinzlee97]

So I couldn't replicate the failure case after I rebooted my machine. In my recent reboots, the connection was successful though. Here's the output:

And here's the terminal contents:

> Executing task in folder Mashup Project: ".\tasks-server.bat" 49963 <

[main] INFO com.github.badsyntax.gradletasks.GradleTasksServer - Server started, listening on 49963
May 22, 2020 7:48:39 PM io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport notifyTerminated
INFO: Transport failed
java.net.SocketException: Connection reset
        at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:345)
        at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:376)
        at io.grpc.netty.shaded.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
        at io.grpc.netty.shaded.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
        at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) 
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)    
        at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:830)

I will try to reconnect after I encounter the failure case, and I'll report it here. For now, everything seems fine if those exceptions don't affect the connection and thanks for the fix anyways.

[h0use]

I’ve been AFK for a couple of days but I’ll give it a go this weekend when I’m back to work! On 21 May 2020, at 19:22, Richard Willis <[email protected]> wrote:  Great news, thanks @vinzlee97 <https://github.com/vinzlee97>! @h0use <https://github.com/h0use> can you confirm the latest version fixes things for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#397 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTYJ24RY6LEZMRZUL6T7TRSVWPBANCNFSM4NBS7PYQ> .

[badsyntax]

@vinzlee97 Cool. I'll do a bit of research to better understand the cause of those exceptions. Is the speed in your GIF real-time or slowed down?

@h0use cool thanks! No rush, whenever you get a chance. It's be useful to know if things are fixed for you.

[vinzlee97]

@badsyntax since i maxed out the GIF quality, the speed itself became a bit of laggy. But, in the last GIF, it's realtime, because the process itself was slow on my machine (I shouldn't have cropped out the Gradle: Connecting indicator 🤦. It spun during the course of the GIF until the process was finished).

[badsyntax]

That's fine. Indeed this is quite slow. I'm not sure the slow speed is specific to the extension, it looks like generally your machine is quite slow. I don't think there's much I can do to improve the speed for you sadly. I need to use Java to use the Gradle Tooling API and unfortunately the JVM is a bit slow to start up, but once the the server has run the initial Gradle build it should be fairly quick thereafter even on slow machines. Can you copy/paste the the JVM args that are output in the output panel? I'm curious to see what is being used there.

[vinzlee97]

I reopened VSCode after the last GIF recording, but I believe the JVM Args remains the same no matter how many times I reload VSCode.

Here you go:

[info] JVM Args: --add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.base/java.lang=ALL-UNNAMED,--add-opens,java.base/java.lang.invoke=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,-Xmx1536m,-Dfile.encoding=windows-1252,-Duser.country=US,-Duser.language=en,-Duser.variant

[vinzlee97]

After I boot up and open VSCode today, the extension doesn't manage to connect to Gradle server at all (the output panel doesn't show anything, but in the terminal panel, it notified "Server started, listening on XXX"), and strangely, the failure connection message didn't pop up. What I did was to reconnect it manually:

1. Ctrl + C (Windows) and then terminate batch job.
2. Press Restart Server on the shown message popup.
3. It reconnects successfully.

I hope this will become handy to someone.

[badsyntax]

@vinzlee97 When you say "the output panel doesn't show anything" do you mean literally nothing is displayed, in that the "Gradle Tasks" output panel is just empty? Sometimes vscode takes a little while to render the output, but there should be some output displayed in all cases. Next time this happens, can you wait a bit to give vscode some time to render the output. And can you paste a screenshot of the output panel when this happens again? Thanks!

[vinzlee97]

Yeah, it didn't show anything and I waited for it in case some debugging info appeared, but unfortunately nothing came out.

I'm not so sure what caused it though, but I'll try to wait for a bit longer next time (and maybe I'll test by disabling some extensions beforehand).

Anyways, what did you notice in the JVM Args? @badsyntax

[studro]

I'm seeing issues with the client connecting in v3.0.7.

Other system info:

------------------------------------------------------------
Gradle 5.2.1
------------------------------------------------------------

Build time:   2019-02-08 19:00:10 UTC
Revision:     f02764e074c32ee8851a4e1877dd1fea8ffb7183

Kotlin DSL:   1.1.3
Kotlin:       1.3.20
Groovy:       2.5.4
Ant:          Apache Ant(TM) version 1.9.13 compiled on July 10 2018
JVM:          1.8.0_252 (AdoptOpenJDK 25.252-b09)
OS:           Windows 10 10.0 amd64

The server begins listening, and it's as if the client never attempts to connect. I can connect to the server by using telnet by targeting either 127.0.0.1 or my machine's ethernet IP.

Server:

[main] INFO com.github.badsyntax.gradletasks.GradleTasksServer - Server started, listening on 64362

Client (Gradle Tasks):

[info] Gradle server started
[error] Error connecting to gradle server: Failed to connect before the deadline
[error] The client has state: SHUTDOWN

I've tried various combinations of closing the server and attempting to re-connect / restart the server.

I can't see any dropped packets in Windows Firewall logs.

How does the client determine which server host/port to connect to? My suspicion is that my corporate network setup is at fault.

[badsyntax]

The server begins listening, and it's as if the client never attempts to connect. I can connect to the server by using telnet by targeting either 127.0.0.1 or my machine's ethernet IP.

How does the client determine which server host/port to connect to? My suspicion is that my corporate network setup is at fault.

The client attempts to connect using localhost. After server start, the extension waits for the tcp socket to be open before handing off to the grpc client to connect. The fact that you see the client error means the extension has been able to connect to the socket, but I'm not sure why the grpc client is unable to connect thereafter.

If you like you can hack the extension to get the grpc client to use 127.0.0.1 instead of localhost to see if it's a host issue.

In %USERPROFILE%\.vscode\extensions\richardwillis.vscode-gradle-3.0.7\dist\extension.js

(warning: it's a big ol file)

replace GradleTasksClient("localhost:"+this.server.getPort() with GradleTasksClient("127.0.0.1:"+this.server.getPort() and restart vscode.

[studro]

Sorry for the delay in getting back to you on this one. I modified the line you suggested and found the following:

• localhost does not work (unmodified)
• 127.0.0.1 also does not work
• funnily enough, 192.168... (my local ethernet address) does work.

I was going to suggest that this could be related to the bind address that the server component is using, but that doesn't seem likely. As mentioned earlier, I have no problem connecting to the server on any of the above three addesses from another process (e.g. telnet).

I might raise this with our networking team and see if there are any diagnostics they can perform on my machine to work out what is blocking the connections via the loopback address.

[h0use]

I've just tried localhost, 127.0.0.1, my local ethernet address and my work VPN address to no avail. I'm guessing it's a problem with the VPN or other network security malarkey they've installed on this Mac. Not sure what to try next.

[studro]

@h0use - I installed Wireshark and was able to see where my packets were going on a connection. This helped immensely - perhaps vscode is sending them to a HTTP proxy?

@badsyntax - thank you very much for your help. My localhost / 127.0.0.1 connections were being routed to a proxy. I added a NO_PROXY environment variable in Windows and it's now working fine.

NO_PROXY: 192.168.0.0/16,localhost,127.0.0.0/8 in case this helps anyone else.

I don't know much about gRPC (it appears you're using that for the client connection), but is it possible to invoke it with no proxy for your extension? It seems unlikely that a proxy would be needed for inter-process communication on the same machine.

I'd love to contribute code here, but I'm not an expert on typescript or gRPC.

My guess would be that on line ~90 of client.ts, the following change could be made:

this.grpcClient = new GrpcClient(
        `localhost:${this.server.getPort()}`,
        grpc.credentials.createInsecure()
      );

to

this.grpcClient = new GrpcClient(
        `localhost:${this.server.getPort()}`,
        grpc.credentials.createInsecure(),
        { 'grpc.enable_http_proxy': 0 }
      );

[studro]

@badsyntax - I've actually given this a try by hacking at the minified file you pointed me towards, but it didn't work (despite being documented that it should work). Is vscode intercepting http calls? 😕(That may be a stupid question, but I'm not super familiar with the Node / Electron / TypeScript stack.)

[badsyntax]

@studro thanks for this investigation, this is super useful. I'm using the grpc-js package, which is a fairly new package and only recently came out of beta. I have previously discovered some issues with it, so there's a chance it's just not honouring that setting. I will spend a bit of time understanding the grpc proxying and try determine if it's an issue with the package and get back to you.

[studro]

@badsyntax - No problem at all.

I also found this comment regarding vscode patching the http and https modules in Node. That might mean that nothing can be done anyway and you will need to rely on users having no_proxy configured correctly.

Let me know if you need any help testing any fixes or digging up information from my config.

[badsyntax]

if you have a system level proxy then grpc will attempt to use that proxy. (you can see grpc attempting to use system level proxy here). this is also why setting no_proxy works. http proxying localhost doesn't make sense and that's why this is failing imo.

i've been able to replicate this issue on my machine:

i set a bogus http proxy env var: http_proxy=http://192.168.1.10 and the grpc client never connects. i disable the grpc proxy for localhost using either no_proxy or no_grpc_proxy env vars and the grpc client successfully connects. i can also set this env var in the extension at runtime (via process.env) and that also works.

so as you discovered we just need to somehow tell the grpc client to not use the system proxy. i don't want to set env vars at runtime and i don't want to rely on the user setting an env var. it would be ideal if we could just disable the proxying in the grpc client with grcp.enable_http_proxy as you've demonstrated. having looked through the code it doesn't appear as though it is reading that setting, so i will speak with the author/s of @grpc/grpc-js to understand why. in the meantime you'll have to set no_proxy or no_grpc_proxy via env vars.

@h0use does this fix work for you? try setting no_grpc_proxy=localhost,127.0.0.0/8 and restart vscode

[badsyntax]

After speaking with the authors of grpc-js, I'm going to add the client option to grpc-js to disable http proxying in the client. PR here: grpc/grpc-node#1454

Interestingly, I found a fix waiting to be released that will fix the race condition issue mentioned further up in this thread, see grpc/grpc-node#1446 Once that fix is released I'll be able to remove my workaround code.

[h0use]

Setting no_grpc_proxy fixed it!! Awesome stuff. Thanks so much. Happy to help test any more permanent fixes for this when they're ready.

[badsyntax]

Awesome! Thanks to @studro for figuring out it was a proxy issue 👍 Now I'm just waiting on @grpc/[email protected] to be released before I can add a permanent fix.

[murgatroid99]

@grpc/grpc-js version 1.0.5 is now out.

[badsyntax]

thanks @murgatroid99!

i will try get the permanent fix out today.

[badsyntax]

@h0use @studro i've now released 3.0.8 with the permanent proxy fix. You can remove the no_proxy/no_grpc_proxy env vars now.

I believe the original issue is now fixed so i'm gonna close this issue, please re-open if it's not fixed for you.

@vinzlee97 as your issue isn't related to the issue OP mentioned, please create a new issue if you're still having problems.

thanks all.

[h0use]

Awesome stuff! Thanks to you and everyone else for figuring it out and fixing it!

[Chasson1992]

I'm having the same issue of "No connection to Gradle server" in v3.12.5 and adding localhost to no_grpc_proxy resolved the issue.

Setup:
VS Code 1.72.1
OS: Rocky8
Gradle for Java v3.12.5
Language Support for Java v1.12.0

I am using VS Code Remote-SSH Extension to connect to a virtual machine running Rocky8 with Gradle version 6.8.3. The host OS is Windows 10.