a transient error with quotapool (multiple connections)

[amandeepgautam]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

grpc-go (1.10.0-dev)

What version of Go are you using (go version)?

gcc-go (go version go1.9 gccgo (GCC) 8.0.0 20171205 (experimental) aix/ppc)

What operating system (Linux, Windows, …) and version?

AIX 7.2

What did you do?

If possible, provide a recipe for reproducing the error.
Trying to reproduce is hard, but there is another issue (#1857) which probably discusses similar thing.

What did you expect to see?

Program should run successfully.

What did you see instead?

I am seeing this transient error:

goroutine 153 [select]:
google_golang_org_grpc_transport.get.pN42_google_golang_org_grpc_transport.quotaPool
/go/src/google.golang.org/grpc/transport/control.go:191
google_golang_org_grpc_transport.Write.pN44_google_golang_org_grpc_transport.http2Server
/go/src/google.golang.org/grpc/transport/http2_server.go:882
google_golang_org_grpc.sendResponse.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:769
google_golang_org_grpc.processUnaryRPC.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:945
google_golang_org_grpc.handleStream.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:1142
google_golang_org_grpc._nested82
/go/src/google.golang.org/grpc/server.go:637
created by google_golang_org_grpc._nested81
/go/src/google.golang.org/grpc/server.go:635 +316

goroutine 154 [select]:
google_golang_org_grpc_transport.get.pN42_google_golang_org_grpc_transport.quotaPool
/go/src/google.golang.org/grpc/transport/control.go:191
google_golang_org_grpc_transport.Write.pN44_google_golang_org_grpc_transport.http2Server
/go/src/google.golang.org/grpc/transport/http2_server.go:882
google_golang_org_grpc.sendResponse.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:769
google_golang_org_grpc.processUnaryRPC.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:945
google_golang_org_grpc.handleStream.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:1142
google_golang_org_grpc._nested82
/go/src/google.golang.org/grpc/server.go:637
created by google_golang_org_grpc._nested81
/go/src/google.golang.org/grpc/server.go:635 +316

goroutine 155 [select]:
google_golang_org_grpc_transport.get.pN42_google_golang_org_grpc_transport.quotaPool
/go/src/google.golang.org/grpc/transport/control.go:191
google_golang_org_grpc_transport.Write.pN44_google_golang_org_grpc_transport.http2Server
/go/src/google.golang.org/grpc/transport/http2_server.go:882
google_golang_org_grpc.sendResponse.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:769
google_golang_org_grpc.processUnaryRPC.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:945
google_golang_org_grpc.handleStream.pN29_google_golang_org_grpc.Server
/go/src/google.golang.org/grpc/server.go:1142
google_golang_org_grpc._nested82
/go/src/google.golang.org/grpc/server.go:637
created by google_golang_org_grpc._nested81
/go/src/google.golang.org/grpc/server.go:635 +316

[MakMukhi]

Can you clarify what's the error that you see? In the log you posted I only see goroutine traces.

[amandeepgautam]

I assumed this was the error. I use glog and this was in .FATAL file. I have lost the client side logs so I am not sure what exactly the error was but the end result was that the rpc call failed.

[dfawley]

In order to look into this, we will need more information:

• What does your program do, specifically?
• What other errors are you seeing? The goroutine trace above only shows that the server is blocked on flow control quota, which might mean the client was not receiving the server's messages quickly enough. These are not errors, and shouldn't lead to an error. If there is a FATAL file with errors in it, there should be more lines in there.

#1857 is most likely unrelated.

[MakMukhi]

Based on your response on grpc.io, can we close this issue then?

[amandeepgautam]

Yes. Should I reopen it or create a new issue later on?

[menghanl]

Closing this issue now.
If you find a way to reproduce the issue, please reply to this.