Driver fail to reconnect if cluster changes

[marvinkite]

• Neo4j version: Neo4J Aura (4 Enterprise)
• Neo4j Mode: HA cluster with 3 members/Casual cluster with 1 core 2 read-replica
• Driver version: 1.8 Go driver v1.8.3
• Operating system: Linux Alpine
• Steps to reproduce
  Time to time execute some QueryTransaction
• Expected behavior
  Always returns a result and recovers from errors if possible.
• Actual behavior
  The application was not used over night when I assume the routing table contained the server 4 and 5 and 6. When the first query hit the server the operation failed but the second query succeeded.
  Running the CALL dbms.cluster.routing.getRoutingTable({}) cypher showed the cluster is reconfigured to server 7 (WRITE) and 8,9 (READ). The operation was executed with session from driver.Session(neo4j.AccessModeRead) so following the number logic the lowest number might be the write node so seems odd that the operation was trying to connect to server 4 and log shows it may checked 5 and 6 before. My application received the EOF error instead of driver recovering and returning with the result.

1. router 1:Reading routing table for '' from any of [neo4j-core-xxxxx-6.production-orch-0001.neo4j.io:7687 neo4j-core-xxxxx-5.production-orch-0001.neo4j.io:7687 neo4j-core-xxxxxxxx-4.production-orch-0001.neo4j.io:7687]

2. pool 1:No server connection available to any of [neo4j-core-xxxxx-6.production-orch-0001.neo4j.io:7687]

3. pool 1:No server connection available to any of [neo4j-core-xxxxx-5.production-orch-0001.neo4j.io:7687]

4. [email protected]:7687/v4:write tcp 10.xxxxx:43874->xxx:7687: write: broken pipe

   github.com/neo4j/neo4j-go-driver/neo4j.adaptorLogger.Error
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/logging.go:121
   github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).sendMsg
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:139
   github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).Close
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:663"
   

5. pool 1:No server connection available to any of [neo4j-core-xxxxxx-4.production-orch-0001.neo4j.io:7687]

6. router 1:Unable to retrieve routing table from neo4j-core-xxxxxx-4.production-orch-0001.neo4j.io:7687: EOF

   github.com/neo4j/neo4j-go-driver/neo4j.adaptorLogger.Error
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/logging.go:121
   github.com/neo4j/neo4j-go-driver/neo4j/internal/router.(*Router).getTable
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/router/router.go:108
   github.com/neo4j/neo4j-go-driver/neo4j/internal/router.(*Router).Readers
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/router/router.go:122
   github.com/neo4j/neo4j-go-driver/neo4j.(*session).borrowConn
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/session.go:379
   github.com/neo4j/neo4j-go-driver/neo4j.(*session).Run
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/session.go:456
   

7. [email protected]:7687/v4:write tcp 10.8.1.168:43868->35.187.125.190:7687: write: broken pipe

   github.com/neo4j/neo4j-go-driver/neo4j.adaptorLogger.Error
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/logging.go:121
   github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).sendMsg
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:139
   github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).Close
       /vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:663
   

[2hdddg]

Thanks for the very well written issue!
I can see that the driver doesn't fall back to the original URL when last known routers fail. A patch is on the way...

[marvinkite]

Hi,

I'm using the driver with this fix and I'm still connecting to Aura service and regularly see the EOF error messages.

There is only few hundreds or less of request per hour and I see this same error message 1-3 times per hour. Compare to the number of call it's very often.

1. router 1:Reading routing table for '' from previously known routers: [neo4j-core-xxxxx-12.production-orch-0001.neo4j.io:7687 neo4j-core-xxxxx-11.production-orch-0001.neo4j.io:7687 neo4j-core-xxxxx-10.production-orch-0001.neo4j.io:7687]

2. [email protected]:7687/v4:Connected

3. [email protected]:7687/v4:EOF

4. pool 1:Pruning neo4j-core-xxxxx-10.production-orch-0001.neo4j.io:7687 for connections that might be dead

5. pool 1:Unregistering dead or too old connection to neo4j-core-d86dc369-10.production-orch-0001.neo4j.io:7687

My application catch the io.EOF error message here and invokes the same operation again.

1. [email protected]:7687/v4:Connected

"github.com/indykite/jarvis/pkg/cypher.(*logger).Errorf
	/pkg/cypher/client.go:297
github.com/neo4j/neo4j-go-driver/neo4j.adaptorLogger.Error
	/vendor/github.com/neo4j/neo4j-go-driver/neo4j/logging.go:121
github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).receiveMsg
	/vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:148
github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).receiveSuccess
	/vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:172
github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).run
	/vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:431
github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt.(*bolt4).Run
	/vendor/github.com/neo4j/neo4j-go-driver/neo4j/internal/bolt/bolt4.go:480
github.com/neo4j/neo4j-go-driver/neo4j.(*session).Run
	/vendor/github.com/neo4j/neo4j-go-driver/neo4j/session.go:466