Does MySQLRouter Exporter connect with (either incoming or outgoing) MySQL?

[shayancanonical]

This issue is related to the time_wait issue that resulted in the implementation of graceful shutdowns.

I am still noticing connections in TIME_WAIT when I shutdown mysqlrouter-exporter:

root@mysql-router-k8s-0:/# netstat -nat  | grep 49152                                                                                                                                                              
tcp        0      0 10.1.241.240:49152      10.1.241.239:3306       TIME_WAIT                                                                                                                                      
root@mysql-router-k8s-0:/# netstat -ap | grep 49152                                                                                                                                                                
tcp        0      0 mysql-router-k8s-:49152 mysql-k8s-0.mysql-:3306 TIME_WAIT   

The address 10.1.241.239:3306 is for the mysql pod that I have in my deployment:

Unit                         Workload  Agent  Address       Ports  Message    
mysql-k8s/0*                 active    idle   10.1.241.239         Primary    
mysql-router-k8s/0*          active    idle   10.1.241.240      
mysql-test-app/0*            active    idle   10.1.241.193      
self-signed-certificates/0*  active    idle   10.1.241.247      

My question would be where in the router-exporter code do we connect with MySQL? Or is there a way MySQL can connect to the router exporter? If so, how? I have traced all the code in this repo as well as mysqlrouter-go repo, but fail to see connections with MySQL

Some additional context: mysql-router-k8s has an integration with mysql-k8s. mysql-router-k8s is using a self-signed TLS certificate

[rluisr]

As you said, the exporter doesn't connect to MySQL but to the MySQL Router.

Your output is strange.

BTW, How many seconds did you wait after sending the SIGTERM, SIGINT, Interrupt to an exporter to check its output?
After 5s the exporter is forced to shut down.

[shayancanonical]

Agreed the output is very strange.

I dug a little deeper and realized that the default --listen-port (49152) is an ephemeral port. I am noticing there are usually about 4000 connections in TIME_WAIT between mysql router and mysql (see below) and also realized that the above issue occurs when MySQL Router is using port 49152 to which MySQL Router Exporter tries to connect. This results in a bind address in use error

@rluisr Is it sensible to change the default listen-port to a reserved port number? Also, is it sensible to add an environment variable for the listen-port instead of just having it as a command line option?

...
tcp        0      0 10.1.241.203:59250      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:60108      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:56954      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:59562      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:33726      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:53318      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:56622      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:59868      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:55028      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:59668      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:45428      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57640      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:34340      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:54790      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:35480      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:56770      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:42806      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:34262      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:58274      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:56286      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:58936      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:42546      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:35150      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:33966      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57354      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:59730      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:58866      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:56178      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:39066      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:60532      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57672      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:53898      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:55914      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:59412      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:44564      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:53472      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:58826      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:42440      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:59662      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:58336      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57466      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:60018      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:34026      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57794      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57012      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57794      10.1.241.202:3306       TIME_WAIT  
tcp        0      0 10.1.241.203:57012      10.1.241.202:3306       TIME_WAIT  
root@mysql-router-k8s-0:/# netstat -nat | grep 3306 | wc -l
4004

[rluisr]

I don't know if I can solve this problem, but I will change the default listen port to 9152.

[shayancanonical]

In my opinion, changing the default listen port to 9152 should resolve bind address issues as MySQL Router would not be using that port to connect to MySQL 👍

[rluisr]

I have released v6.0.0. Could you please try it?

[shayancanonical]

Thanks! Testing now. Will report back here once issue stabilization confirmed (testing in canonical/mysql-router-operator#158)

[shayancanonical]

@rluisr We havent yet been able to adopt v6.0.0, but we have not encountered this issue again after changing our listen-port to 9152. Thus, closing this issue