Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

a node issue could cause to be locked out from the cluster #8

Open
dottgonzo opened this issue Sep 11, 2017 · 3 comments
Open

a node issue could cause to be locked out from the cluster #8

dottgonzo opened this issue Sep 11, 2017 · 3 comments

Comments

@dottgonzo
Copy link

Hi,
after sometime in wich the cluster runs great (a bit slow, but works), today i found it down (on the logs, the nodes seems that fail to connect between), so i tryied to investigate the problem, but i can't spend much time to debug while many services was down. All that i know is that yesterday i restarted one node. My question is: the cluster is affected of some known issue? What happen if the container that bootstrap the cluster restarts?

Sorry for my english :D
@dottgonzo dottgonzo changed the title cluster fail cluster known issues? Sep 11, 2017
@toughIQ
Copy link
Owner

toughIQ commented Sep 11, 2017

If you do it the way I described in the README, if the first container would be created new, it would behave just as any other container started later. Because it detects the others already running containers and does not go in bootstrap mode.
In the beginning, when I started this project, I found that the swarm network sometimes did strange things which could interfere with the cluster.

@dottgonzo
Copy link
Author

dottgonzo commented Sep 11, 2017 •
edited
Loading

I've noticed that often the swarm networking is not stable and may cause several issues. Now i'm trying to deploy many services and, sometimes, the swarm become unstable, but the other services still continue to be working, this mariadb cluster instead become unreachable from other apps. Now my phpmyadmin instance, for example, show this error:

mysqli_real_connect(): (HY000/1130): Host '10.0.10.5' is not allowed to connect to this MariaDB server

is possible that there is some permission error for root user, or it is allowed to login from everywhere?

@dottgonzo
Copy link
Author

dottgonzo commented Sep 11, 2017 •
edited
Loading

i can confirm that working on swarm, deploying/removing services/stacks can cause errors on this service (but not on others!). I think that containers are changing theirs ip on overlay network and something stop to work. This weekend i will try to debug the issue, probabily the cluster don't change the ip of the nodes when it changes. Is possible to use hostnames instead of ips? is not elegant and not allow to use mode global, but could be useful

UPDATE:

i'm trying to reproduce the issue. With a fresh installation, the cluster seems solid as a rock. Starting to create some databases and populating them with only some data from a some apps (i'm trying to migrate from my docker vms to a swarm) and trying to restart docker service (today i've upgraded docker, so the system restart the docker service), the cluster seems that is able to reconnects the nodes, but the instances stop to connects to mariadb, root user is locked out from anywhere (if it exists somewhere).

STEP TO REPRODUCE:

docker-compose.yml

version: '3'

services:
  mariacluster:
    image: toughiq/mariadb-cluster
    networks:
      - mariadb
    environment:
      - DB_SERVICE_NAME=mariacluster
      - MYSQL_ROOT_PASSWORD=XXXXXXXXX
    deploy:
      mode: global
      placement:
        constraints:
          - node.labels.mariadb == true
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 0
        window: 10s



networks:
  mariadb:
    external: true

  1. Start a cluster of mariadb without persistent volumes (wait for connection, start to replicate the instance, etc...).

  2. populate the cluster with some data, (the dump of my test is 1 MB as .tar.gz).

  3. restart docker on a node

  4. wait some time (with a fresh installation you are locked out for a minute)

now the cluster is locked out

if try to access to the container restarted and you can't!
but it seems to have recovered the tables on /var/lib/mysql

if you try to access to the other containers on the cluster except the one that is not restarted (by cli) they works and i can show the tables

this is the log (docker service logs ...)

NODE 0 is the one on which i run systemctl restart docker


NODE 1    | Initializing database
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 64 ...
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Using Linux native AIO
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Using SSE crc32 instructions
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Completed initialization of buffer pool
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
NODE 1    | 2017-09-11 20:21:38 140528882051008 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Warning] InnoDB: New log files created, LSN=45883
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: Doublewrite buffer not found: creating new
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: Doublewrite buffer created
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Warning] InnoDB: Creating foreign key constraint system tables.
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: Foreign key constraint system tables created
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: Creating tablespace and datafile system tables.
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: Tablespace and datafile system tables created.
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB: Waiting for purge to start
NODE 1    | 2017-09-11 20:21:39 140528882051008 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 0
NODE 1    | 2017-09-11 20:21:40 140528041588480 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 1    | 2017-09-11 20:21:43 140171867363264 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 92 ...
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Using Linux native AIO
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Using SSE crc32 instructions
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Completed initialization of buffer pool
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB: Waiting for purge to start
NODE 1    | 2017-09-11 20:21:44 140171867363264 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616799
NODE 1    | 2017-09-11 20:21:44 140171047597824 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 121 ...
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Using Linux native AIO
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Using SSE crc32 instructions
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Completed initialization of buffer pool
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB: Waiting for purge to start
NODE 1    | 2017-09-11 20:21:47 140203190314944 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616809
NODE 1    | 2017-09-11 20:21:47 140202370660096 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 1    | 
NODE 1    | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
NODE 1    | To do so, start the server, then issue the following commands:
NODE 1    | 
NODE 1    | '/usr/bin/mysqladmin' -u root password 'new-password'
NODE 1    | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
NODE 1    | 
NODE 1    | Alternatively you can run:
NODE 1    | '/usr/bin/mysql_secure_installation'
NODE 1    | 
NODE 1    | which will also give you the option of removing the test
NODE 1    | databases and anonymous user created by default.  This is
NODE 1    | strongly recommended for production servers.
NODE 1    | 
NODE 1    | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
NODE 1    | MySQL manual for more instructions.
NODE 1    | 
NODE 1    | Please report any problems at http://mariadb.org/jira
NODE 1    | 
NODE 1    | The latest information about MariaDB is available at http://mariadb.org/.
NODE 1    | You can find additional information about the MySQL part at:
NODE 1    | http://dev.mysql.com
NODE 1    | Consider joining MariaDB's strong and vibrant community:
NODE 1    | https://mariadb.org/get-involved/
NODE 1    | 
NODE 1    | Database initialized
NODE 1    | MySQL init process in progress...
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 154 ...
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Using Linux native AIO
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Using SSE crc32 instructions
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Completed initialization of buffer pool
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB: Waiting for purge to start
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616819
NODE 1    | 2017-09-11 20:21:50 140248172459776 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 1    | 2017-09-11 20:21:50 140248992470976 [Note] Plugin 'FEEDBACK' is disabled.
NODE 1    | 2017-09-11 20:21:51 140248992470976 [Warning] 'user' entry 'root@ce06989cce60' ignored in --skip-name-resolve mode.
NODE 1    | 2017-09-11 20:21:51 140248992470976 [Warning] 'proxies_priv' entry '@% root@ce06989cce60' ignored in --skip-name-resolve mode.
NODE 1    | 2017-09-11 20:21:51 140248992470976 [Note] mysqld: ready for connections.
NODE 1    | Version: '10.1.26-MariaDB-1~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
NODE 1    | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
NODE 1    | 2017-09-11 20:21:52 140248991603456 [Warning] 'proxies_priv' entry '@% root@ce06989cce60' ignored in --skip-name-resolve mode.
NODE 1    | 
NODE 1    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_cluster_conf.sh
NODE 1    | 
NODE 1    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_galera_user.sh
NODE 1    | 
NODE 1    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_galera_user.sql
NODE 1    | 
NODE 1    | 
NODE 1    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_maxscale_user.sh
NODE 1    | 
NODE 1    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_maxscale_user.sql
NODE 1    | 
NODE 1    | 
NODE 1    | 2017-09-11 20:21:52 140248991300352 [Note] mysqld: Normal shutdown
NODE 1    | 
NODE 1    | 2017-09-11 20:21:52 140248991300352 [Note] Event Scheduler: Purging the queue. 0 events
NODE 1    | 2017-09-11 20:21:52 140248231159552 [Note] InnoDB: FTS optimize thread exiting.
NODE 1    | 2017-09-11 20:21:52 140248991300352 [Note] InnoDB: Starting shutdown...
NODE 1    | 2017-09-11 20:21:53 140248991300352 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
NODE 1    | 2017-09-11 20:21:54 140248991300352 [Note] InnoDB: Shutdown completed; log sequence number 1616829
NODE 1    | 2017-09-11 20:21:54 140248991300352 [Note] mysqld: Shutdown complete
NODE 1    | 
NODE 1    | 
NODE 1    | MySQL init process done. Ready for start up.
NODE 1    | 
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 1 ...
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: Read nil XID from storage engines, skipping position init
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: wsrep_load(): Galera 25.3.20(r3703) by Codership Oy <[email protected]> loaded successfully.
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: CRC-32C: using hardware acceleration.
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.0.10.10; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 256M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ign
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1)
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: wsrep_sst_grab()
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: Start replication
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: protonet asio version 0
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: Using CRC-32C for message checksums.
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: backend: asio
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: gcomm thread scheduling priority set to other:0 
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: restore pc from disk failed
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: GMCast version 0
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: EVS version 0
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: gcomm: connecting to group 'docker_cluster', peer '10.0.10.13:,10.0.10.10:,10.0.10.9:'
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') connection established to da8f1995 tcp://10.0.10.10:4567
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Warning] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') address 'tcp://10.0.10.10:4567' points to own listening address, blacklisting
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') connection established to bb41cf40 tcp://10.0.10.9:4567
NODE 1    | 2017-09-11 20:21:54 140207059843008 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
NODE 1    | 2017-09-11 20:21:55 140207059843008 [Note] WSREP: gcomm: connected
NODE 1    | 2017-09-11 20:21:55 140207059843008 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
NODE 1    | 2017-09-11 20:21:55 140207059843008 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
NODE 1    | 2017-09-11 20:21:55 140207059843008 [Note] WSREP: Opened channel 'docker_cluster'
NODE 1    | 2017-09-11 20:21:55 140207059843008 [Note] WSREP: Waiting for SST to complete.
NODE 1    | 2017-09-11 20:21:56 140206569158400 [Note] WSREP: declaring bb41cf40 at tcp://10.0.10.9:4567 stable
NODE 1    | 2017-09-11 20:21:56 140206569158400 [Note] WSREP: Node bb41cf40 state prim
NODE 1    | 2017-09-11 20:21:56 140206569158400 [Note] WSREP: view(view_id(PRIM,bb41cf40,2) memb {
NODE 1    | 	bb41cf40,0
NODE 1    | 	da8f1995,0
NODE 1    | } joined {
NODE 1    | } left {
NODE 1    | } partitioned {
NODE 1    | })
NODE 1    | 2017-09-11 20:21:56 140206569158400 [Note] WSREP: save pc into disk
NODE 1    | 2017-09-11 20:21:56 140206569158400 [Note] WSREP: discarding pending addr without UUID: tcp://10.0.10.13:4567
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: STATE EXCHANGE: sent state msg: db7a19db-972e-11e7-932c-8bf22f1a0f4e
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: STATE EXCHANGE: got state msg: db7a19db-972e-11e7-932c-8bf22f1a0f4e from 0 (dd8e29fd9f66)
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: STATE EXCHANGE: got state msg: db7a19db-972e-11e7-932c-8bf22f1a0f4e from 1 (ce06989cce60)
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: Quorum results:
NODE 1    | 	version    = 4,
NODE 1    | 	component  = PRIMARY,
NODE 1    | 	conf_id    = 1,
NODE 1    | 	members    = 1/2 (joined/total),
NODE 1    | 	act_id     = 0,
NODE 1    | 	last_appl. = -1,
NODE 1    | 	protocols  = 0/7/3 (gcs/repl/appl),
NODE 1    | 	group UUID = bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: Flow-control interval: [23, 23]
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: State transfer required: 
NODE 1    | 	Group state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:0
NODE 1    | 	Local state: 00000000-0000-0000-0000-000000000000:-1
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: New cluster view: global state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:0, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Warning] WSREP: Gap in state sequence. Need state transfer.
NODE 1    | 2017-09-11 20:21:56 140206531409664 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.0.10.10' --datadir '/var/lib/mysql/'   --parent '1'  '' '
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: Prepared SST request: rsync|10.0.10.10:4444/rsync_sst
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: REPL Protocols: 7 (3, 2)
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
NODE 1    | 2017-09-11 20:21:56 140206618801920 [Note] WSREP: Service thread queue flushed.
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bb426dc3-972e-11e7-b8b3-1e15f88fb462): 1 (Operation not permitted)
NODE 1    | 	 at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: Member 1.0 (ce06989cce60) requested state transfer from '*any*'. Selected 0.0 (dd8e29fd9f66)(SYNCED) as donor.
NODE 1    | 2017-09-11 20:21:56 140206560765696 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
NODE 2    | Initializing database
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 63 ...
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: Requesting state transfer: success, donor: 0
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 1    | 2017-09-11 20:21:56 140207059528448 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(bb426dc3-972e-11e7-b8b3-1e15f88fb462:0)
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 1    | 2017-09-11 20:21:57 140206569158400 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') connection established to dc4c65ac tcp://10.0.10.13:4567
NODE 1    | 2017-09-11 20:21:57 140206569158400 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') connection to peer da8f1995 with addr tcp://10.0.10.10:4567 timed out, no messages seen in PT3S
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Using Linux native AIO
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Using SSE crc32 instructions
NODE 1    | 2017-09-11 20:21:59 140206569158400 [Note] WSREP: declaring bb41cf40 at tcp://10.0.10.9:4567 stable
NODE 1    | 2017-09-11 20:21:59 140206569158400 [Note] WSREP: declaring dc4c65ac at tcp://10.0.10.13:4567 stable
NODE 1    | 2017-09-11 20:22:00 140206569158400 [Note] WSREP: Node bb41cf40 state prim
NODE 1    | 2017-09-11 20:22:00 140206569158400 [Note] WSREP: view(view_id(PRIM,bb41cf40,3) memb {
NODE 1    | 	bb41cf40,0
NODE 1    | 	da8f1995,0
NODE 1    | 	dc4c65ac,0
NODE 1    | } joined {
NODE 1    | } left {
NODE 1    | } partitioned {
NODE 1    | })
NODE 1    | 2017-09-11 20:22:00 140206569158400 [Note] WSREP: save pc into disk
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: STATE EXCHANGE: sent state msg: ddd368a5-972e-11e7-9b64-8b0692856627
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: STATE EXCHANGE: got state msg: ddd368a5-972e-11e7-9b64-8b0692856627 from 0 (dd8e29fd9f66)
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: STATE EXCHANGE: got state msg: ddd368a5-972e-11e7-9b64-8b0692856627 from 1 (ce06989cce60)
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: STATE EXCHANGE: got state msg: ddd368a5-972e-11e7-9b64-8b0692856627 from 2 (aa8d485890a9)
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: Quorum results:
NODE 1    | 	version    = 4,
NODE 1    | 	component  = PRIMARY,
NODE 1    | 	conf_id    = 2,
NODE 1    | 	members    = 1/3 (joined/total),
NODE 1    | 	act_id     = 0,
NODE 1    | 	last_appl. = 0,
NODE 1    | 	protocols  = 0/7/3 (gcs/repl/appl),
NODE 1    | 	group UUID = bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: Flow-control interval: [28, 28]
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: 0.0 (dd8e29fd9f66): State transfer to 1.0 (ce06989cce60) complete.
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: Member 0.0 (dd8e29fd9f66) synced with group.
NODE 1    | 2017-09-11 20:22:00 140206560765696 [Note] WSREP: Member 2.0 (aa8d485890a9) requested state transfer from '*any*'. Selected 0.0 (dd8e29fd9f66)(SYNCED) as donor.
NODE 1    | WSREP_SST: [INFO] Joiner cleanup. rsync PID: 242 (20170911 20:22:00.711)
NODE 1    | 2017-09-11 20:22:00 140206569158400 [Note] WSREP: (da8f1995, 'tcp://0.0.0.0:4567') turning message relay requesting off
NODE 1    | WSREP_SST: [INFO] Joiner cleanup done. (20170911 20:22:01.218)
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] WSREP: SST complete, seqno: 0
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Using Linux native AIO
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Using SSE crc32 instructions
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Completed initialization of buffer pool
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: The log sequence number 1616829 in ibdata file do not match the log sequence number 1617019 in the ib_logfiles!
NODE 1    | 2017-09-11 20:22:01 140207059843008 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
NODE 1    | 2017-09-11 20:22:03 140207059843008 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 1    | 2017-09-11 20:22:03 140207059843008 [Note] InnoDB: Waiting for purge to start
NODE 1    | 2017-09-11 20:22:04 140207059843008 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1617019
NODE 1    | 2017-09-11 20:22:04 140205868709632 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 1    | 2017-09-11 20:22:04 140207059843008 [Note] Plugin 'FEEDBACK' is disabled.
NODE 1    | 2017-09-11 20:22:04 140207059843008 [Note] Server socket created on IP: '0.0.0.0'.
NODE 1    | 2017-09-11 20:22:04 140207059843008 [Warning] 'proxies_priv' entry '@% root@dd8e29fd9f66' ignored in --skip-name-resolve mode.
NODE 1    | 2017-09-11 20:22:04 140207059843008 [Note] WSREP: Signalling provider to continue.
NODE 1    | 2017-09-11 20:22:04 140207059843008 [Note] WSREP: SST received: bb426dc3-972e-11e7-b8b3-1e15f88fb462:0
NODE 1    | 2017-09-11 20:22:04 140207059528448 [Note] WSREP: New cluster view: global state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:0, view# 3: Primary, number of nodes: 3, my index: 1, protocol version 3
NODE 1    | 2017-09-11 20:22:04 140207059528448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 1    | 2017-09-11 20:22:04 140207059528448 [Note] WSREP: REPL Protocols: 7 (3, 2)
NODE 1    | 2017-09-11 20:22:04 140207059528448 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
NODE 1    | 2017-09-11 20:22:04 140206618801920 [Note] WSREP: Service thread queue flushed.
NODE 1    | 2017-09-11 20:22:04 140206560765696 [Note] WSREP: 1.0 (ce06989cce60): State transfer from 0.0 (dd8e29fd9f66) complete.
NODE 1    | 2017-09-11 20:22:04 140206560765696 [Note] WSREP: Shifting JOINER -> JOINED (TO: 0)
NODE 1    | 2017-09-11 20:22:04 140207059843008 [Note] mysqld: ready for connections.
NODE 1    | Version: '10.1.26-MariaDB-1~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
NODE 1    | 2017-09-11 20:22:04 140206560765696 [Note] WSREP: Member 1.0 (ce06989cce60) synced with group.
NODE 1    | 2017-09-11 20:22:04 140206560765696 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
NODE 1    | 2017-09-11 20:22:04 140207059528448 [Note] WSREP: Synchronized with group, ready for connections
NODE 1    | 2017-09-11 20:22:04 140207059528448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 1    | 2017-09-11 20:22:06 140206560765696 [Note] WSREP: 0.0 (dd8e29fd9f66): State transfer to 2.0 (aa8d485890a9) complete.
NODE 1    | 2017-09-11 20:22:06 140206560765696 [Note] WSREP: Member 0.0 (dd8e29fd9f66) synced with group.
NODE 1    | 2017-09-11 20:22:13 140206560765696 [Note] WSREP: 2.0 (aa8d485890a9): State transfer from 0.0 (dd8e29fd9f66) complete.
NODE 1    | 2017-09-11 20:22:13 140206560765696 [Note] WSREP: Member 2.0 (aa8d485890a9) synced with group.
NODE 1    | 2017-09-11 20:30:16 140206569158400 [Note] WSREP: declaring bb41cf40 at tcp://10.0.10.9:4567 stable
NODE 1    | 2017-09-11 20:30:16 140206569158400 [Note] WSREP: forgetting dc4c65ac (tcp://10.0.10.13:4567)
NODE 1    | 2017-09-11 20:30:16 140206569158400 [Note] WSREP: Node bb41cf40 state prim
NODE 1    | 2017-09-11 20:30:16 140206569158400 [Note] WSREP: view(view_id(PRIM,bb41cf40,4) memb {
NODE 1    | 	bb41cf40,0
NODE 1    | 	da8f1995,0
NODE 1    | } joined {
NODE 1    | } left {
NODE 1    | } partitioned {
NODE 1    | 	dc4c65ac,0
NODE 1    | })
NODE 1    | 2017-09-11 20:30:16 140206569158400 [Note] WSREP: save pc into disk
NODE 1    | 2017-09-11 20:30:16 140206569158400 [Note] WSREP: forgetting dc4c65ac (tcp://10.0.10.13:4567)
NODE 1    | 2017-09-11 20:30:16 140206560765696 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
NODE 1    | 2017-09-11 20:30:16 140206560765696 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
NODE 1    | 2017-09-11 20:30:16 140206560765696 [Note] WSREP: STATE EXCHANGE: sent state msg: 055ded08-9730-11e7-8a26-729f1b9babc9
NODE 1    | 2017-09-11 20:30:16 140206560765696 [Note] WSREP: STATE EXCHANGE: got state msg: 055ded08-9730-11e7-8a26-729f1b9babc9 from 0 (dd8e29fd9f66)
NODE 1    | 2017-09-11 20:30:16 140206560765696 [Note] WSREP: STATE EXCHANGE: got state msg: 055ded08-9730-11e7-8a26-729f1b9babc9 from 1 (ce06989cce60)
NODE 1    | 2017-09-11 20:30:16 140206560765696 [Note] WSREP: Quorum results:
NODE 1    | 	version    = 4,
NODE 1    | 	component  = PRIMARY,
NODE 1    | 	conf_id    = 3,
NODE 1    | 	members    = 2/2 (joined/total),
NODE 1    | 	act_id     = 971,
NODE 1    | 	last_appl. = 921,
NODE 1    | 	protocols  = 0/7/3 (gcs/repl/appl),
NODE 1    | 	group UUID = bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 1    | 2017-09-11 20:30:16 140206560765696 [Note] WSREP: Flow-control interval: [23, 23]
NODE 1    | 2017-09-11 20:30:16 140207059528448 [Note] WSREP: New cluster view: global state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:971, view# 4: Primary, number of nodes: 2, my index: 1, protocol version 3
NODE 1    | 2017-09-11 20:30:16 140207059528448 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 1    | 2017-09-11 20:30:16 140207059528448 [Note] WSREP: REPL Protocols: 7 (3, 2)
NODE 1    | 2017-09-11 20:30:16 140207059528448 [Note] WSREP: Assign initial position for certification: 971, protocol version: 3
NODE 1    | 2017-09-11 20:30:16 140206618801920 [Note] WSREP: Service thread queue flushed.
NODE 1    | 2017-09-11 20:30:21 140206569158400 [Note] WSREP:  cleaning up dc4c65ac (tcp://10.0.10.13:4567)
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Completed initialization of buffer pool
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
NODE 2    | 2017-09-11 20:20:47 140425863661504 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Warning] InnoDB: New log files created, LSN=45883
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Doublewrite buffer not found: creating new
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Doublewrite buffer created
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Warning] InnoDB: Creating foreign key constraint system tables.
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Foreign key constraint system tables created
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Creating tablespace and datafile system tables.
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Tablespace and datafile system tables created.
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB: Waiting for purge to start
NODE 2    | 2017-09-11 20:20:48 140425863661504 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 0
NODE 2    | 2017-09-11 20:20:48 140425025287936 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 92 ...
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Using Linux native AIO
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Using SSE crc32 instructions
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Completed initialization of buffer pool
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB: Waiting for purge to start
NODE 2    | 2017-09-11 20:20:51 140207606822848 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616799
NODE 2    | 2017-09-11 20:20:51 140206787262208 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 121 ...
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Using Linux native AIO
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Using SSE crc32 instructions
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Completed initialization of buffer pool
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB: Waiting for purge to start
NODE 2    | 2017-09-11 20:20:54 139688517633984 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616809
NODE 2    | 2017-09-11 20:20:54 139687696004864 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 2    | 
NODE 2    | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
NODE 2    | To do so, start the server, then issue the following commands:
NODE 2    | 
NODE 2    | '/usr/bin/mysqladmin' -u root password 'new-password'
NODE 2    | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
NODE 2    | 
NODE 2    | Alternatively you can run:
NODE 2    | '/usr/bin/mysql_secure_installation'
NODE 2    | 
NODE 2    | which will also give you the option of removing the test
NODE 2    | databases and anonymous user created by default.  This is
NODE 2    | strongly recommended for production servers.
NODE 2    | 
NODE 2    | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
NODE 2    | MySQL manual for more instructions.
NODE 2    | 
NODE 2    | Please report any problems at http://mariadb.org/jira
NODE 2    | 
NODE 2    | The latest information about MariaDB is available at http://mariadb.org/.
NODE 2    | You can find additional information about the MySQL part at:
NODE 2    | http://dev.mysql.com
NODE 2    | Consider joining MariaDB's strong and vibrant community:
NODE 2    | https://mariadb.org/get-involved/
NODE 2    | 
NODE 2    | Database initialized
NODE 2    | MySQL init process in progress...
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 154 ...
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Using Linux native AIO
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Using SSE crc32 instructions
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Completed initialization of buffer pool
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB: Waiting for purge to start
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616819
NODE 2    | 2017-09-11 20:20:57 140557766620928 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] Plugin 'FEEDBACK' is disabled.
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Warning] 'user' entry 'root@dd8e29fd9f66' ignored in --skip-name-resolve mode.
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Warning] 'proxies_priv' entry '@% root@dd8e29fd9f66' ignored in --skip-name-resolve mode.
NODE 2    | 2017-09-11 20:20:57 140558586136512 [Note] mysqld: ready for connections.
NODE 2    | Version: '10.1.26-MariaDB-1~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
NODE 2    | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
NODE 2    | 2017-09-11 20:20:59 140558585268992 [Warning] 'proxies_priv' entry '@% root@dd8e29fd9f66' ignored in --skip-name-resolve mode.
NODE 2    | 
NODE 2    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_cluster_conf.sh
NODE 2    | 
NODE 2    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_galera_user.sh
NODE 2    | 
NODE 2    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_galera_user.sql
NODE 2    | 
NODE 2    | 
NODE 2    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_maxscale_user.sh
NODE 2    | 
NODE 2    | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init_maxscale_user.sql
NODE 2    | 
NODE 2    | 
NODE 2    | 2017-09-11 20:21:00 140558584965888 [Note] mysqld: Normal shutdown
NODE 2    | 
NODE 2    | 2017-09-11 20:21:00 140558584965888 [Note] Event Scheduler: Purging the queue. 0 events
NODE 2    | 2017-09-11 20:21:00 140557816928000 [Note] InnoDB: FTS optimize thread exiting.
NODE 2    | 2017-09-11 20:21:00 140558584965888 [Note] InnoDB: Starting shutdown...
NODE 2    | 2017-09-11 20:21:00 140558584965888 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
NODE 2    | 2017-09-11 20:21:02 140558584965888 [Note] InnoDB: Shutdown completed; log sequence number 1616829
NODE 2    | 2017-09-11 20:21:02 140558584965888 [Note] mysqld: Shutdown complete
NODE 2    | 
NODE 2    | 
NODE 2    | MySQL init process done. Ready for start up.
NODE 2    | 
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 1 ...
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Read nil XID from storage engines, skipping position init
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: wsrep_load(): Galera 25.3.20(r3703) by Codership Oy <[email protected]> loaded successfully.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: CRC-32C: using hardware acceleration.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.0.10.9; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 256M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.igno
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1)
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: wsrep_sst_grab()
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Start replication
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: protonet asio version 0
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Using CRC-32C for message checksums.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: backend: asio
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: gcomm thread scheduling priority set to other:0 
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: restore pc from disk failed
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: GMCast version 0
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: (bb41cf40, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: (bb41cf40, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: EVS version 0
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: gcomm: connecting to group 'docker_cluster', peer ''
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: start_prim is enabled, turn off pc_recovery
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Node bb41cf40 state prim
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: view(view_id(PRIM,bb41cf40,1) memb {
NODE 2    | 	bb41cf40,0
NODE 2    | } joined {
NODE 2    | } left {
NODE 2    | } partitioned {
NODE 2    | })
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: save pc into disk
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: gcomm: connected
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Opened channel 'docker_cluster'
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: Waiting for SST to complete.
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: Starting new group from scratch: bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: STATE_EXCHANGE: sent state UUID: bb428389-972e-11e7-8d2b-cb947d3bc4c7
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: STATE EXCHANGE: sent state msg: bb428389-972e-11e7-8d2b-cb947d3bc4c7
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: bb428389-972e-11e7-8d2b-cb947d3bc4c7 from 0 (dd8e29fd9f66)
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: Quorum results:
NODE 2    | 	version    = 4,
NODE 2    | 	component  = PRIMARY,
NODE 2    | 	conf_id    = 0,
NODE 2    | 	members    = 1/1 (joined/total),
NODE 2    | 	act_id     = 0,
NODE 2    | 	last_appl. = -1,
NODE 2    | 	protocols  = 0/7/3 (gcs/repl/appl),
NODE 2    | 	group UUID = bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: Flow-control interval: [16, 16]
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: Restored state OPEN -> JOINED (0)
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: Member 0.0 (dd8e29fd9f66) synced with group.
NODE 2    | 2017-09-11 20:21:02 139624143906560 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
NODE 2    | 2017-09-11 20:21:02 139624640211712 [Note] WSREP: New cluster view: global state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] WSREP: SST complete, seqno: 0
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Using Linux native AIO
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Using SSE crc32 instructions
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Completed initialization of buffer pool
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Highest supported file format is Barracuda.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB: Waiting for purge to start
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616829
NODE 2    | 2017-09-11 20:21:02 139623460239104 [Note] InnoDB: Dumping buffer pool(s) not yet started
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] Plugin 'FEEDBACK' is disabled.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] Server socket created on IP: '0.0.0.0'.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Warning] 'proxies_priv' entry '@% root@dd8e29fd9f66' ignored in --skip-name-resolve mode.
NODE 2    | 2017-09-11 20:21:02 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:21:02 139624640211712 [Note] WSREP: REPL Protocols: 7 (3, 2)
NODE 2    | 2017-09-11 20:21:02 139624640211712 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
NODE 2    | 2017-09-11 20:21:02 139624201942784 [Note] WSREP: Service thread queue flushed.
NODE 2    | 2017-09-11 20:21:02 139624640211712 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(bb426dc3-972e-11e7-b8b3-1e15f88fb462:0)
NODE 2    | 2017-09-11 20:21:02 139624640211712 [Note] WSREP: Synchronized with group, ready for connections
NODE 2    | 2017-09-11 20:21:02 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:21:02 139624640526272 [Note] mysqld: ready for connections.
NODE 2    | Version: '10.1.26-MariaDB-1~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
NODE 2    | 2017-09-11 20:21:54 139624152299264 [Note] WSREP: (bb41cf40, 'tcp://0.0.0.0:4567') connection established to da8f1995 tcp://10.0.10.10:4567
NODE 2    | 2017-09-11 20:21:54 139624152299264 [Note] WSREP: (bb41cf40, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
NODE 2    | 2017-09-11 20:21:56 139624152299264 [Note] WSREP: declaring da8f1995 at tcp://10.0.10.10:4567 stable
NODE 2    | 2017-09-11 20:21:56 139624152299264 [Note] WSREP: Node bb41cf40 state prim
NODE 2    | 2017-09-11 20:21:56 139624152299264 [Note] WSREP: view(view_id(PRIM,bb41cf40,2) memb {
NODE 2    | 	bb41cf40,0
NODE 2    | 	da8f1995,0
NODE 2    | } joined {
NODE 2    | } left {
NODE 2    | } partitioned {
NODE 2    | })
NODE 2    | 2017-09-11 20:21:56 139624152299264 [Note] WSREP: save pc into disk
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: STATE_EXCHANGE: sent state UUID: db7a19db-972e-11e7-932c-8bf22f1a0f4e
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: STATE EXCHANGE: sent state msg: db7a19db-972e-11e7-932c-8bf22f1a0f4e
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: db7a19db-972e-11e7-932c-8bf22f1a0f4e from 0 (dd8e29fd9f66)
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: db7a19db-972e-11e7-932c-8bf22f1a0f4e from 1 (ce06989cce60)
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: Quorum results:
NODE 2    | 	version    = 4,
NODE 2    | 	component  = PRIMARY,
NODE 2    | 	conf_id    = 1,
NODE 2    | 	members    = 1/2 (joined/total),
NODE 2    | 	act_id     = 0,
NODE 2    | 	last_appl. = 0,
NODE 2    | 	protocols  = 0/7/3 (gcs/repl/appl),
NODE 2    | 	group UUID = bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: Flow-control interval: [23, 23]
NODE 2    | 2017-09-11 20:21:56 139624640211712 [Note] WSREP: New cluster view: global state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
NODE 2    | 2017-09-11 20:21:56 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:21:56 139624640211712 [Note] WSREP: REPL Protocols: 7 (3, 2)
NODE 2    | 2017-09-11 20:21:56 139624640211712 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
NODE 2    | 2017-09-11 20:21:56 139624201942784 [Note] WSREP: Service thread queue flushed.
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: Member 1.0 (ce06989cce60) requested state transfer from '*any*'. Selected 0.0 (dd8e29fd9f66)(SYNCED) as donor.
NODE 2    | 2017-09-11 20:21:56 139624143906560 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 0)
NODE 2    | 2017-09-11 20:21:56 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:21:56 139623254718208 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.0.10.10:4444/rsync_sst' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/'     '' --gtid 'bb426dc3-972e-11e7-b8b3-1e15f88fb462:0' --gtid-domain-id '0''
NODE 2    | 2017-09-11 20:21:56 139624640211712 [Note] WSREP: sst_donor_thread signaled with 0
NODE 2    | 2017-09-11 20:21:56 139623254718208 [Note] WSREP: Flushing tables for SST...
NODE 2    | 2017-09-11 20:21:56 139623254718208 [Note] WSREP: Provider paused at bb426dc3-972e-11e7-b8b3-1e15f88fb462:0 (5)
NODE 2    | 2017-09-11 20:21:56 139623254718208 [Note] WSREP: Tables flushed.
NODE 2    | 2017-09-11 20:21:57 139624152299264 [Note] WSREP: (bb41cf40, 'tcp://0.0.0.0:4567') connection established to dc4c65ac tcp://10.0.10.13:4567
NODE 2    | 2017-09-11 20:21:59 139624152299264 [Note] WSREP: declaring da8f1995 at tcp://10.0.10.10:4567 stable
NODE 2    | 2017-09-11 20:21:59 139624152299264 [Note] WSREP: declaring dc4c65ac at tcp://10.0.10.13:4567 stable
NODE 2    | 2017-09-11 20:22:00 139623254718208 [Note] WSREP: resuming provider at 5
NODE 2    | 2017-09-11 20:22:00 139623254718208 [Note] WSREP: Provider resumed.
NODE 2    | 2017-09-11 20:22:00 139624152299264 [Note] WSREP: Node bb41cf40 state prim
NODE 2    | 2017-09-11 20:22:00 139624152299264 [Note] WSREP: view(view_id(PRIM,bb41cf40,3) memb {
NODE 2    | 	bb41cf40,0
NODE 2    | 	da8f1995,0
NODE 2    | 	dc4c65ac,0
NODE 2    | } joined {
NODE 2    | } left {
NODE 2    | } partitioned {
NODE 2    | })
NODE 2    | 2017-09-11 20:22:00 139624152299264 [Note] WSREP: save pc into disk
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: STATE_EXCHANGE: sent state UUID: ddd368a5-972e-11e7-9b64-8b0692856627
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: STATE EXCHANGE: sent state msg: ddd368a5-972e-11e7-9b64-8b0692856627
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: ddd368a5-972e-11e7-9b64-8b0692856627 from 0 (dd8e29fd9f66)
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: ddd368a5-972e-11e7-9b64-8b0692856627 from 1 (ce06989cce60)
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: ddd368a5-972e-11e7-9b64-8b0692856627 from 2 (aa8d485890a9)
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: Quorum results:
NODE 2    | 	version    = 4,
NODE 2    | 	component  = PRIMARY,
NODE 2    | 	conf_id    = 2,
NODE 2    | 	members    = 1/3 (joined/total),
NODE 2    | 	act_id     = 0,
NODE 2    | 	last_appl. = 0,
NODE 2    | 	protocols  = 0/7/3 (gcs/repl/appl),
NODE 2    | 	group UUID = bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: Flow-control interval: [28, 28]
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: New cluster view: global state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:0, view# 3: Primary, number of nodes: 3, my index: 0, protocol version 3
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: REPL Protocols: 7 (3, 2)
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
NODE 2    | 2017-09-11 20:22:00 139624201942784 [Note] WSREP: Service thread queue flushed.
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: 0.0 (dd8e29fd9f66): State transfer to 1.0 (ce06989cce60) complete.
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: Member 0.0 (dd8e29fd9f66) synced with group.
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: Synchronized with group, ready for connections
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: Member 2.0 (aa8d485890a9) requested state transfer from '*any*'. Selected 0.0 (dd8e29fd9f66)(SYNCED) as donor.
NODE 2    | 2017-09-11 20:22:00 139624143906560 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 0)
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:22:00 139623246325504 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.0.10.13:4444/rsync_sst' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/'     '' --gtid 'bb426dc3-972e-11e7-b8b3-1e15f88fb462:0' --gtid-domain-id '0''
NODE 2    | 2017-09-11 20:22:00 139624640211712 [Note] WSREP: sst_donor_thread signaled with 0
NODE 2    | 2017-09-11 20:22:00 139623246325504 [Note] WSREP: Flushing tables for SST...
NODE 2    | 2017-09-11 20:22:00 139623246325504 [Note] WSREP: Provider paused at bb426dc3-972e-11e7-b8b3-1e15f88fb462:0 (10)
NODE 2    | 2017-09-11 20:22:00 139623246325504 [Note] WSREP: Tables flushed.
NODE 2    | 2017-09-11 20:22:00 139624152299264 [Note] WSREP: (bb41cf40, 'tcp://0.0.0.0:4567') turning message relay requesting off
NODE 2    | 2017-09-11 20:22:04 139624143906560 [Note] WSREP: 1.0 (ce06989cce60): State transfer from 0.0 (dd8e29fd9f66) complete.
NODE 2    | 2017-09-11 20:22:04 139624143906560 [Note] WSREP: Member 1.0 (ce06989cce60) synced with group.
NODE 2    | 2017-09-11 20:22:06 139623246325504 [Note] WSREP: resuming provider at 10
NODE 2    | 2017-09-11 20:22:06 139623246325504 [Note] WSREP: Provider resumed.
NODE 2    | 2017-09-11 20:22:06 139624143906560 [Note] WSREP: 0.0 (dd8e29fd9f66): State transfer to 2.0 (aa8d485890a9) complete.
NODE 2    | 2017-09-11 20:22:06 139624143906560 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
NODE 2    | 2017-09-11 20:22:06 139624143906560 [Note] WSREP: Member 0.0 (dd8e29fd9f66) synced with group.
NODE 2    | 2017-09-11 20:22:06 139624143906560 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
NODE 2    | 2017-09-11 20:22:06 139624640211712 [Note] WSREP: Synchronized with group, ready for connections
NODE 2    | 2017-09-11 20:22:06 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:22:13 139624143906560 [Note] WSREP: 2.0 (aa8d485890a9): State transfer from 0.0 (dd8e29fd9f66) complete.
NODE 2    | 2017-09-11 20:22:13 139624143906560 [Note] WSREP: Member 2.0 (aa8d485890a9) synced with group.
NODE 2    | 2017-09-11 20:30:16 139624152299264 [Note] WSREP: declaring da8f1995 at tcp://10.0.10.10:4567 stable
NODE 2    | 2017-09-11 20:30:16 139624152299264 [Note] WSREP: forgetting dc4c65ac (tcp://10.0.10.13:4567)
NODE 2    | 2017-09-11 20:30:16 139624152299264 [Note] WSREP: Node bb41cf40 state prim
NODE 2    | 2017-09-11 20:30:16 139624152299264 [Note] WSREP: view(view_id(PRIM,bb41cf40,4) memb {
NODE 2    | 	bb41cf40,0
NODE 2    | 	da8f1995,0
NODE 2    | } joined {
NODE 2    | } left {
NODE 2    | } partitioned {
NODE 2    | 	dc4c65ac,0
NODE 2    | })
NODE 2    | 2017-09-11 20:30:16 139624152299264 [Note] WSREP: save pc into disk
NODE 2    | 2017-09-11 20:30:16 139624152299264 [Note] WSREP: forgetting dc4c65ac (tcp://10.0.10.13:4567)
NODE 2    | 2017-09-11 20:30:16 139624143906560 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
NODE 2    | 2017-09-11 20:30:16 139624143906560 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 055ded08-9730-11e7-8a26-729f1b9babc9
NODE 2    | 2017-09-11 20:30:16 139624143906560 [Note] WSREP: STATE EXCHANGE: sent state msg: 055ded08-9730-11e7-8a26-729f1b9babc9
NODE 2    | 2017-09-11 20:30:16 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: 055ded08-9730-11e7-8a26-729f1b9babc9 from 0 (dd8e29fd9f66)
NODE 2    | 2017-09-11 20:30:16 139624143906560 [Note] WSREP: STATE EXCHANGE: got state msg: 055ded08-9730-11e7-8a26-729f1b9babc9 from 1 (ce06989cce60)
NODE 2    | 2017-09-11 20:30:16 139624143906560 [Note] WSREP: Quorum results:
NODE 2    | 	version    = 4,
NODE 2    | 	component  = PRIMARY,
NODE 2    | 	conf_id    = 3,
NODE 2    | 	members    = 2/2 (joined/total),
NODE 2    | 	act_id     = 971,
NODE 2    | 	last_appl. = 921,
NODE 2    | 	protocols  = 0/7/3 (gcs/repl/appl),
NODE 2    | 	group UUID = bb426dc3-972e-11e7-b8b3-1e15f88fb462
NODE 2    | 2017-09-11 20:30:16 139624143906560 [Note] WSREP: Flow-control interval: [23, 23]
NODE 2    | 2017-09-11 20:30:16 139624640211712 [Note] WSREP: New cluster view: global state: bb426dc3-972e-11e7-b8b3-1e15f88fb462:971, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 3
NODE 2    | 2017-09-11 20:30:16 139624640211712 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
NODE 2    | 2017-09-11 20:30:16 139624640211712 [Note] WSREP: REPL Protocols: 7 (3, 2)
NODE 2    | 2017-09-11 20:30:16 139624640211712 [Note] WSREP: Assign initial position for certification: 971, protocol version: 3
NODE 2    | 2017-09-11 20:30:16 139624201942784 [Note] WSREP: Service thread queue flushed.
NODE 2    | 2017-09-11 20:30:21 139624152299264 [Note] WSREP:  cleaning up dc4c65ac (tcp://10.0.10.13:4567)
NODE 0    | Initializing database
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~jessie) starting as process 65 ...
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Using mutexes to ref count buffer pool pages
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: The InnoDB memory heap is disabled
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Compressed tables use zlib 1.2.8
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Using Linux native AIO
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Using SSE crc32 instructions
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Initializing buffer pool, size = 256.0M
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Completed initialization of buffer pool
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
NODE 0    | 2017-09-11 20:30:39 140685285701568 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
NODE 0    | 2017-09-11 20:30:40 140685285701568 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
NODE 0    | 2017-09-11 20:30:40 140685285701568 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
NODE 0    | 2017-09-11 20:30:40 140685285701568 [Warning] InnoDB: New log files created, LSN=45883
NODE 0    | 2017-09-11 20:30:40 140685285701568 [Note] InnoDB: Doublewrite buffer not found: creating new
NODE 0    | 2017-09-11 20:30:40 140685285701568 [Note] InnoDB: Doublewrite buffer created
NODE 0    | 2017-09-11 20:30:40 140685285701568 [Note] InnoDB: 128 rollback segment(s) are active.
NODE 0    | 2017-09-11 20:30:40 140685285701568 [Warning] InnoDB: Creating foreign key constraint system tables.
NODE 0    | 2017-09-11 20:30:41 140685285701568 [Note] InnoDB: Foreign key constraint system tables created
NODE 0    | 2017-09-11 20:30:41 140685285701568 [Note] InnoDB: Creating tablespace and datafile system tables.
NODE 0    | 2017-09-11 20:30:41 140685285701568 [Note] InnoDB: Tablespace and datafile system tables created.
NODE 0    | 2017-09-11 20:30:41 140685285701568 [Note] InnoDB: Waiting for purge to start
NODE 0    | 2017-09-11 20:30:41 140685285701568 [Note] InnoDB:  Percona XtraDB

this logs refers on a test on which i've tried to bind the address to 0.0.0.0 on my.conf, to try to start a cli inside the container after the boot process was terminated, but i can't to log in or to create a new password for root user

i don't understand why node 1 and 2 log 'Initializing database' etc.. so it seems restarting mysql service (while the node restarted is the 0).

@dottgonzo dottgonzo changed the title cluster known issues? a node issue can lock out root user from the cluster Sep 11, 2017
@dottgonzo dottgonzo changed the title a node issue can lock out root user from the cluster a node issue can lock out root user from the cluster/damage the cluster (i'm investigating) Sep 11, 2017
@dottgonzo dottgonzo changed the title a node issue can lock out root user from the cluster/damage the cluster (i'm investigating) a node issue cause to be lock out from the cluster Sep 11, 2017
@dottgonzo dottgonzo changed the title a node issue cause to be lock out from the cluster a node issue cause to be locked out from the cluster Sep 11, 2017
@dottgonzo dottgonzo changed the title a node issue cause to be locked out from the cluster a node issue could cause to be locked out from the cluster Sep 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dottgonzo @toughIQ