Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When i restarted the server,master was read-only and never change to write status #1533

Closed
jichre opened this issue Mar 1, 2016 · 4 comments

Comments

@jichre
Copy link

jichre commented Mar 1, 2016

The first time to start the server,anything is ok.But,when i restarted the server,the master did something wrong.I can't write to the master,the master type was read-only.I just flowed the step of example.The "error.log" of mysql listed nothing.What can i do next?

Note only error log:
WARNING:root:vtclient_exception for test_keyspace.None.master: ('VitessError', u'shard, host: test_keyspace.0.master, uid:100 host:"gim-server2" port_map:<key:"grpc" value:16100 > port_map:<key:"mysql" value:33100 > port_map:<key:"vt" value:15100 > , vttablet: rpc error: code = 9 desc = "gRPCServerError: retry: The MySQL server is running with the --read-only option so it cannot execute this statement (errno 1290) during query: insert into test_table(msg) values ('V is for speed') /* _stream test_table (id ) (null ); // vtgate:: filtered_replication_unfriendly */"', 'ExecuteKeyRanges', 8, 'INSERT INTO test_table (msg) VALUES (:msg)', 'keyranges=, keyspace=test_keyspace, tablet_type=master')
Traceback (most recent call last):

@enisoc
Copy link
Member

enisoc commented Mar 1, 2016

If you restarted the mysqld that was the master at the time, then this would happen. We leave all servers as read-only at startup to prevent accidental writes on a partitioned master (a failure scenario called split brain). Generally if you need to restart the master, you should first issue PlannedReparentShard to failover to a different replica.

If you get into this state accidentally, you should be able to recover by doing EmergencyReparentShard and specifying whichever tablet has the highest GTID position. If you don't know which tablet that is, you should be able to list the GTID positions with ShardReplicationPositions.

@jichre
Copy link
Author

jichre commented Mar 1, 2016

I did it with your advice,but it did't work.The command i used:
zenist@gim-server2:/go/vtdataroot$ $VTROOT/bin/vtctlclient -server localhost:15999 EmergencyReparentShard test_keyspace/0 test-0000000100
E0301 10:52:19.983439 16089 main.go:43] Remote error: rpc error: code = 2 desc = "master-elect tablet test-0000000100 is already the master"
zenist@gim-server2:
/go/vtdataroot$ $VTROOT/bin/vtctlclient -server localhost:15999 PlannedReparentShard test_keyspace/0 test-0000000100
E0301 10:53:08.094881 16094 main.go:43] Remote error: rpc error: code = 2 desc = "master-elect tablet test-0000000100 is already the master"

Note:I used the "example/local/vttablet-down.sh" to down server and used "vttablet-up.sh" to start server.

@enisoc
Copy link
Member

enisoc commented Mar 1, 2016

Sorry I thought that EmergencyReparentShard could handle the case of re-electing the same master, but I was wrong. I'll work on that.

In the meantime, you can recover from this kind of thing manually, for example:

# master
SetReadWrite test-100
# slave 1
ReparentTablet test-101
StartSlave test-101
# slave 2
ReparentTablet test-102
StartSlave test-102

@jichre
Copy link
Author

jichre commented Mar 1, 2016

Thank you very much. The problem was resolved.

@jichre jichre closed this as completed Mar 1, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants