Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docdb] Add new flag for how many servers to wait for before the transactions table #6632

Closed
bmatican opened this issue Dec 11, 2020 · 0 comments
Assignees
Labels
area/docdb YugabyteDB core features

Comments

@bmatican
Copy link
Contributor

We have this code in CatalogManager, which currently waits for just RF number of TS to be up, before trying to create the txn status table. However, by default, we pick a number of tablets, that depends on the number of TS alive at moment of creation, for example 8 * number of servers. Worst case, we'll just create the table based on just the minimum (default RF = 3) tablets, so 24. In a big enough clusters, with say >24 nodes, this will not be enough to guarantee that we can get one txn status tablet leader on every node!

  if (FLAGS_enable_ysql) {
    LOG_WITH_PREFIX(INFO)
        << "YSQL is enabled, will create the transaction status table when "
        << FLAGS_replication_factor << " tablet servers are online";
    master_->ts_manager()->SetTSCountCallback(FLAGS_replication_factor, [this] {
      LOG_WITH_PREFIX(INFO)
          << FLAGS_replication_factor
          << " tablet servers registered, creating the transaction status table";
      // Retry table creation until it succeedes. It might fail initially because placement UUID
      // of live replicas is set through an RPC from YugaWare, and we won't be able to calculate
      // the number of primary (non-read-replica) tablet servers until that happens.
      while (true) {
        const auto s = CreateTransactionsStatusTableIfNeeded(/* rpc */ nullptr);

We should add a new flag for this, for a user to specify how many servers to wait for upfront, before pre-creating the table. Then our platform could specify this at universe creation, as we always know how many nodes we create in a universe. There's no backwards compatibility issues, but for a sensible default, we can just default this flag to replication_factor.

cc @psudheer21 @m-iancu

@bmatican bmatican added the area/docdb YugabyteDB core features label Dec 11, 2020
@bmatican bmatican self-assigned this Dec 11, 2020
@bmatican bmatican assigned sanketkedia and unassigned bmatican Dec 16, 2020
sanketkedia pushed a commit that referenced this issue Jan 20, 2021
…he transactions table is

created

Summary:
Currently in the catalog manager, we wait for just RF number of TS before creating the
transaction status table. However, when we move to large clusters (> 24 nodes) then just the RF
number of TS might not be enough to guarantee one Txn status tablet leader on every node.

Support has been added to specify the number of TS to wait for before creating the transaction
table. User can specify the flag "txn_table_wait_ts_count" for this purpose. The default value of
this flag is RF.

Test Plan:
./bin/yb-ctl create --master_flags "txn_table_wait_ts_count=3" --tserver_flags "txn_table_wait_ts_count=3"
Verify that Transaction table hasn't been created despite RF being 1
./bin/yb-ctl add_node --tserver_flags "txn_table_wait_ts_count=3"
./bin/yb-ctl add_node --tserver_flags "txn_table_wait_ts_count=3"
Verify that Transaction table gets created

Also, added a unit test in create-table-itest.cc that simulates the above sequence

Reviewers: bogdan, rsami

Reviewed By: bogdan, rsami

Subscribers: zyu, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D10258
polarweasel pushed a commit to lizayugabyte/yugabyte-db that referenced this issue Mar 9, 2021
…before the transactions table is

created

Summary:
Currently in the catalog manager, we wait for just RF number of TS before creating the
transaction status table. However, when we move to large clusters (> 24 nodes) then just the RF
number of TS might not be enough to guarantee one Txn status tablet leader on every node.

Support has been added to specify the number of TS to wait for before creating the transaction
table. User can specify the flag "txn_table_wait_ts_count" for this purpose. The default value of
this flag is RF.

Test Plan:
./bin/yb-ctl create --master_flags "txn_table_wait_ts_count=3" --tserver_flags "txn_table_wait_ts_count=3"
Verify that Transaction table hasn't been created despite RF being 1
./bin/yb-ctl add_node --tserver_flags "txn_table_wait_ts_count=3"
./bin/yb-ctl add_node --tserver_flags "txn_table_wait_ts_count=3"
Verify that Transaction table gets created

Also, added a unit test in create-table-itest.cc that simulates the above sequence

Reviewers: bogdan, rsami

Reviewed By: bogdan, rsami

Subscribers: zyu, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D10258
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features
Projects
None yet
Development

No branches or pull requests

2 participants