[improvement](binlog) Gc BE binlog metas when tablet is dropped

[deadlinefen]

Proposed changes

GC BE binlog metas when tablet is dropped.

Add a timed GC binlog metas task in the timed GC task of BE. When cleaning, it will traverse all keys starting with binlog_meta_ in the metadata, and get its corresponding rowset meta and check whether the tablet belonging to the rowset meta exists. If it does not exist (that is, it has been dropped), clean up all binlog_meta and binlog_data corresponding to this tablet.

In the implementation details, because the key of rocksdb is internally ordered, the same tablet will be traversed continuously, so if the tablet uuid obtained this time is the same as the last time, you can skip the check directly and execute the last operation (cleaned up or skipped).

At the same time, the deletion of binlog_meta and binlog_data corresponding to the same tablet-rowset must be in the same transaction, in order to ensure that metadata leakage does not occur.

Note

If there are existing binlog meta and tombstone in the cluster, they need to be GCed before upgrading to this version. This upgrade is not a smooth upgrade.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[deadlinefen]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.53 seconds
stream load tsv: 511 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162348464 Bytes

[deadlinefen]

run P0

[deadlinefen]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.37 seconds
stream load tsv: 547 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162112315 Bytes

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[deadlinefen]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.18 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162034130 Bytes

[dataroaring]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[JackDrogon]

改成使用~，来递增，重新使用rocksdb seek减少过滤吧
比如key是binlog_meta_6943f1585fe834b5-e542c2b83a21d0b7_00000000000000000069_020000000000000135449d7cd7eadfe672aa0f928fa99593', extract last part '020000000000000135449d7cd7eadfe672aa0f928fa99593, 之后改成binlog_meta_6943f1585fe834b5-e542c2b83a21d0b7~, 用这个key去过滤rocksdb

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[deadlinefen]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.41 seconds
stream load tsv: 509 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162193812 Bytes

[JackDrogon]

LGTM

[dataroaring]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.