Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Merged
merged 31 commits into from
Nov 20, 2024
Merged

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
1783421
restore: implement restorer for compacted SST/Snapshot/log files
3pointer Nov 7, 2024
d449c73
resolve conflicts
3pointer Nov 7, 2024
5fab677
Merge branch 'master' into feat/compacted_logs_2nd
3pointer Nov 12, 2024
73959f4
refine code after merge master
3pointer Nov 12, 2024
c6d643a
add test cases
3pointer Nov 12, 2024
3674e0e
rename some names && more test cases
3pointer Nov 12, 2024
7ef0f7d
remove useless code
3pointer Nov 12, 2024
23a5d18
fix build
3pointer Nov 13, 2024
0ec3cb2
polish code
3pointer Nov 13, 2024
c775e12
add NewSnapFileImporterOptions
3pointer Nov 13, 2024
329bf69
add tryMap for br iter
3pointer Nov 14, 2024
4dcd764
fix bazel
3pointer Nov 14, 2024
ed5d95e
fix integration tests
3pointer Nov 14, 2024
57d15db
address comments
3pointer Nov 14, 2024
57191e0
address comments
3pointer Nov 15, 2024
fbb71a9
fix test
3pointer Nov 16, 2024
3c0f79e
fix integration test
3pointer Nov 17, 2024
886b8f9
fix br_txn
3pointer Nov 17, 2024
0784b43
fix br_raw
3pointer Nov 17, 2024
a565ed0
refine code
3pointer Nov 17, 2024
b715725
rename some functions
3pointer Nov 17, 2024
f05b60e
fix reset limit
3pointer Nov 17, 2024
5e2eac4
update case
3pointer Nov 17, 2024
fafb0d2
update cases
3pointer Nov 17, 2024
d182593
Merge branch 'master' into feat/compacted_logs_2nd
3pointer Nov 17, 2024
1f1b636
fix the test after merge master
3pointer Nov 17, 2024
b9651ed
fix cases
3pointer Nov 18, 2024
c406c7a
fix ut
3pointer Nov 18, 2024
5346932
fix ut after filterout moving forward
3pointer Nov 18, 2024
b7bda58
resolve conflicts
3pointer Nov 19, 2024
fed41b5
address comment
3pointer Nov 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions br/pkg/restore/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,33 @@ go_library(
srcs = [
"import_mode_switcher.go",
"misc.go",
"restorer.go",
],
importpath = "github.com/pingcap/tidb/br/pkg/restore",
visibility = ["//visibility:public"],
deps = [
"//br/pkg/checkpoint",
"//br/pkg/conn",
"//br/pkg/conn/util",
"//br/pkg/errors",
"//br/pkg/logutil",
"//br/pkg/pdutil",
"//br/pkg/restore/split",
"//br/pkg/restore/utils",
"//br/pkg/summary",
"//br/pkg/utils",
"//br/pkg/utils/iter",
"//pkg/domain",
"//pkg/kv",
"//pkg/meta",
"//pkg/meta/model",
"//pkg/parser/model",
"//pkg/util",
"@com_github_go_sql_driver_mysql//:mysql",
"@com_github_opentracing_opentracing_go//:opentracing-go",
"@com_github_pingcap_errors//:errors",
"@com_github_pingcap_failpoint//:failpoint",
"@com_github_pingcap_kvproto//pkg/brpb",
"@com_github_pingcap_kvproto//pkg/import_sstpb",
"@com_github_pingcap_log//:log",
"@com_github_tikv_client_go_v2//oracle",
Expand All @@ -34,6 +42,7 @@ go_library(
"@org_golang_google_grpc//credentials/insecure",
"@org_golang_x_sync//errgroup",
"@org_uber_go_zap//:zap",
"@org_uber_go_zap//zapcore",
],
)

Expand All @@ -43,21 +52,28 @@ go_test(
srcs = [
"import_mode_switcher_test.go",
"misc_test.go",
"restorer_test.go",
],
flaky = True,
race = "off",
shard_count = 6,
shard_count = 13,
deps = [
":restore",
"//br/pkg/conn",
"//br/pkg/mock",
"//br/pkg/pdutil",
"//br/pkg/restore/split",
"//br/pkg/restore/utils",
"//br/pkg/utils/iter",
"//pkg/kv",
"//pkg/parser/model",
"//pkg/session",
"//pkg/tablecodec",
"//pkg/util",
"//pkg/util/codec",
"@com_github_coreos_go_semver//semver",
"@com_github_pingcap_errors//:errors",
"@com_github_pingcap_failpoint//:failpoint",
"@com_github_pingcap_kvproto//pkg/brpb",
"@com_github_pingcap_kvproto//pkg/import_sstpb",
"@com_github_pingcap_kvproto//pkg/metapb",
"@com_github_stretchr_testify//require",
Expand Down
13 changes: 7 additions & 6 deletions br/pkg/restore/import_mode_switcher.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package restore
import (
"context"
"crypto/tls"
"sync"
"time"

_ "github.com/go-sql-driver/mysql" // mysql driver
Expand Down Expand Up @@ -47,8 +48,8 @@ func NewImportModeSwitcher(
}

// switchToNormalMode switch tikv cluster to normal mode.
func (switcher *ImportModeSwitcher) switchToNormalMode(ctx context.Context) error {
close(switcher.switchCh)
func (switcher *ImportModeSwitcher) SwitchToNormalMode(ctx context.Context) error {
sync.OnceFunc(func() { close(switcher.switchCh) })
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should save and call the returned function or nothing will happen...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a strange behavior exists a long time. in stream restore RestorePreWork will skip the switch tikv mode, but in RestorePostWork it will switch back to normal. so this sync.OnceFunc make us don't close channel twice.
I can change this behavior in next PR. I think that will be more clear.

Copy link
Contributor

@YuJuncen YuJuncen Nov 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, but for now, this line of code is actually a nop... We will close the channel exactly zero times.

return switcher.switchTiKVMode(ctx, import_sstpb.SwitchMode_Normal)
}

Expand Down Expand Up @@ -113,8 +114,8 @@ func (switcher *ImportModeSwitcher) switchTiKVMode(
return nil
}

// switchToImportMode switch tikv cluster to import mode.
func (switcher *ImportModeSwitcher) switchToImportMode(
// SwitchToImportMode switch tikv cluster to import mode.
func (switcher *ImportModeSwitcher) SwitchToImportMode(
ctx context.Context,
) {
// tikv automatically switch to normal mode in every 10 minutes
Expand Down Expand Up @@ -163,7 +164,7 @@ func RestorePreWork(

if switchToImport {
// Switch TiKV cluster to import mode (adjust rocksdb configuration).
switcher.switchToImportMode(ctx)
switcher.SwitchToImportMode(ctx)
}

return mgr.RemoveSchedulersWithConfig(ctx)
Expand All @@ -186,7 +187,7 @@ func RestorePostWork(
ctx = context.Background()
}

if err := switcher.switchToNormalMode(ctx); err != nil {
if err := switcher.SwitchToNormalMode(ctx); err != nil {
log.Warn("fail to switch to normal mode", zap.Error(err))
}
if err := restoreSchedulers(ctx); err != nil {
Expand Down
10 changes: 7 additions & 3 deletions br/pkg/restore/log_client/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@ go_library(
name = "log_client",
srcs = [
"client.go",
"compacted_file_strategy.go",
"import.go",
"import_retry.go",
"log_file_manager.go",
"log_file_map.go",
"log_split_strategy.go",
"migration.go",
],
importpath = "github.com/pingcap/tidb/br/pkg/restore/log_client",
Expand All @@ -26,6 +28,7 @@ go_library(
"//br/pkg/restore/ingestrec",
"//br/pkg/restore/internal/import_client",
"//br/pkg/restore/internal/rawkv",
"//br/pkg/restore/snap_client",
"//br/pkg/restore/split",
"//br/pkg/restore/tiflashrec",
"//br/pkg/restore/utils",
Expand All @@ -43,6 +46,7 @@ go_library(
"//pkg/util",
"//pkg/util/codec",
"//pkg/util/redact",
"//pkg/util/sqlexec",
"//pkg/util/table-filter",
"@com_github_fatih_color//:color",
"@com_github_gogo_protobuf//proto",
Expand Down Expand Up @@ -86,12 +90,13 @@ go_test(
],
embed = [":log_client"],
flaky = True,
shard_count = 42,
shard_count = 45,
deps = [
"//br/pkg/errors",
"//br/pkg/glue",
"//br/pkg/gluetidb",
"//br/pkg/mock",
"//br/pkg/restore",
"//br/pkg/restore/internal/import_client",
"//br/pkg/restore/split",
"//br/pkg/restore/utils",
Expand All @@ -113,17 +118,16 @@ go_test(
"//pkg/util/codec",
"//pkg/util/sqlexec",
"//pkg/util/table-filter",
"@com_github_docker_go_units//:go-units",
"@com_github_pingcap_errors//:errors",
"@com_github_pingcap_failpoint//:failpoint",
"@com_github_pingcap_kvproto//pkg/brpb",
"@com_github_pingcap_kvproto//pkg/encryptionpb",
"@com_github_pingcap_kvproto//pkg/errorpb",
"@com_github_pingcap_kvproto//pkg/import_sstpb",
"@com_github_pingcap_kvproto//pkg/metapb",
"@com_github_pingcap_kvproto//pkg/pdpb",
"@com_github_pingcap_log//:log",
"@com_github_stretchr_testify//require",
"@com_github_tikv_pd_client//:client",
"@org_golang_google_grpc//codes",
"@org_golang_google_grpc//keepalive",
"@org_golang_google_grpc//status",
Expand Down
Loading
Loading