-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
pkg/migration: support waiting for minimum engine version
As part of the MVCC Bulk Operations project, [Pebble range keys][1] will need to be enabled on _all_ engines of a cluster before nodes can start using the feature to read and write SSTables that contain the range key features (a backward-incompatible change). Adding a cluster version is necessary, but not sufficient in guaranteeing that nodes are ready to generate, but importantly _receive_ SSTabbles with range key support. Specifically, there exists a race condition where nodes are told to update their engines as part of the existing Pebble major format update process, but there is no coordination between the nodes. One node (a sender) may complete its engine upgrade and enable the new SSTable features _before_ another node (the receiver). The latter will panic on receipt of an SSTable with the newer features written by the former. Add an server RPC endpoint that provides a means of waiting on a node to update its store to a version that is compatible with a cluster version. This endpoint is used as part of a system migration to ensure that all nodes in a cluster are running with an engine version that is at least compatible with a given cluster version. Expose the table format major version on `storage.Engine`. This will be used elsewhere in Cockroach (for example, SSTable generation for ingest and backup). Add a `WaitForCompatibleEngineVersion` function on the `storage.Engine` interface that provides a mechanism to block until an engine is running at a format major version that compatible with a given cluster version. Expose the engine format major version as a `storage.TestingKnob` to allow tests to alter the Pebble format major version. Add a new cluster version to coordinate the upgrade of all engines in a cluster to `pebble.FormatBlockPropertyCollector` (Pebble,v1), and the system migration required for coordinating the engine upgrade to the latest Pebble table format version. This patch also fixes an existing issue where a node may write SSTables with block properties as part of a backup that are then ingested by an older node. This patch provides the infrastructure necessary for making these "cluster-external" SSTable operations engine-aware. Nodes should only use a table format version that other nodes in the cluster understand. Informs cockroachdb/pebble#1339. [1]: cockroachdb/pebble#1339 Release note: None
- Loading branch information
Showing
16 changed files
with
407 additions
and
33 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| // Copyright 2022 The Cockroach Authors. | ||
| // | ||
| // Use of this software is governed by the Business Source License | ||
| // included in the file licenses/BSL.txt. | ||
| // | ||
| // As of the Change Date specified in that file, in accordance with | ||
| // the Business Source License, use of this software will be governed | ||
| // by the Apache License, Version 2.0, included in the file | ||
| // licenses/APL.txt. | ||
|
|
||
| package migrations | ||
|
|
||
| import ( | ||
| "context" | ||
|
|
||
| "github.com/cockroachdb/cockroach/pkg/clusterversion" | ||
| "github.com/cockroachdb/cockroach/pkg/jobs" | ||
| "github.com/cockroachdb/cockroach/pkg/migration" | ||
| "github.com/cockroachdb/cockroach/pkg/server/serverpb" | ||
| ) | ||
|
|
||
| // ensureEngineVersionAtLeast waits for the engine version to be at least | ||
| // compatible with the given clusterversion.ClusterVersion, on all nodes in the | ||
| // cluster. | ||
| func ensureEngineVersionAtLeast( | ||
| ctx context.Context, v clusterversion.ClusterVersion, deps migration.SystemDeps, _ *jobs.Job, | ||
| ) error { | ||
| return deps.Cluster.UntilClusterStable(ctx, func() error { | ||
| return deps.Cluster.ForEveryNode(ctx, "ensure-engine-version", | ||
| func(ctx context.Context, client serverpb.MigrationClient) error { | ||
| req := &serverpb.WaitForEngineVersionRequest{ | ||
| Version: &v.Version, | ||
| } | ||
| _, err := client.WaitForEngineVersion(ctx, req) | ||
| return err | ||
| }) | ||
| }) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| // Copyright 2022 The Cockroach Authors. | ||
| // | ||
| // Use of this software is governed by the Business Source License | ||
| // included in the file licenses/BSL.txt. | ||
| // | ||
| // As of the Change Date specified in that file, in accordance with | ||
| // the Business Source License, use of this software will be governed | ||
| // by the Apache License, Version 2.0, included in the file | ||
| // licenses/APL.txt. | ||
|
|
||
| package migrations_test | ||
|
|
||
| import ( | ||
| "context" | ||
| "testing" | ||
|
|
||
| "github.com/cockroachdb/cockroach/pkg/base" | ||
| "github.com/cockroachdb/cockroach/pkg/clusterversion" | ||
| "github.com/cockroachdb/cockroach/pkg/kv/kvserver" | ||
| "github.com/cockroachdb/cockroach/pkg/server" | ||
| "github.com/cockroachdb/cockroach/pkg/storage" | ||
| "github.com/cockroachdb/cockroach/pkg/testutils/sqlutils" | ||
| "github.com/cockroachdb/cockroach/pkg/testutils/testcluster" | ||
| "github.com/cockroachdb/cockroach/pkg/util/leaktest" | ||
| "github.com/cockroachdb/cockroach/pkg/util/log" | ||
| "github.com/cockroachdb/pebble" | ||
| "github.com/stretchr/testify/require" | ||
| ) | ||
|
|
||
| // TestEnsureEngineVersion_SingleNode verifies that the migration of a single | ||
| // node waits for the node's engine version to be at least at a minimum | ||
| // version, blocking until it occurs. | ||
| func TestEnsureEngineVersion_SingleNode(t *testing.T) { | ||
| defer leaktest.AfterTest(t)() | ||
| defer log.Scope(t).Close(t) | ||
|
|
||
| ctx := context.Background() | ||
|
|
||
| tc := testcluster.StartTestCluster(t, 1, base.TestClusterArgs{ | ||
| ServerArgs: base.TestServerArgs{ | ||
| Knobs: base.TestingKnobs{ | ||
| Server: &server.TestingKnobs{ | ||
| DisableAutomaticVersionUpgrade: 1, | ||
| BinaryVersionOverride: clusterversion.ByKey( | ||
| // Start at early version of the binary. | ||
| clusterversion.Start22_1, | ||
| ), | ||
| }, | ||
| Store: &kvserver.StoreTestingKnobs{ | ||
| StorageKnobs: storage.TestingKnobs{ | ||
| // Start at early engine version. | ||
| FormatMajorVersion: pebble.FormatMostCompatible, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }) | ||
|
|
||
| defer tc.Stopper().Stop(ctx) | ||
| ts := tc.Server(0) | ||
|
|
||
| getFormatVersion := func() (pebble.FormatMajorVersion, error) { | ||
| e := ts.Engines()[0] | ||
| // Wait for the current version to stabilize. | ||
| currentVers := ts.ClusterSettings().Version.ActiveVersion(ctx) | ||
| if err := e.WaitForCompatibleEngineVersion(ctx, currentVers.Version); err != nil { | ||
| return 0, err | ||
| } | ||
| return e.FormatMajorVersion(), nil | ||
| } | ||
|
|
||
| // We start at Pebble major format version 4 (SetWithDelete). Note that the | ||
| // server was started with an earlier engine version (v0, "most compatible"), | ||
| // on start it will ratchet up to a compatible version. | ||
| v, err := getFormatVersion() | ||
| require.NoError(t, err) | ||
| require.Equal(t, v, pebble.FormatSetWithDelete) | ||
|
|
||
| // Bump the cluster version to include block properties. | ||
| tdb := sqlutils.MakeSQLRunner(tc.ServerConn(0)) | ||
| tdb.ExecSucceedsSoon(t, | ||
| "SET CLUSTER SETTING version = $1", | ||
| clusterversion.ByKey(clusterversion.PebbleFormatVersionBlockProperties).String(), | ||
| ) | ||
|
|
||
| // The store version has been ratcheted to version 5 (Block property | ||
| // collectors). | ||
| v, err = getFormatVersion() | ||
| require.NoError(t, err) | ||
| require.Equal(t, v, pebble.FormatBlockPropertyCollector) | ||
| } | ||
|
|
||
| // TestEnsureEngineVersion_MultiNode is the same as the above, except that it | ||
| // runs on a cluster of three nodes. | ||
| func TestEnsureEngineVersion_MultiNode(t *testing.T) { | ||
| defer leaktest.AfterTest(t)() | ||
| defer log.Scope(t).Close(t) | ||
|
|
||
| ctx := context.Background() | ||
|
|
||
| const nNodes = 3 | ||
| tc := testcluster.StartTestCluster(t, nNodes, base.TestClusterArgs{ | ||
| ServerArgs: base.TestServerArgs{ | ||
| Knobs: base.TestingKnobs{ | ||
| Server: &server.TestingKnobs{ | ||
| DisableAutomaticVersionUpgrade: 1, | ||
| BinaryVersionOverride: clusterversion.ByKey( | ||
| clusterversion.Start22_1, | ||
| ), | ||
| }, | ||
| Store: &kvserver.StoreTestingKnobs{ | ||
| StorageKnobs: storage.TestingKnobs{ | ||
| FormatMajorVersion: pebble.FormatMostCompatible, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }) | ||
|
|
||
| defer tc.Stopper().Stop(ctx) | ||
|
|
||
| getFormatVersions := func(tc *testcluster.TestCluster) ( | ||
| []pebble.FormatMajorVersion, error, | ||
| ) { | ||
| var versions [nNodes]pebble.FormatMajorVersion | ||
| for i := 0; i < nNodes; i++ { | ||
| ts := tc.Server(i) | ||
| e := ts.Engines()[0] | ||
| // Wait for the current version to stabilize. | ||
| currentVers := ts.ClusterSettings().Version.ActiveVersion(ctx) | ||
| if err := e.WaitForCompatibleEngineVersion(ctx, currentVers.Version); err != nil { | ||
| return nil, err | ||
| } | ||
| versions[i] = e.FormatMajorVersion() | ||
| } | ||
| return versions[:], nil | ||
| } | ||
|
|
||
| all := func(v pebble.FormatMajorVersion) []pebble.FormatMajorVersion { | ||
| var vs [nNodes]pebble.FormatMajorVersion | ||
| for i := 0; i < nNodes; i++ { | ||
| vs[i] = v | ||
| } | ||
| return vs[:] | ||
| } | ||
|
|
||
| // All nodes start at Pebble major format version 4 (SetWithDelete). | ||
| vs, err := getFormatVersions(tc) | ||
| require.NoError(t, err) | ||
| require.Equal(t, all(pebble.FormatSetWithDelete), vs) | ||
|
|
||
| // Bump the cluster version to include block properties. | ||
| tdb := sqlutils.MakeSQLRunner(tc.ServerConn(0)) | ||
| tdb.ExecSucceedsSoon(t, | ||
| "SET CLUSTER SETTING version = $1", | ||
| clusterversion.ByKey(clusterversion.PebbleFormatVersionBlockProperties).String(), | ||
| ) | ||
|
|
||
| // The store versions have been ratcheted to version 5 (Block property | ||
| // collectors). | ||
| vs, err = getFormatVersions(tc) | ||
| require.NoError(t, err) | ||
| require.Equal(t, all(pebble.FormatBlockPropertyCollector), vs) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.