Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

settings,*: update the settings.Setting API #108902

Merged
merged 1 commit into from
Aug 18, 2023

Conversation

knz
Copy link
Contributor

@knz knz commented Aug 17, 2023

Fixes #108903.
Informs: #108508
Informs: #109046
Epic: CRDB-27642

TLDR: this commit introduces an API distinction between:

  • the name of a cluster setting used for user-facing UX, including the SQL syntax and error messages.
  • the key of a cluster setting used to store its value in system.settings and organize various data structures.

(In this commit, the name and key remain equivalent to each other; only the interfaces change.)

This change achieves the following:

  • it introduces specific Go types for both key and name. The benefit here is to avoid the go string type; and force users of the API to consider what data is used through the settings API.

    As a side benefit, it ensures that every setting key or name included in logs and error messages is not redacted away.

  • it paves the way for a later change where the name is allowed to diverge from the key. This will allow us to enhance UX without breaking compatibility.

  • as a side-benefit, it also marks the "setting origin" attribute
    as non-redactable.

Salient API change, prior to this change:

type Setting interface {
  // Key returns the name of the specific cluster setting.
  Key() string
}

After this change:

type Setting interface {
  // InternalKey returns the internal key used to store the setting.
  // To display the name of the setting (eg. in errors etc) or the
  // SET/SHOW CLUSTER SETTING statements, use the Name() method instead.
  //
  // The internal key is suitable for use with:
  // - direct accesses to system.settings.
  // - rangefeed logic associated with system.settings.
  // - (in unit tests only) interchangeably with the name for SET CLUSTER SETTING
  InternalKey() InternalKey

  // Name is the user-visible (display) name of the setting.
  // This is suitable for:
  // - SHOW/SET CLUSTER SETTING.
  // - inclusion in error messages.
  Name() SettingName
}

Release note: None

@knz knz requested a review from a team as a code owner August 17, 2023 13:24
@knz knz requested a review from a team August 17, 2023 13:24
@knz knz requested review from a team as code owners August 17, 2023 13:24
@knz knz requested a review from a team August 17, 2023 13:24
@knz knz requested a review from a team as a code owner August 17, 2023 13:24
@knz knz requested review from dhartunian, lidorcarmel, jayshrivastava and yuzefovich and removed request for a team August 17, 2023 13:24
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@knz knz force-pushed the 20230817-rename-cluster-settings branch 4 times, most recently from 1d3b3f4 to 5458d35 Compare August 17, 2023 15:57
@knz knz force-pushed the 20230817-rename-cluster-settings branch 3 times, most recently from 4c9ce54 to 7b6910e Compare August 17, 2023 19:23
@knz knz requested a review from stevendanna August 17, 2023 20:16
Copy link
Member

@yuzefovich yuzefovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I have some nits that are related to the following questions:

  • do we want to apply sql.defaults restriction to only keys, only names, or both?
  • it seems like in some unit tests we could be using Name in more places. Is there a reason why not? It seems like it'd be nice to use Name unless it doesn't work because we need InternalKey type.

Reviewed 70 of 70 files at r1, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @knz and @stevendanna)


pkg/cli/gen.go line 265 at r1 (raw file):

				alterRoleLink = `<a href="alter-role.html"><code>ALTER ROLE... SET</code></a>`
			}
			if strings.Contains(string(name), "sql.defaults") {

Shouldn't this be key? Or perhaps both?


pkg/server/admin.go line 2041 at r1 (raw file):

			for _, k := range settingsKeys {
				if slices.Contains(settings.ConsoleKeys(), k) {
					newSettingsKeys = append(settingsKeys, k)

I think this should be newSettingKeys = append(newSettingKeys, k).


pkg/server/settingswatcher/settings_watcher.go line 269 at r1 (raw file):

		}
		if setting.Class() != settings.TenantWritable {
			log.Warningf(ctx, "ignoring read-only setting %s", settingKey)

nit: should we use Name here?


pkg/server/tenantsettingswatcher/overrides_store.go line 149 at r1 (raw file):

		after = append(after, setting)
	}
	// Skip any existing setting for this name.

nit: perhaps replace "name" with "setting key" or something.


pkg/settings/registry.go line 320 at r1 (raw file):

// NameToKey returns the key associated with a setting name.
func NameToKey(name SettingName) (InternalKey, bool) {
	// TODO(...): improve this.

nit: I think our style guide says always include the github username in TODOs even if you're personally not planning to address it.


pkg/settings/registry.go line 349 at r1 (raw file):

// LookupForReporting returns a Setting by name. Used when a setting is being
// retrieved for reporting.

nit: perhaps insert an empty line (for symmetry with the comment below).


pkg/sql/show_cluster_setting.go line 168 at r1 (raw file):

	}

	if strings.HasPrefix(n.Name, "sql.defaults") {

nit: perhaps use name here too.


pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go line 1462 at r1 (raw file):

	}
	for _, setting := range settings {
		sysDB.Exec(t, fmt.Sprintf("ALTER TENANT ALL SET CLUSTER SETTING %s = $1", setting.InternalKey()), setting.Default()*100)

nit: why not use Name here?


pkg/sql/show_test.go line 1217 at r1 (raw file):

	for rows.Next() {
		var settingKey, sType, desc string

nit: technically speaking, shouldn't it be named settingName? IIUC SHOW ALL CLUSTER SETTINGS uses crdb_internal.cluster_settings virtual table which is now updated to return the setting name.

Relatedly, perhaps we want to run the linter on both the key and the name?


pkg/sql/catalog/schematelemetry/schema_telemetry_test.go line 67 at r1 (raw file):

	qSet = fmt.Sprintf(`SET CLUSTER SETTING %s = '* * * * *'`,
		schematelemetrycontroller.SchemaTelemetryRecurrence.InternalKey())

nit: this also seems like it could be Name.


pkg/upgrade/upgrades/ensure_sql_schema_telemetry_schedule_test.go line 99 at r1 (raw file):

		// Check that the schedule can have its recurrence altered.
		tdb.Exec(t, fmt.Sprintf(`SET CLUSTER SETTING %s = '* * * * *'`,
			schematelemetrycontroller.SchemaTelemetryRecurrence.InternalKey()))

nit: ditto here and below.

@knz knz force-pushed the 20230817-rename-cluster-settings branch from 7b6910e to fec3267 Compare August 18, 2023 08:12
Copy link
Contributor Author

@knz knz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to apply sql.defaults restriction to only keys, only names, or both?

Let's do both.

it seems like in some unit tests we could be using Name in more places. Is there a reason why not?

There's no good reason. Let's do it.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @stevendanna and @yuzefovich)


pkg/cli/gen.go line 265 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

Shouldn't this be key? Or perhaps both?

Done.


pkg/server/admin.go line 2041 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

I think this should be newSettingKeys = append(newSettingKeys, k).

Good catch. Fixed.


pkg/server/settingswatcher/settings_watcher.go line 269 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: should we use Name here?

We could, but it's not in scope. I don't feel it's worth performing a lookup.


pkg/server/tenantsettingswatcher/overrides_store.go line 149 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: perhaps replace "name" with "setting key" or something.

Done.


pkg/settings/registry.go line 320 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: I think our style guide says always include the github username in TODOs even if you're personally not planning to address it.

Done.


pkg/settings/registry.go line 349 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: perhaps insert an empty line (for symmetry with the comment below).

Done.


pkg/sql/show_cluster_setting.go line 168 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: perhaps use name here too.

Done.


pkg/ccl/multitenantccl/tenantcostclient/tenant_side_test.go line 1462 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: why not use Name here?

IMHO in tests it doesn't matter as much. But ok - fixed.


pkg/sql/show_test.go line 1217 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: technically speaking, shouldn't it be named settingName? IIUC SHOW ALL CLUSTER SETTINGS uses crdb_internal.cluster_settings virtual table which is now updated to return the setting name.

Relatedly, perhaps we want to run the linter on both the key and the name?

Actually the linter is right here -- it only needs to operate on names, which is the only thing that's user-facing. So the logic was right from the start.
Just needed to rename settingKey -> settingName here. Done.


pkg/sql/catalog/schematelemetry/schema_telemetry_test.go line 67 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: this also seems like it could be Name.

Indeed. Fixed.


pkg/upgrade/upgrades/ensure_sql_schema_telemetry_schedule_test.go line 99 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: ditto here and below.

Done.

Copy link
Member

@yuzefovich yuzefovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 15 of 15 files at r2, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @knz and @stevendanna)


pkg/kv/kvpb/api.proto line 3259 at r2 (raw file):

}

// TenantSetting contains the name and value of a tenant setting.

nit: perhaps s/name/internal key/.


pkg/server/serverpb/admin.proto line 620 at r2 (raw file):

      bool public = 4;
      google.protobuf.Timestamp last_updated = 5 [(gogoproto.nullable) = true, (gogoproto.stdtime) = true];
      string name = 6; // for display purposes.

nit: this comment is included in generated docs, should we make it a full sentence that starts with the capital?

TLDR: this commit introduces an API distinction between:

- the *name* of a cluster setting used for user-facing UX,
  including the SQL syntax and error messages.
- the *key* of a cluster setting used to store its value
  in `system.settings` and organize various data structures.

(In this commit, the name and key remain equivalent to each other;
only the interfaces change.)

This change achieves the following:

- it introduces specific Go types for both key and name. The benefit
  here is to avoid the go `string` type; and force users of the API to
  consider what data is used through the settings API.

  As a side benefit, it ensures that every setting key or name
  included in logs and error messages is not redacted away.

- it paves the way for a later change where the name is allowed to
  diverge from the key. This will allow us to enhance UX without
  breaking compatibility.

- as a side-benefit, it also marks the "setting origin" attribute
  as non-redactable.

Salient API change, prior to this change:
```go
type Setting interface {
  // Key returns the name of the specific cluster setting.
  Key() string
}
```

After this change:

```go
type Setting interface {
  // InternalKey returns the internal key used to store the setting.
  // To display the name of the setting (eg. in errors etc) or the
  // SET/SHOW CLUSTER SETTING statements, use the Name() method instead.
  //
  // The internal key is suitable for use with:
  // - direct accesses to system.settings.
  // - rangefeed logic associated with system.settings.
  // - (in unit tests only) interchangeably with the name for SET CLUSTER SETTING
  InternalKey() InternalKey

  // Name is the user-visible (display) name of the setting.
  // This is suitable for:
  // - SHOW/SET CLUSTER SETTING.
  // - inclusion in error messages.
  Name() SettingName
}
```

Release note: None
@knz knz force-pushed the 20230817-rename-cluster-settings branch from f8ce0c2 to 56e07c1 Compare August 18, 2023 20:56
Copy link
Contributor Author

@knz knz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @stevendanna)


pkg/kv/kvpb/api.proto line 3259 at r2 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: perhaps s/name/internal key/.

Done.


pkg/server/serverpb/admin.proto line 620 at r2 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: this comment is included in generated docs, should we make it a full sentence that starts with the capital?

Done.

@knz
Copy link
Contributor Author

knz commented Aug 18, 2023

TFYR!

bors r=yuzefovich

@craig
Copy link
Contributor

craig bot commented Aug 18, 2023

Build succeeded:

@craig craig bot merged commit 9ce61b0 into cockroachdb:master Aug 18, 2023
@knz knz deleted the 20230817-rename-cluster-settings branch August 19, 2023 07:29
craig bot pushed a commit that referenced this pull request Aug 19, 2023
109012: settings,*: use the option pattern with the setting register functions r=yuzefovich a=knz

First commit from  #108902.
Fixes #109011.
Informs #109046.

TLDR: this patch introduces the "functional option style" in the
settings API.

For example, this call:
```go
var s = func() *settings.BoolSetting {
  s := settings.RegisterBoolSetting(...).WithPublic()
  s.SetReportable(true)
}()
```

can now be written as:
```go
var s = settings.RegisterBoolSetting(...,
  settings.WithPublic,
  settings.WithReportable(true),
)
```

Internally, the register function now take a last argument of type
`...SettingOption`.
The following options are defined:

```go
var Retired SettingOption
var WithPublic SettingOption
func WithReportable(reportable bool) SettingOption
func WithVisibility(v Visibility) SettingOption

func WithValidateDuration(fn func(time.Duration) error) SettingOption // NEW
var NonNegativeDuration SettingOption
var PositiveDuration SettingOption
func NonNegativeDurationWithMaximum(maxValue time.Duration) SettingOption
func DurationWithMinimum(minValue time.Duration) SettingOption // RENAMED
func DurationWithMinimumOrZeroDisable(minValue time.Duration) SettingOption // NEW
func DurationInRange(minVal, maxVal int64) SettingOption // NEW

func WithValidateFloat(fn func(float64) error) SettingOption // NEW
var NonNegativeFloat SettingOption
var PositiveFloat SettingOption
func NonNegativeFloatWithMaximum(maxValue float64) SettingOption
func FloatWithMinimum(minValue float64) SettingOption // NEW
func FloatWithMinimumOrZeroDisable(minValue float64) SettingOption // NEW
var NonZeroFloat SettingOption // NEW
var Fraction SettingOption // NEW
var FractionUpperExclusive SettingOption // NEW
func FloatInRange(minVal, maxVal float64) SettingOption // NEW
func FloatInRangeUpperExclusive(minVal, maxVal float64) SettingOption // NEW

func WithValidateInt(fn func(int64) error) SettingOption // NEW
var NonNegativeInt SettingOption
var PositiveInt SettingOption
func NonNegativeIntWithMaximum(maxValue int64) SettingOption
func IntWithMinimum(minValue int64) SettingOption // NEW
func IntInRange(minVal, maxVal int64) SettingOption // NEW
func IntInRangeOrZeroDisable(minVal, maxVal int64) SettingOption // NEW

func WithValidateProto(fn func(*Values, protoutil.Message) error) SettingOption // NEW

func WithValidateString(fn func(*Values, string) error) SettingOption // NEW

func ByteSizeWithMinimum(minVal int64) SettingOption // NEW
```

Release note: None
Epic: CRDB-28893

Co-authored-by: Raphael 'kena' Poss <[email protected]>
knz added a commit to knz/cockroach that referenced this pull request Aug 20, 2023
This (hidden) bug was left over from cockroachdb#108902.
There is no visible negative effect to this bug being present until
setting aliases are effectively introduced, which hasn't happened yet.

Release note: None
@knz knz mentioned this pull request Aug 20, 2023
craig bot pushed a commit that referenced this pull request Aug 20, 2023
109098: sql: fix a leftover bug r=stevendanna a=knz

This (hidden) bug was left over from #108902.
There is no visible negative effect to this bug being present until setting aliases are effectively introduced, which hasn't happened yet.

No tests included here -- this is exercised by #109077 and #109074 and the CI in those PRs readily fail without this fix.

 Epic: CRDB-27642

Co-authored-by: Raphael 'kena' Poss <[email protected]>
knz added a commit to knz/cockroach that referenced this pull request Aug 21, 2023
This (hidden) bug was left over from cockroachdb#108902.
There is no visible negative effect to this bug being present until
setting aliases are effectively introduced, which hasn't happened yet.

The code modified here is already properly exercised by logic tests.

Release note: None
knz added a commit to knz/cockroach that referenced this pull request Aug 21, 2023
This (hidden) bug was left over from cockroachdb#108902.
There is no visible negative effect to this bug being present until
setting aliases are effectively introduced, which hasn't happened yet.

The code modified here is already properly exercised by logic tests.

Release note: None
craig bot pushed a commit that referenced this pull request Aug 21, 2023
109118: sql: fix another buglet with tenant settings r=stevendanna a=knz

This (hidden) bug was left over from #108902.
There is no visible negative effect to this bug being present until setting aliases are effectively introduced, which hasn't happened yet.

The code modified here is already properly exercised by logic tests.
Also, this is exercised by #109077 and #109074 and the CI in those PRs readily fail without this fix.

Epic: CRDB-27642

Co-authored-by: Raphael 'kena' Poss <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sql: the cluster names are too often marked redactable in logs and errors
3 participants