Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multitenant: handle missing NodeDescriptor in crdb_internal.ranges_no_leases #93127

Merged
merged 1 commit into from
Dec 8, 2022
Merged

multitenant: handle missing NodeDescriptor in crdb_internal.ranges_no_leases #93127

merged 1 commit into from
Dec 8, 2022

Conversation

ecwall
Copy link
Contributor

@ecwall ecwall commented Dec 6, 2022

Fixes #92915

This change matches the previous behavior of using "" for locality if the NodeDescriptor is not found instead of returning an error when generating crdb_internal.ranges_no_leases.

Release note: None

@ecwall ecwall requested review from knz, rafiss and arulajmani December 6, 2022 15:14
@ecwall ecwall requested a review from a team as a code owner December 6, 2022 15:14
@ecwall ecwall requested a review from a team December 6, 2022 15:14
@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Contributor

@knz knz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani, @ecwall, and @rafiss)


pkg/sql/crdb_internal.go line 3728 at r1 (raw file):

				// Use empty string if nodeDesc is not found.
				// See https://github.com/cockroachdb/cockroach/issues/92915.
				if err == nil {

Not all errors indicate the desc was not found. Some errors should be propagated. Please use a more precise predicate here. Something like:

if err != nil {
   if !xxx(err) {
       return err
   }
} else {
  replicaLocalityString = nodeDesc.Locality.String()
}

Copy link
Collaborator

@arulajmani arulajmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the quick fix! To confirm, the linked roachtest passes with this patch, right?

Reviewed all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ecwall and @rafiss)


pkg/sql/crdb_internal.go line 3726 at r1 (raw file):

				var replicaLocalityString string
				nodeDesc, err := p.ExecCfg().NodeDescs.GetNodeDescriptor(replica.NodeID)
				// Use empty string if nodeDesc is not found.

nit: could you expand on this comment a bit to say that we don't want this table to not render if locality information isn't available for a particular node for some reason?

@ecwall ecwall requested a review from a team as a code owner December 6, 2022 15:43
@ecwall
Copy link
Contributor Author

ecwall commented Dec 6, 2022

@arulajmani @knz getting a "Sorry, you've run out of GitHub API quota." error in reviewable, but I addressed your comments.

Working on running roachtest now.

Copy link
Contributor Author

@ecwall ecwall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani, @knz, and @rafiss)


pkg/sql/crdb_internal.go line 3726 at r1 (raw file):

Previously, arulajmani (Arul Ajmani) wrote…

nit: could you expand on this comment a bit to say that we don't want this table to not render if locality information isn't available for a particular node for some reason?

Updated.


pkg/sql/crdb_internal.go line 3728 at r1 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

Not all errors indicate the desc was not found. Some errors should be propagated. Please use a more precise predicate here. Something like:

if err != nil {
   if !xxx(err) {
       return err
   }
} else {
  replicaLocalityString = nodeDesc.Locality.String()
}

Updated.

Copy link
Contributor

@knz knz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 10 of 13 files at r3, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani, @ecwall, and @rafiss)


pkg/util/errorutil/descriptor.go line 20 at r3 (raw file):

)

const descriptorNotFound = "unable to look up descriptor for"

The API here is fine, but the implementation is not. We don't use Error() comparisons any more in CockroachDB.

Try this:

type descriptorNotFound struct{msg string}
func (e *descriptorNotFound) Error() string { return e.msg }

pkg/util/errorutil/descriptor.go line 23 at r3 (raw file):

func IsDescriptorNotFoundError(err error) bool {
	return strings.HasPrefix(err.Error(), descriptorNotFound)

return errors.HasType(err, (*descriptorNotFound)(nil))


pkg/util/errorutil/descriptor.go line 27 at r3 (raw file):

func NewNodeNotFoundError(nodeID roachpb.NodeID) error {
	return errors.Errorf("%s n%d", descriptorNotFound, nodeID)

return &descriptorNotFound{fmt.Sprintf("unable to look up descriptor for n%d", nodeID")}


pkg/util/errorutil/descriptor.go line 31 at r3 (raw file):

func NewStoreNotFoundError(storeID roachpb.StoreID) error {
	return errors.Errorf("%s storeID %d", descriptorNotFound, storeID)

ditto & reuse the original error message for stores

Copy link
Contributor

@knz knz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani, @ecwall, and @rafiss)


pkg/sql/crdb_internal.go line 3726 at r1 (raw file):

Previously, ecwall (Evan Wall) wrote…

Updated.

Arul: what do you think, should we render NULL in that case?

Copy link
Contributor Author

@ecwall ecwall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani, @knz, and @rafiss)


pkg/util/errorutil/descriptor.go line 20 at r3 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

The API here is fine, but the implementation is not. We don't use Error() comparisons any more in CockroachDB.

Try this:

type descriptorNotFound struct{msg string}
func (e *descriptorNotFound) Error() string { return e.msg }

Done.


pkg/util/errorutil/descriptor.go line 23 at r3 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

return errors.HasType(err, (*descriptorNotFound)(nil))

Done.


pkg/util/errorutil/descriptor.go line 27 at r3 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

return &descriptorNotFound{fmt.Sprintf("unable to look up descriptor for n%d", nodeID")}

Done.


pkg/util/errorutil/descriptor.go line 31 at r3 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

ditto & reuse the original error message for stores

Done.

Copy link
Contributor

@knz knz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but let's let arul finalize the review

Reviewed 2 of 13 files at r3, 2 of 2 files at r4, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani, @ecwall, and @rafiss)

@ecwall
Copy link
Contributor Author

ecwall commented Dec 6, 2022

I'm getting a different error trying to reproduce this:

test_impl.go:347: test failure #1: (test_impl.go:291).Fatal: output in run_183835.339062000_n1-4_workload_init_schemachange: ./workload init schemachange returned: COMMAND_PROBLEM: exit status 126

Copy link
Collaborator

@arulajmani arulajmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once we've confirmed the roachtest passes.

Reviewed 10 of 13 files at r3, 2 of 2 files at r4, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ecwall, @knz, and @rafiss)


pkg/sql/crdb_internal.go line 3726 at r1 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

Arul: what do you think, should we render NULL in that case?

I think it's a good idea -- that way, we aren't conflating no locality string with one that couldn't be found.

Copy link
Contributor Author

@ecwall ecwall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani, @knz, and @rafiss)


pkg/sql/crdb_internal.go line 3726 at r1 (raw file):

Previously, arulajmani (Arul Ajmani) wrote…

I think it's a good idea -- that way, we aren't conflating no locality string with one that couldn't be found.

Done.

…_leases

Fixes #92915

This change matches the previous behavior of using "" for locality if the
NodeDescriptor is not found instead of returning an error when generating
crdb_internal.ranges_no_leases.

Release note: None
@ecwall
Copy link
Contributor Author

ecwall commented Dec 8, 2022

bors r=arulajmani

@craig
Copy link
Contributor

craig bot commented Dec 8, 2022

Build succeeded:

@craig craig bot merged commit 51f951c into cockroachdb:master Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

roachtest: replicate/wide failed
4 participants