Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/sqlproxyccl: include DRAINING pods in the directory cache #79368

Merged

Conversation

jaylim-crl
Copy link
Collaborator

@jaylim-crl jaylim-crl commented Apr 4, 2022

Previously, #67452 removed DRAINING pods from the directory cache. This commit
adds that back. The connector will now need to filter for RUNNING pods manually
before invoking the balancer. This is needed so that we could track DRAINING
pods, and wait until 60 seconds has elapsed before transferring connections
away from them. To support that, we also update the Pod's proto definition to
include a StateTimestamp field to reprevent that timestamp that the state field
was last updated.

The plan is to have a polling mechanism every X seconds to check DRAINING pods,
and use that information to start migrating connections.

Release note: None

Jira issue: CRDB-14759

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@jaylim-crl jaylim-crl marked this pull request as ready for review April 4, 2022 19:38
@jaylim-crl jaylim-crl requested review from a team as code owners April 4, 2022 19:38
@jaylim-crl jaylim-crl requested review from jeffswenson and andy-kimball and removed request for a team April 4, 2022 19:39
Copy link
Contributor

@andy-kimball andy-kimball left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @jaylim-crl and @jeffswenson)


pkg/ccl/sqlproxyccl/tenant/directory.proto, line 48 at r1 (raw file):

  // TenantID is the tenant that owns the pod.
  uint64 tenant_id = 2 [(gogoproto.customname) = "TenantID"];
  // addr is the ip and port combination identifying the tenant pod, (e.g.

NIT: The first letter of these comments should be capitalized, b/c this comment gets written to the generated Go file, where the field is capitalized. Our convention is to make the comment match the generated Go definition, not the protobuf definition.


pkg/ccl/sqlproxyccl/tenant/directory_cache.go, line 197 at r1 (raw file):

	// Trigger resumption if there are no RUNNING pods.
	runningPods := make([]*Pod, 0, len(tenantPods))

You're not actually using runningPods here. Instead, you can iterate through the tenantPods list and keep a foundRunning boolean that you set if you find a RUNNING pod.

Previously, cockroachdb#67452 removed DRAINING pods from the directory cache. This commit
adds that back. The connector will now need to filter for RUNNING pods manually
before invoking the balancer. This is needed so that we could track DRAINING
pods, and wait until 60 seconds has elapsed before transferring connections
away from them. To support that, we also update the Pod's proto definition to
include a StateTimestamp field to reprevent that timestamp that the state field
was last updated.

The plan is to have a polling mechanism every X seconds to check DRAINING pods,
and use that information to start migrating connections.

Release note: None
@jaylim-crl jaylim-crl force-pushed the jay/220404-cache-draining-pods branch from 5c78e85 to 7586717 Compare April 5, 2022 05:07
@jaylim-crl jaylim-crl requested a review from andy-kimball April 5, 2022 05:08
Copy link
Collaborator Author

@jaylim-crl jaylim-crl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR!

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andy-kimball and @jeffswenson)


pkg/ccl/sqlproxyccl/tenant/directory.proto, line 48 at r1 (raw file):

Previously, andy-kimball (Andy Kimball) wrote…

NIT: The first letter of these comments should be capitalized, b/c this comment gets written to the generated Go file, where the field is capitalized. Our convention is to make the comment match the generated Go definition, not the protobuf definition.

Done.


pkg/ccl/sqlproxyccl/tenant/directory_cache.go, line 197 at r1 (raw file):

Previously, andy-kimball (Andy Kimball) wrote…

You're not actually using runningPods here. Instead, you can iterate through the tenantPods list and keep a foundRunning boolean that you set if you find a RUNNING pod.

Whoops - done.

Copy link
Collaborator

@jeffswenson jeffswenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jaylim-crl
Copy link
Collaborator Author

TFTR!

bors r=JeffSwenson

@craig
Copy link
Contributor

craig bot commented Apr 5, 2022

Already running a review

@craig craig bot merged commit 985344a into cockroachdb:master Apr 5, 2022
@craig
Copy link
Contributor

craig bot commented Apr 5, 2022

Build succeeded:

@jaylim-crl jaylim-crl deleted the jay/220404-cache-draining-pods branch November 30, 2022 16:55
@jaylim-crl jaylim-crl restored the jay/220404-cache-draining-pods branch November 30, 2022 16:55
@jaylim-crl jaylim-crl deleted the jay/220404-cache-draining-pods branch November 30, 2022 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants