Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove --http-port and --port from the cockroach operator #1058

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

prafull01
Copy link
Collaborator

Fixes: #1056

@@ -1084,12 +1084,12 @@ spec:
type: boolean
type: object
grpcPort:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still just a port? Do we need to update the CRD here for something like grpcAddr and httpAddr

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we change the fields from port to grpcAddr and http-port to httpAddr, we need to have some kind of migration to actually migrate the old CRs to these new values. This is why to keep it simple I have just changed the description and not fields.

@@ -31,7 +31,7 @@ spec:
- /bin/bash
- -ecx
- 'exec /cockroach/cockroach.sh start --advertise-host=$(POD_NAME).test-cluster.test-ns
--certs-dir=/cockroach/cockroach-certs/ --http-port=8080 --sql-addr=:26257
--certs-dir=/cockroach/cockroach-certs/ --http-addr=:8080 --sql-addr=:26257
--listen-addr=:26258 --log="{sinks: {stderr: {channels: [OPS, HEALTH], redact:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, did we already make the change to listen-addr in the operator? What value is it pulling from?

Copy link
Collaborator Author

@prafull01 prafull01 Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we already have change of listen-addr which is equal to :grpcPort, I did the same thing from http-port to http-addr=:httpPort
You can check existing values coming from here

@@ -1086,12 +1086,12 @@ spec:
type: boolean
type: object
grpcPort:
description: '(Optional) The database port (`--port` CLI parameter
description: '(Optional) The database port (`--listen-addr` CLI parameter
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these changes backwards compatible?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I have only changed the description and not the field so it is backward compatible.

@@ -215,7 +215,7 @@ func (d *CockroachNodeDrainer) executeDrainCmd(ctx context.Context, id uint, gRP
// a node which will transition it from `decommissioning` to `decommissioned`. This should be executed
// after it's confirmed that there are 0 replicas on the node.
func (d *CockroachNodeDrainer) markNodeAsDecommissioned(ctx context.Context, id uint, gRPCPort int32) error {
cmd := []string{"./cockroach", "node", "decommission", fmt.Sprintf("%d", id), fmt.Sprintf("--port=%d", gRPCPort)}
cmd := []string{"./cockroach", "node", "decommission", fmt.Sprintf("%d", id), fmt.Sprintf("--host=:%d", gRPCPort)}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does --host=:<grpcPort> default to? Is it localhost?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Description specifies:

--host <addr/host>[:<port>]           
                                             CockroachDB node to connect to. This can be specified either as an
                                             address/hostname, or together with a port number as in -s myhost:26257. If
                                             the port number is left unspecified, it defaults to 26257. An IPv6 address
                                             can also be specified with the notation [...], for example [::1]:26257 or
                                             [fe80::f6f2:::]:26257.
                                             Environment variable: COCKROACH_HOST
                                             (default :26257)

It is similar to listenAddr or httpAddr where we can only specify port with :port

@prafull01 prafull01 force-pushed the remove-http-port branch 2 times, most recently from 8c47b28 to f1bf6cf Compare October 25, 2024 12:28
Copy link

@udnay udnay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure that the tests all pass on the branch, other than that LGTM

@@ -59,16 +59,28 @@ func (r *CrdbCluster) SetupWebhookWithManager(mgr ctrl.Manager) error {
func (r *CrdbCluster) Default() {
webhookLog.Info("default", "name", r.Name)

if r.Spec.GRPCPort == nil {
r.Spec.GRPCPort = &DefaultGRPCPort
if r.Spec.GRPCPort == nil && r.Spec.ListenAddr == nil {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work, I'm not super familiar with it. Do we need to persist the spec somewhere? I know we pull it often from the k8s api, do we need to update it if we change some of the values here?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Default() func always gets called in mutating configuration so whenever a create or update request comes on this func gets called and the changes we mutate on crdbCluster goes to apiserver and it gets persisted to etcd.

Now when in controller you get the crdbcluster object from apiserver it returns the object with all the mutation we did in Default() func.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if I do kubectl get crdbcluster foo -o yaml I'll see listenAddr set now?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. If you create a crdbcluster CR with these changes you will see the listenAddr, httpAddr and sqlAddr set.

If you have old CR, then you don't see it (so no automatic rolling restart). However as soon as you update that CR with new image, your rolling restart will happen and now you see addresses instead of ports.

@@ -48,7 +49,6 @@ const (

func NewCluster(original *api.CrdbCluster) Cluster {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I'm curious about, does the original crdb cluster get the defaults or will we always need to do the migration. I guess we'll always do the migration so that there is no conflict between the persisted object and whatever the customer is using to update it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will always gets the object which is processed by default (see my prev comment). Whenever we get the object in controller we get the object which has already mutated using Default func. so we don't have to migrate it again. This is the reason I have removed the call to Default func again. This could cause migration on non migrated object.

If that call is present, after upgrade the migration doesn't happen on the actual CR, but in our logic that func does the migration which is two different states for the same object. We should always use the state which is persisted in etcd.

@cockroachdb cockroachdb deleted a comment from prafull11 Oct 25, 2024
@prafull01
Copy link
Collaborator Author

Please make sure that the tests all pass on the branch, other than that LGTM

For tests to pass, I am starting a webhook server with controller manager so that this Default() func gets called and does the basic mutation on crdbcluster object. It will need some changes on test side so I will ask for a review again.

@cockroachdb cockroachdb deleted a comment from prafull11 Oct 25, 2024
@cockroachdb cockroachdb deleted a comment from prafull11 Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Depreciated --port & --http-port are used in statefulset
3 participants