RedisInstance host and port not showing up after creation

[Veres72]

Show RedisInstance privateip in describe command like for SQLInstance.
I can get SQLInstance private ip by command kubectl describe sqlinstance sql-example and see in the output

...
  First Ip Address:        10.10.2.12  
  Ip Address:  
    Ip Address:        10.10.2.12  
    Type:              PRIVATE  
  Private Ip Address:  10.10.2.12  
...

But not for RedisInstance

[spew]

hi @Veres72 thanks for your request and we will let you know when it is available in this issue.

Question: is there something specific you use the private IP for?

[Veres72]

Nothing special. We create Redis with a project which attaches to shared vpc.

[spew]

Thank you, we will prioritize this in the near future and let you know when we have a target date / completion.

[jcanseco]

Hi @Veres72, maybe I'm just missing something, but it doesn't seem like the REST API for Redis instances has a private IP field. Is there a particular field in there that you're referring to?

The only field I see that says anything about internal IP addresses is the reservedIpRange field, which we already support in Config Connector.

[Veres72]

HI @jcanseco
What about host and port fields?
It's exactrly what I want - see Connection properties for directly connect to Redis like CloudSQL with PrivateIP.
I'm using command for get this now.
gcloud redis instances list --project <project_id> --region <regian>

[jcanseco]

Hi @Veres72, yes, RedisInstance resources have a host and port field in the status block (see this page). Can you confirm if those are sufficient for you?

[Veres72]

I see these fields in the documentation, but not in describe command.
Maybe a problem in my version of Config Connector (1.9.2). I will try to update and came back with the result. Thank you.

[jcanseco]

Hi @Veres72, were you able to confirm if it worked for you?

[evegner]

Hi @jcanseco , we've just updated CC to latest version 1.22.0, but describe command like:
kubectl -n myns describe redisinstance myredis
Still doesn't show host and port fields.

[jcanseco]

Hi @evegner, can you run kubectl -n myns get redisinstance myredis -o yaml and share the output? The host and port should be under the status block.

If your RedisInstance doesn't have a status block at all, then it's likely that the KCC controller isn't even running. To check, can you run kubectl -n cnrm-system get pods and verify that all of them are running?

[evegner]

apiVersion: redis.cnrm.cloud.google.com/v1beta1
kind: RedisInstance
metadata:
  annotations:
    cnrm.cloud.google.com/management-conflict-prevention-policy: resource
    cnrm.cloud.google.com/project-id: myproject_id
    kubectl.kubernetes.io/last-applied-configuration: |
      {mylastconfwashere}}
  creationTimestamp: "2020-09-30T13:00:30Z"
  finalizers:
  - cnrm.cloud.google.com/finalizer
  - cnrm.cloud.google.com/deletion-defender
  generation: 2
  labels:
    mylabels
  name: myname
  namespace: myns
  resourceVersion: "281009632"
  selfLink: selflink
  uid: uid
spec:
  mysomespec
status:
  conditions:
  - lastTransitionTime: "2020-09-30T13:00:30Z"
    message: Update in progress
    reason: Updating
    status: "False"
    type: Ready

How we can see i have the status block, but without information host,port.
I also checked pods, all are running.

[jcanseco]

Thanks @evegner. It seems your RedisInstance still hasn't finished creation as seen by its Updating status. Once it has reached UpToDate, it should show a status.host and status.port field.

That said, I am not sure why it's taking so long for your RedisInstance to be created. I do know that RedisInstance can take awhile to be created depending on your configuration, but I'm not sure if it's normal for it to take this long. I can check with the team internally to further investigate.

Would it be possible for you to share your spec so that we can try reproducing your resource on our side?

[evegner]

I apply this one:

apiVersion: redis.cnrm.cloud.google.com/v1beta1
kind: RedisInstance
metadata:
  name: nameredis
spec:
  displayName: nameredis
  region: us-east4
  tier: BASIC
  memorySizeGb: 1
  redisVersion: "REDIS_4_0"
  connectMode: PRIVATE_SERVICE_ACCESS
  authorizedNetworkRef:
    external: oursharedvpc

[jcanseco]

Thanks @evegner. We'll try reproducing your resource on our side. Please note that until the RedisInstance finishes creation, you won't be able to see a status.host and status.port field.

While you are waiting, would you be able to create another RedisInstance and see if you are able to successfully create one?

[jcanseco]

@evegner. I was able to create a RedisInstance successfully with your configuration in ~5 mins.

One thing that may be influencing your RedisInstance's creation is your VPC configuration. I did some digging and found this page. It talks about how your Redis instance's creation can time out if your service project and host project are not in the same service perimeter. Can you try the troubleshooting steps described there and see if the issue applies to you?

Also, what KCC version are you on?

kubectl get ns cnrm-system -o jsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com/version}

[evegner]

@jcanseco Hello!
It is strange, because i have fully working redisinstance, no network issue or anything else, i can connect to it. Also i checked the doc, no messages in audit log like that.

$ kubectl get ns cnrm-system -o jsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com/version}'
1.22.0

Also i tried to create other redis instances, the same problem

I see that redisinstance is up-to-date:

Events:
  Type    Reason    Age                   From                      Message
  ----    ------    ----                  ----                      -------
  Normal  Updating  17m                   redisinstance-controller  Update in progress
  Normal  UpToDate  3m19s (x18 over 14m)  redisinstance-controller  The resource is up to date

[kibbles-n-bytes]

Hey @evegner , can you share as well what the status of the object in question looks like? If there is no status, we may have a situation that the event is being fired but the status is unable to be updated with the information you're expecting.

If there is a status, but still no host or port, can you confirm through gcloud redis instances describe that the expected information is available on the underlying object to begin with? And if so, could you shared what the output of that command looks like to see if it's in a spot our controller is not expecting?

[evegner]

Hello @kibbles-n-bytes !
Let me summarize, the kubectl describe:

kubectl -n myns describe redisinstance myinstance
Name:         myinstance
Namespace:    myns
Labels:       <none>
Annotations:  cnrm.cloud.google.com/management-conflict-prevention-policy: resource
              cnrm.cloud.google.com/project-id: myprojid
API Version:  redis.cnrm.cloud.google.com/v1beta1
Kind:         RedisInstance
Metadata:
  Creation Timestamp:  2020-10-16T07:27:53Z
  Finalizers:
    cnrm.cloud.google.com/finalizer
    cnrm.cloud.google.com/deletion-defender
  Generation:        2
  Resource Version:  294069211
  Self Link:         /apis/redis.cnrm.cloud.google.com/v1beta1/namespaces/myns/redisinstances/myinstance
  UID:               0541b942-efc7-4c45-bf85-342778f012a4
Spec:
  Authorized Network Ref:
    External:      mysharednetwork
  Connect Mode:    PRIVATE_SERVICE_ACCESS
  Display Name:    nameredis
  Memory Size Gb:  1
  Redis Version:   REDIS_4_0
  Region:          us-east4
  Tier:            BASIC
Status:
  Conditions:
    Last Transition Time:  2020-10-16T07:27:53Z
    Message:               Update in progress
    Reason:                Updating
    Status:                False
    Type:                  Ready
Events:
  Type    Reason    Age                    From                      Message
  ----    ------    ----                   ----                      -------
  Normal  Updating  26m (x235 over 5d10h)  redisinstance-controller  Update in progress
  Normal  UpToDate  10m (x486 over 5d10h)  redisinstance-controller  The resource is up to date

As i see, the Type is Ready, also i can connect to this instance using telnet, or redis-cli.

Here is gcloud output:

$ gcloud redis instances describe myinstance --project myprojid --region us-east4
authorizedNetwork: mynetwork
connectMode: PRIVATE_SERVICE_ACCESS
createTime: '2020-10-16T07:27:54.753273408Z'
currentLocationId: us-east4-b
displayName: nameredis
host: 10.204.*.* - i see ip of redis here, it is correct ip, i checked. I replaced some figures for security.
labels:
  cnrm-lease-expiration: '1603828127'
  cnrm-lease-holder-id: bqarm768d1j4llk65450
  managed-by-cnrm: 'true'
locationId: us-east4-b
memorySizeGb: 1
name: projects/projid/locations/us-east4/instances/myinstance
persistenceIamIdentity: mySA
port: 6379
redisVersion: REDIS_4_0
reservedIpRange: 10.204.*.*/29
state: READY
tier: BASIC

I see the host field here, is it enough or something missing? Thanks for your response.

[caieo]

Hi @evegner , it looks like your resource is stuck with an "Updating" status or is jumping between "UpToDate" and "Updating". We're not entirely sure how this instance got into this buggy state, but it's a good sign to see that the RedisInstance was actually created properly. As a workaround to try to get this resource back to 'normal', can you abandon & delete the K8s resource and then acquire the existing RedisInstance? To do this, add the abandon annotation as describe here, then just run kubectl delete on your RedisInstance (this should just delete the k8s instance and not your GCP resource). After that, you can reapply the resource (feel free to keep/remove the 'abandon' annotation), which should acquire the existing RedisInstance. When the resource is marked as "UpToDate", you should be able to see a status.host field with the relevant IP address. Let us know if the status is still stuck at Updating if too much time has elapsed -- I'm hoping this will reset the resource to normal.

Can you also run kubectl logs -n cnrm-system cnrm-controller-manager-0 and paste any relevant logs on the RedisInstance here? I also tried to recreate your issue (with the same configuration) and was unable to replicate the behavior.

[Veres72]

Hi.
Controller logs:
{"severity":"error","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"redisinstance-controller","request":"test-cnrm-170/testing-exporter-redis-06","error":"error with update call to API server: admission webhook \"deny-immutable-field-updates.cnrm.cloud.google.com\" denied the request: cannot make changes on immutable fields: [reservedIpRange locationId]"}
I see that controller is trying to change immutable fields. But as you see above, we didn't fill these parameters in our Redis manifest. I tried to delete Redis, as you told, and create it again. The situation didn't change. Redis status and controller error log exactly the same.
I added parameters reservedIpRange and locationId and recreated Redis object again. After that in the controller's log appeared errors
{"severity":"error","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"redisinstance-controller","request":"test-cnrm-170/testing-exporter-redis-06","error":"error with update call to API server: admission webhook \"deny-immutable-field-updates.cnrm.cloud.google.com\" denied the request: cannot make changes on immutable fields: [alternativeLocationId]"}
I added alternativeLocationId to manifest and recreated the object again. After that Redis object become to "Reade" state, all error in controller's log gone, and I get Host:ip fields in describe.
It looks strange.
Why the controller had this error? Of course, I can fill locationId and alternativeLocationId in Redis object manifests. But I can't do it for reservedIpRange cause I don't know this parameter before the instance will be created. Also, I didn't try to change immutable fields. I just didn't fill these fields in the manifest and in documentation these fields are optional.

[jcanseco]

Hi @Veres72, thank you for the log statements. I think I may be beginning to understand the issue.

Can you share with me which version of KCC you're on?

kubectl get ns cnrm-system -o jsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com/version}'

And do you know if you're on cluster-mode or namespaced-mode? If you do not know, we can try finding out by inspecting your cnrm-system namespace:

kubectl get -n cnrm-system all

[Veres72]

Hi.
KCC version 1.22.0
As I understand we use namespaced-mode ( controller pod for each namespace )

[jcanseco]

Gotcha, thanks @Veres72.

Can you also please share the state of your controller pod for the namespace that contains your RedisInstance?

# Replace ${NAMESPACE} with your namespace name
kubectl get pod -n cnrm-system -l cnrm.cloud.google.com/scoped-namespace=${NAMESPACE} -o yaml

[Veres72]

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      cnrm.cloud.google.com/version: 1.22.0
    generateName: cnrm-controller-manager-test-cnrm-170-7b98d79584-
    labels:
      cnrm.cloud.google.com/component: cnrm-controller-manager
      cnrm.cloud.google.com/scoped-namespace: test-cnrm-170
      cnrm.cloud.google.com/system: "true"
      pod-template-hash: 7b98d79584
    name: cnrm-controller-manager-test-cnrm-170-7b98d79584-vckgk
    namespace: cnrm-system
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: cnrm-controller-manager-test-cnrm-170-7b98d79584
      uid: 131a9f58-8831-49df-9708-b7ce2a03a017
    resourceVersion: "319727140"
    selfLink: /api/v1/namespaces/cnrm-system/pods/cnrm-controller-manager-test-cnrm-170-7b98d79584-vckgk
    uid: 7ef5677f-3aa8-4302-b90e-7d2d27cf5569
  spec:
    automountServiceAccountToken: true
    containers:
    - args:
      - --scoped-namespace=test-cnrm-170
      - --stderrthreshold=INFO
      - --prometheus-scrape-endpoint=:8888
      command:
      - /configconnector/manager
      image: gcr.io/cnrm-eap/controller:ec9fd05
      imagePullPolicy: Always
      name: manager
      readinessProbe:
        exec:
          command:
          - cat
          - /tmp/ready
        failureThreshold: 3
        initialDelaySeconds: 3
        periodSeconds: 3
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits:
          cpu: 200m
          memory: 512Mi
        requests:
          cpu: 100m
          memory: 256Mi
      securityContext:
        allowPrivilegeEscalation: true
        privileged: false
        readOnlyRootFilesystem: false
        runAsGroup: 0
        runAsNonRoot: true
        runAsUser: 1000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: cnrm-cm-rc-k3-test-cnrm-170-token-jg89c
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: gke-semrush-gke-rc-0-rc-k3-np-e2-cust-8022aa4a-0nax
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: cnrm-cm-rc-k3-test-cnrm-170
    serviceAccountName: cnrm-cm-rc-k3-test-cnrm-170
    shareProcessNamespace: false
    terminationGracePeriodSeconds: 10
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: cnrm-cm-rc-k3-test-cnrm-170-token-jg89c
      secret:
        defaultMode: 420
        secretName: cnrm-cm-rc-k3-test-cnrm-170-token-jg89c
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2020-11-13T08:05:06Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2020-11-13T08:05:23Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2020-11-13T08:05:23Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2020-11-13T08:05:06Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://552acf22a41984f1e9bc809d4265590ad58dae1dc47b3d96b80b1cf3465fe654
      image: gcr.io/cnrm-eap/controller:ec9fd05
      imageID: gcr.io/cnrm-eap/controller@sha256:d5cf7771624d4e0b62a2c8e23b6482742e99213d935008fdb859af4622d7f0e9
      lastState: {}
      name: manager
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2020-11-13T08:05:15Z"
    hostIP: 10.108.128.99
    phase: Running
    podIP: 172.25.5.3
    podIPs:
    - ip: 172.25.5.3
    qosClass: Burstable
    startTime: "2020-11-13T08:05:06Z"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[jcanseco]

Thanks @Veres72, I can see why the issue is occurring now. The spec.serviceAccountName of the controller is not compliant with the expected format of cnrm-controller-manager-${NAMESPACE}, which prevented it from being recognized by the webhook.

The controller's configuration had likely been modified at some point (e.g. prior to KCC being installed) in a way that wasn't intended to be modified.

My recommendation is to uninstall and reinstall KCC if you can. If you can't afford to abandon your existing KCC resources, you could try deleting all KCC system components with the exception of the CRDs, and then re-install an equal-or-newer version of KCC on top. Details on how you can delete all KCC system components other the CRDs can be found here.

Instructions on how to install KCC in namespaced-mode using the operator can be found here.

Please let me know if you need any help.

[jcanseco]

Hi @Veres72 @evegner, just wanted to follow-up: were you able to verify that the issue was what I found it to be, and were you able to fix it by re-installing KCC?

[Veres72]

Hi @jcanseco . Yes, we solved the problem by rename k8s service account to expected format nut without reinstall KCC. Now we can see IP address in describe command. Thank you for helping!

[xiaobaitusi]

Go ahead on mark it as fixed. Feel free to reopen it or create a new issue if you still have other questions.