Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unknown field "consolidation" in sh.karpenter.v1alpha5.Provisioner.spec #2335

Closed
gaganyaan2 opened this issue Aug 22, 2022 · 2 comments
Closed

Unknown field "consolidation" in sh.karpenter.v1alpha5.Provisioner.spec #2335

gaganyaan2 opened this issue Aug 22, 2022 · 2 comments
Assignees
@tzneal
Labels
question Further information is requested

Comments

@gaganyaan2
Copy link

gaganyaan2 commented Aug 22, 2022 •
edited
Loading

I'm trying to use consolidation feature in karpenter 0.15.0 but having below error. Here is feature #2123 merge request.

Version

Karpenter: v0.15.0 upgraded from 0.12.1 using helm upgrade --install

Kubernetes: v1.20.11-eks

Expected Behavior

should accept consolidation

spec:
  consolidation:
    enabled: true

Actual Behavior

When I apply below Provisioner it gives this error

error: error validating "dev_karpenter_provisioner_worker.yaml": error validating data: ValidationError(Provisioner.spec): unknown field "consolidation" in sh.karpenter.v1alpha5.Provisioner.spec; if you choose to ignore these errors, turn validation off with --validate=false

worker-provisioner.yaml

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: worker-provisioner
spec:
  consolidation:
    enabled: true
  ttlSecondsAfterEmpty: 30
  labels:
    intent: worker
    role: worker
  requirements:
    - key: "topology.kubernetes.io/zone" 
      operator: In
      values: ["eu-west-1a","eu-west-1b"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand","spot"]
    - key: kubernetes.io/arch
      operator: In
      values: ["amd64"]
    - key: node.kubernetes.io/instance-type
      operator: In
      values: ["c5a.2xlarge","c5a.4xlarge","c5.4xlarge","c5.4xlarge","m5a.4xlarge"]
  limits:
    resources:
      cpu: "5000000000"
      memory: "1000000000000"
  provider:
    launchTemplate: dev-Karpenter-cluster-name-lt
    subnetSelector:
      karpeter.sh/zone: "*"
    tags:
      Name: karpenter.sh/worker
      aws-node-termination-handler/managed: "true"
      karpenter.sh/nodegroup: "worker"

Steps to Reproduce the Problem

  1. upgrade karpenter from 0.12.1 to 0.15.0
  2. Apply above worker-provisioner.yaml

Resource Specs and Logs

user@laptop$ helm ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
karpenter       karpenter       3               2022-08-22 06:26:15.465459979 +0000 UTC deployed        karpenter-0.15.0        0.15.0

karpenter deployment

user@laptop$ k get deploy karpenter -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
    meta.helm.sh/release-name: karpenter
    meta.helm.sh/release-namespace: karpenter
  creationTimestamp: "2022-08-08T06:22:04Z"
  generation: 4
  labels:
    app.kubernetes.io/instance: karpenter
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: karpenter
    app.kubernetes.io/version: 0.15.0
    helm.sh/chart: karpenter-0.15.0
  name: karpenter
  namespace: karpenter
  resourceVersion: "797854638"
  uid: 28082d46-2583-42c6-9160-a8893e942bd9
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: karpenter
      app.kubernetes.io/name: karpenter
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-08-17T05:41:14Z"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: karpenter
        app.kubernetes.io/name: karpenter
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: karpenter.sh/provisioner-name
                operator: DoesNotExist
      containers:
      - env:
        - name: CLUSTER_NAME
          value: masked
        - name: CLUSTER_ENDPOINT
          value: masked
        - name: KARPENTER_SERVICE
          value: karpenter
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: public.ecr.aws/karpenter/controller:v0.15.0@sha256:7fc8a4bd3c2a7eea29e40c0db287eeb9deab0797987f0089c08c71da91c98523
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 8080
          name: http-metrics
          protocol: TCP
        - containerPort: 8081
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - args:
        - -port=8443
        env:
        - name: CLUSTER_NAME
          value: masked
        - name: KUBERNETES_MIN_VERSION
          value: 1.19.0-0
        - name: CLUSTER_ENDPOINT
          value: masked
        - name: KARPENTER_SERVICE
          value: karpenter
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: KUBERNETES_MIN_VERSION
          value: 1.20.15-eks-84b4fe6
        image: public.ecr.aws/karpenter/webhook:v0.15.0@sha256:e046a8810150ce145c737e3330259ab297ad11fb32cf70dc50c97622b7d500d8
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: https-webhook
            scheme: HTTPS
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: webhook
        ports:
        - containerPort: 8443
          name: https-webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: https-webhook
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 200m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 50Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
      serviceAccount: karpenter
      serviceAccountName: karpenter
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: karpenter
            app.kubernetes.io/name: karpenter
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway

karpenter pod log

user@laptop:$ k logs -f karpenter-758f7584c6-bc7pk
Defaulted container "controller" out of: controller, webhook
2022-08-22T07:49:29.071Z        INFO    Successfully created the logger.
2022-08-22T07:49:29.071Z        INFO    Logging level set to: debug
2022-08-22T07:49:29.076Z        INFO    controller      Initializing with version v0.15.0       {"commit": "3d87474"}
{"level":"info","ts":1661154569.076588,"logger":"fallback","caller":"injection/injection.go:61","msg":"Starting informers..."}
2022-08-22T07:49:29.113Z        DEBUG   controller.aws  Using AWS region eu-west-1      {"commit": "3d87474"}
2022-08-22T07:49:29.198Z        INFO    controller.aws.pricing  Updating EC2 pricing information        {"commit": "3d87474"}
2022-08-22T07:49:29.199Z        DEBUG   controller.aws  Discovered caBundle, length 1025        {"commit": "3d87474"}
2022-08-22T07:49:29.199Z        INFO    controller      loading config from karpenter/karpenter-global-settings {"commit": "3d87474"}
I0822 07:49:29.309597       1 leaderelection.go:243] attempting to acquire leader lease karpenter/karpenter-leader-election...
2022-08-22T07:49:29.310Z        INFO    controller      starting metrics server {"commit": "3d87474", "path": "/metrics"}
2022-08-22T07:49:29.561Z        INFO    controller.aws.pricing  updated spot pricing with 540 instance types    {"commit": "3d87474"}
2022-08-22T07:49:38.276Z        INFO    controller.aws.pricing  updated on-demand pricing with 547 instance types       {"commit": "3d87474"}
I0822 07:49:45.786796       1 leaderelection.go:253] successfully acquired lease karpenter/karpenter-leader-election
2022-08-22T07:49:45.786Z        DEBUG   controller.aws.launchtemplate   Hydrating the launch template cache with tags matching "karpenter.k8s.aws/cluster: masked"     {"commit": "3d87474"}
2022-08-22T07:49:45.787Z        INFO    controller.controller.provisioning      Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.787Z        INFO    controller.controller.provisioning      Starting Controller     {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-08-22T07:49:45.788Z        INFO    controller.controller.provisioning      Starting workers        {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 10}
2022-08-22T07:49:45.788Z        INFO    controller.controller.node-state        Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.788Z        INFO    controller.controller.node-state        Starting Controller     {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-08-22T07:49:45.788Z        INFO    controller.controller.pod-state Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.788Z        INFO    controller.controller.pod-state Starting Controller     {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-08-22T07:49:45.789Z        INFO    controller.controller.node      Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.789Z        INFO    controller.controller.node      Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.789Z        INFO    controller.controller.node      Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.789Z        INFO    controller.controller.node      Starting Controller     {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-08-22T07:49:45.789Z        INFO    controller.controller.termination       Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.789Z        INFO    controller.controller.termination       Starting Controller     {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-08-22T07:49:45.789Z        INFO    controller.controller.podmetrics        Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.789Z        INFO    controller.controller.podmetrics        Starting Controller     {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-08-22T07:49:45.790Z        INFO    controller.controller.provisionermetrics        Starting EventSource    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.790Z        INFO    controller.controller.provisionermetrics        Starting Controller     {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-08-22T07:49:45.790Z        INFO    controller.controller.counter   Starting EventSource    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.790Z        INFO    controller.controller.counter   Starting EventSource    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-08-22T07:49:45.790Z        INFO    controller.controller.counter   Starting Controller     {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-08-22T07:49:45.851Z        DEBUG   controller.aws.launchtemplate   Finished hydrating the launch template cache with 0 items       {"commit": "3d87474"}
2022-08-22T07:49:45.889Z        INFO    controller.controller.pod-state Starting workers        {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 10}
2022-08-22T07:49:45.891Z        INFO    controller.controller.podmetrics        Starting workers        {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 1}
2022-08-22T07:49:45.992Z        INFO    controller.controller.node-state        Starting workers        {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-08-22T07:49:45.992Z        INFO    controller.controller.termination       Starting workers        {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-08-22T07:49:45.993Z        INFO    controller.controller.counter   Starting workers        {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 10}
2022-08-22T07:49:46.046Z        INFO    controller.controller.node      Starting workers        {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-08-22T07:49:46.059Z        INFO    controller.controller.provisionermetrics        Starting workers        {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 1}
2022-08-22T07:49:57.244Z        DEBUG   controller.consolidation        Discovered 542 EC2 instance types       {"commit": "3d87474"}
2022-08-22T07:49:57.320Z        DEBUG   controller.consolidation        Discovered subnets: [subnet-masked(eu-west-1b) subnet-masked(eu-west-1a) subnet-masked(eu-west-1b) subnet-masked(eu-west-1a) subnet-masked(eu-west-1b)]  {"commit": "3d87474"}
2022-08-22T07:49:57.463Z        DEBUG   controller.consolidation        Discovered EC2 instance types zonal offerings for subnets {"karpeter.sh/zone":"*"} {"commit": "3d87474"}

Please help.
@gaganyaan2 gaganyaan2 added the bug Something isn't working label Aug 22, 2022
@tzneal
Copy link
Contributor

tzneal commented Aug 22, 2022

Helm doesn't upgrade CRDs so you'll need to upgrade the provisioner CRD manually. See the upgrade guide at https://karpenter.sh/v0.15.0/upgrade-guide/#upgrading-to-v0150 for instructions.

@tzneal tzneal added question Further information is requested and removed bug Something isn't working labels Aug 22, 2022
@tzneal tzneal self-assigned this Aug 22, 2022
@gaganyaan2
Copy link
Author

After applying the CRD of 0.15.0 I'm able to use consolidation.

kubectl replace -f https://raw.githubusercontent.com/aws/karpenter/v0.15.0/charts/karpenter/crds/karpenter.sh_provisioners.yaml

Also removed ttlSecondsAfterEmpty as it can not work with consolidation.

I did try deleting the helm chart with helm delete karpneter and installed it again but it had no luck. I think it's the same reason that you mention that helm does not upgrade CRD.

Thank you! @tzneal

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tzneal tzneal

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tzneal @gaganyaan2