Could not schedule pod - incompatible with provisioner - no new nodes added #2899
Labels
question
Issues that are support related questions
Comments
|
I'm not sure if this is the issue with the workload you are scheduling but there's a typo in one of your instance types: Are you able to post the Deployment spec that is not resulting in a new node? |
bwagner5
added
question
|
So we running now v0.19.3 same issue a lot of our gitlab jobs not sheduled - let me grep some manifests to dig deeper to find the root cause |
|
@bwagner5 thanks we fixed the typo in provisioner - and bumped karpenter to v0.19.3 - but we still have the problem - looks like that this is one example pod: removed env specific config - but in general it looks like this: |
|
Here is an other example of a pod manifest hat lead to the same error: apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: eks.privileged
pod-cleanup.gitlab.com/ttl: 2h
policies.kyverno.io/last-applied-patches: |
k4-alpha-dev-gitlab-runner--podman-fuse-svc.k4-alpha-dev-gitlab-runner--podman-fuse-svc.kyverno.io: replaced
/spec/containers/1/image
k4-alpha-dev-gitlab-runner-podman-fuse-build.k4-alpha-dev-gitlab-runner--podman-fuse-build.kyverno.io: removed
/spec/containers/1/command
creationTimestamp: "2022-12-05T09:57:55Z"
generateName: runner-wyassimf-project-25-concurrent-3
labels:
pod: runner-wyassimf-project-25-concurrent-3
name: runner-wyassimf-project-25-concurrent-37jkbf
namespace: k4-alpha-dev-gitlab-runner-gitlab-runner
resourceVersion: "335850895"
uid: 82bc87e7-2620-4487-bc17-1e96a50dfeb0
spec:
affinity: {}
containers:
- command:
- sh
- -c
- "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x
/usr/bin/bash ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec
/bin/bash \nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif
[ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec
/bin/sh \nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh \nelse\n\techo shell
not found\n\texit 1\nfi\n\n"
image: registry.dev.sh/banking-platform/tooling-docker-images/x-ray-tools:22.04.25
imagePullPolicy: IfNotPresent
name: build
resources:
limits:
cpu: "2"
github.com/fuse: "1"
memory: 4Gi
requests:
cpu: 500m
github.com/fuse: "1"
memory: 2Gi
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
- SYS_ADMIN
- MKNOD
- SYS_CHROOT
- SETFCAP
privileged: true
stdin: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /scripts-25-8747526
name: scripts
- mountPath: /logs-25-8747526
name: logs
- mountPath: /builds
name: repo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-xv7rp
readOnly: true
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
- command:
- sh
- -c
- "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x
/usr/bin/bash ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec
/bin/bash \nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif
[ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec
/bin/sh \nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh \nelse\n\techo shell
not found\n\texit 1\nfi\n\n"
image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-7178588d
imagePullPolicy: IfNotPresent
name: helper
resources:
limits:
cpu: 900m
memory: 1Gi
requests:
cpu: 500m
memory: 500Mi
securityContext:
capabilities:
add:
- SYS_ADMIN
- MKNOD
- SYS_CHROOT
- SETFCAP
- NET_ADMIN
- NET_RAW
privileged: true
stdin: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /scripts-25-8747526
name: scripts
- mountPath: /logs-25-8747526
name: logs
- mountPath: /builds
name: repo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-xv7rp
readOnly: true
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostAliases:
- hostnames:
- 12345.dkr.ecr.eu-central-1.amazonaws.com-devops-tools-tooling-docker-images-podman-runner
- podman
ip: 127.0.0.1
imagePullSecrets:
- name: docker-io-image-pull-secret
- name: runner-wyassimf-project-25-concurrent-3wfv9b
initContainers:
- command:
- sh
- -c
- touch /logs-25-8747526/output.log && (chmod 777 /logs-25-8747526/output.log
|| exit 0)
image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-7178588d
imagePullPolicy: IfNotPresent
name: init-permissions
resources:
limits:
cpu: 900m
memory: 1Gi
requests:
cpu: 500m
memory: 500Mi
securityContext:
capabilities:
add:
- MKNOD
- SYS_CHROOT
- SETFCAP
- NET_ADMIN
- NET_RAW
- SYS_ADMIN
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /scripts-25-8747526
name: scripts
- mountPath: /logs-25-8747526
name: logs
- mountPath: /builds
name: repo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-xv7rp
readOnly: true
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
nodeName: ip-100-64-36-105.eu-central-1.compute.internal
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: k4-alpha-dev-gitlab-runner-gitlab-runner-gitlab-runner
serviceAccountName: k4-alpha-dev-gitlab-runner-gitlab-runner-gitlab-runner
terminationGracePeriodSeconds: 0
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
- effect: NoSchedule
key: github.com/fuse
operator: Exists
volumes:
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
- emptyDir: {}
name: repo
- configMap:
defaultMode: 511
name: runner-wyassimf-project-25-concurrent-3-scriptsxflxq
optional: false
name: scripts
- emptyDir: {}
name: logs
- name: kube-api-access-xv7rp
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespaceand the corresponding log from carpenter: |
|
There is a See issue kubernetes-sigs/karpenter#751 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
Karpenter Version: 0.16.3
Kubernetes Version: Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-fb459a0", GitCommit:"b07006b2e59857b13fe5057a956e86225f0e82b7", GitTreeState:"clean", BuildDate:"2022-10-24T20:32:54Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Expected Behavior
karpenter will upscale and add new nodes to the cluster
Actual Behavior
karpenter will not upscale and no new node is added to the cluster
Steps to Reproduce the Problem
Resource Specs and Logs
the example-services-tenant provisioner spec looks like:
Community Note
The text was updated successfully, but these errors were encountered: