cluster-autoscaler logging another pod's env-vars.

[arukaen]

Hello,

I am running K8S v1.16.3 and k8s.gcr.io/cluster-autoscaler:v1.16.3 and I've noticed in Loggly that cluster-autoscaler will randomly dump all the environment variables of another pod to logs. Is anyone else experiencing this?

[MaciekPytel]

Can you provide a specific logline you're seeing (minus any confidential data, etc). It's certainly possible that CA accidentally logs a Pod object somewhere, but it would be much easier to find with more data.

[arukaen]

@MaciekPytel Heres an example (I shorted it because its pretty long):

"log":"-----END RSA PRIVATE KEY-----,ValueFrom:nil,},EnvVar{Name:SAMPLE_BOARD_IDS,Value:,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS,Value:,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_HIGH_FIDELITY_PROTOTYPE,Value:5,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_INTERACTIVE_WIREFRAME,Value:4,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_MOBILE_HIGH_FIDELITY_PROTOTYPE,Value:6,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_MOBILE_INTERACTIVE_WIREFRAME,Value:7,ValueFrom:nil,},EnvVar{Name:SEGMENT_WRITE_KEY,Value:verysecret,ValueFrom:nil,},EnvVar{Name:SELECTIVE_SYNC_URL,Value:http://service-url,ValueFrom:nil,},EnvVar{Name:SHARE_PASSWORD_SECRET_KEY,Value:XDhGIWkGJg2nOEU3HCagHg==,ValueFrom:nil,},EnvVar{Name:SHARE_REQUEST_SECRET_KEY,Value:verysecret,ValueFrom:nil,},EnvVar{Name:SHARE_SCREEN_VERSIONS_URL,Value:https://service-url,ValueFrom:nil,},EnvVar{Name:SHARE_URL,Value:https://service-url,ValueFrom:nil,},EnvVar{Name:SLACK_CLIENT_ID,Value:verysecret,ValueFrom:nil,},EnvVar{Name:SLACK_CLIENT_SECRET,Value:verysecret,ValueFrom:nil,},EnvVar{Name:SNS_COMMENT_MADE_URL,Value:arn:aws:sns:us-east-1:aws-account:CommentMade,ValueFrom:nil,},EnvVar{Name:SNS_COMMENT_RESOLVED_URL,Value:arn:aws:sns:us-east-1:aws-account:CommentResolved,ValueFrom:nil,},EnvVar{Name:SS_BOX_CLIENT_ID,Value:,ValueFrom:nil,},EnvVar{Name:SS_BOX_CLIENT_SECRET,Value:,ValueFrom:nil,},EnvVar{Name:SS_BOX_REDIRECT_URL,Value:https://service-url,ValueFrom:nil,},EnvVar{Name:SS_DROPBOX_CLIENT_ID,Value:,ValueFrom:nil,},EnvVar{Name:SS_DROPBOX_CLIENT_SECRET,Value:,ValueFrom:nil,},EnvVar{Name:SS_DROPBOX_REDIRECT_URL,Value:https://service-url,ValueFrom:nil,},EnvVar{Name:SS_GOOGLE_DRIVE_CLIENT_ID,Value:,ValueFrom:nil,},EnvVar{Name:SS_GOOGLE_DRIVE_CLIENT_SECRET,Value:,ValueFrom:nil,},EnvVar{Name:SS_GOOGLE_DRIVE_REDIRECT_URL,Value:https://service-url,ValueFrom:nil,},RestartPolicy:Always,TerminationGracePeriodSeconds:*90,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{roleType: projectsapi,},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip-10-0-31-42.ec2.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,}

[MaciekPytel]

This looks like logging a pod object, but the log looks trimmed. Do you have a full log line (including file+line number or any part of original formatting string?).

[arukaen]

@MaciekPytel This is the full entry I get in Loggly. It looks like cluster-autoscaler dumps these to logs whenever the ec2-host is being terminated.

{
    "kubernetes": {
        "namespace_name": "kube-system",
        "pod_name": "cluster-autoscaler-deployment-5565f64c7f-p64k4",
        "container_name": "cluster-autoscaler",
        "namespace_labels": {
            "kube_environment": "testing",
            "kube_namespace": "kube-system"
        },
        "containers": {
            "cluster-autoscaler": {
                "image": "quay.io/image",
                "image_id": "docker-pullable://quay.io/image"
            }
        },
        "cluster_name": "testing-cluster",
        "labels": {
            "environment": "testing",
            "app.kubernetes.io/name": "cluster-autoscaler",
            "owner": "sres",
            "app": "cluster-autoscaler",
            "log_drain": "loggly",
            "name": "cluster-autoscaler-pod",
            "pod-template-hash": "5565f64c7f",
            "strategy": "rolling-update"
        },
        "host": "ip.ec2.internal",
        "pod_id": "0e2f876e-a7ad-4283-865f-5f9794bba127",
        "container_hash": "7b7679e4ad0904103cd950a126d1adb45d79c2d5b11a5cf020bb805574d521ed"
    },
    "log": "-----END RSA PRIVATE KEY-----,ValueFrom:nil,},EnvVar{Name:SAMPLE_BOARD_IDS,Value:,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS,Value:,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_HIGH_FIDELITY_PROTOTYPE,Value:5,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_INTERACTIVE_WIREFRAME,Value:4,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_MOBILE_HIGH_FIDELITY_PROTOTYPE,Value:6,ValueFrom:nil,},EnvVar{Name:SAMPLE_PROJECT_IDS_MOBILE_INTERACTIVE_WIREFRAME,Value:7,ValueFrom:nil,},EnvVar{Name:SEGMENT_WRITE_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:SELECTIVE_SYNC_URL,Value:http://selectsync-svc,ValueFrom:nil,},EnvVar{Name:SHARE_PASSWORD_SECRET_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:SHARE_REQUEST_SECRET_KEY,Value:NbbhW+dRQlCyAjfVw+q5YA==,ValueFrom:nil,},EnvVar{Name:SHARE_SCREEN_VERSIONS_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:SHARE_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:SLACK_CLIENT_ID,Value:secret,ValueFrom:nil,},EnvVar{Name:SLACK_CLIENT_SECRET,Value:secret,ValueFrom:nil,},EnvVar{Name:SNS_COMMENT_MADE_URL,Value:arn:aws:sns:us-east-1:accnt-id:CommentMade,ValueFrom:nil,},EnvVar{Name:SNS_COMMENT_RESOLVED_URL,Value:arn:aws:sns:us-east-1:accnt-id:CommentResolved,ValueFrom:nil,},EnvVar{Name:SS_BOX_CLIENT_ID,Value:,ValueFrom:nil,},EnvVar{Name:SS_BOX_CLIENT_SECRET,Value:,ValueFrom:nil,},EnvVar{Name:SS_BOX_REDIRECT_URL,Value:https://url,ValueFrom:nil,},EnvVar{Name:SS_DROPBOX_CLIENT_ID,Value:,ValueFrom:nil,},EnvVar{Name:SS_DROPBOX_CLIENT_SECRET,Value:,ValueFrom:nil,},EnvVar{Name:SS_DROPBOX_REDIRECT_URL,Value:https://url,ValueFrom:nil,},EnvVar{Name:SS_GOOGLE_DRIVE_CLIENT_ID,Value:,ValueFrom:nil,},EnvVar{Name:SS_GOOGLE_DRIVE_CLIENT_SECRET,Value:,ValueFrom:nil,},EnvVar{Name:SS_GOOGLE_DRIVE_REDIRECT_URL,Value:https://url,ValueFrom:nil,},EnvVar{Name:SSO_CRAFT_MANAGER_REDIRECT_URL,Value:craft://url,ValueFrom:nil,},EnvVar{Name:SSO_SYNC_PHOTOSHOP_REDIRECT_URL,Value:https://url,ValueFrom:nil,},EnvVar{Name:STATIC_CDN,Value:https://url/,ValueFrom:nil,},EnvVar{Name:STATSD_CONFIG_HOST,Value:127.0.0.1,ValueFrom:nil,},EnvVar{Name:STATSD_CONFIG_PORT,Value:8125,ValueFrom:nil,},EnvVar{Name:SUBDOMAIN_REGEX,Value:\\.[\\w\\W]*,ValueFrom:nil,},EnvVar{Name:SUBSCRIPTIONS_API_URL,Value:http://subscriptionsapi-svc,ValueFrom:nil,},EnvVar{Name:SYSTEM_CURL_ENABLED,Value:true,ValueFrom:nil,},EnvVar{Name:SYSTEM_ENVIRONMENT,Value:Unix,ValueFrom:nil,},EnvVar{Name:SYSTEM_IMAGE_MAGICK_COMPOSITE_PATH,Value:composite,ValueFrom:nil,},EnvVar{Name:SYSTEM_IMAGE_MAGICK_CONVERT_PATH,Value:convert,ValueFrom:nil,},EnvVar{Name:SYSTEM_RDIFF_PATH,Value:rdiff,ValueFrom:nil,},EnvVar{Name:SYSTEM_UNZIP_PATH,Value:unzip,ValueFrom:nil,},EnvVar{Name:TEAMS_API_URL,Value:http://teamsapi-svc,ValueFrom:nil,},EnvVar{Name:TESTING_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:THUMBNAIL_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:TWILIO_CREDENTIALS_ACCOUNT_SID,Value:secret,ValueFrom:nil,},EnvVar{Name:TWILIO_CREDENTIALS_AUTH_TOKEN,Value:secret,ValueFrom:nil,},EnvVar{Name:TWITTER_CREDS_OAUTH_ACCESS_TOKEN,Value:secret,ValueFrom:nil,},EnvVar{Name:TWITTER_CREDS_OAUTH_ACCESS_TOKEN_SECRET,Value:secret,ValueFrom:nil,},EnvVar{Name:TWITTER_CREDS_OAUTH_CONSUMER_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:TWITTER_CREDS_OAUTH_CONSUMER_SECRET,Value:secret,ValueFrom:nil,},EnvVar{Name:UPLOADER_URL,Value:http://upload-svc,ValueFrom:nil,},EnvVar{Name:V6_SIGNUP_OVERRIDE_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:V7_JWT_TOKEN,Value:secret,ValueFrom:nil,},EnvVar{Name:V7_LOGIN_API_URL,Value:http://url/,ValueFrom:nil,},EnvVar{Name:V7_LOGIN_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:V7_SIGNUP_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:WWW_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:ZENDESK_ROOT_URI,Value:https://url/,ValueFrom:nil,},EnvVar{Name:ZENDESK_SSO_ACCESS_URL,Value:https://url/,ValueFrom:nil,},EnvVar{Name:ZENDESK_SSO_SECRET_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:ZENDESK_TOKEN,Value:secret,ValueFrom:nil,},EnvVar{Name:ZENDESK_USER_EMAIL,Value:email,ValueFrom:nil,},EnvVar{Name:ZOHO_API_AUTH_TOKEN,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_CLIENT_ID,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_CLIENT_SECRET_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_COMMUNICATION_PROFILE_ID,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_HOSTED_PAGE_ID,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_PRODUCT_ID,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_PUSH_PASSWORD,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_PUSH_USERNAME,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_RATE_PLAN_CACHE_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:ZUORA_REST_API_URL,Value:https://url/,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},memory: {{5368709120 0} {<nil>} 5Gi BinarySI},},Requests:ResourceList{cpu: {{500 -3} {<nil>} 500m DecimalSI},memory: {{5368709120 0} {<nil>} 5Gi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-lf42l,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:url/,Port:{0 8500 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/url/,Port:{0 80 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:5,},Lifecycle:&Lifecycle{PostStart:nil,PreStop:&Handler{Exec:&ExecAction{Command:[/bin/sleep 60],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*90,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{roleType: api,},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip.ec2.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-quay-secret,},},Hostname:,Subdomain:,Affinity:&Affinity{NodeAffinity:nil,PodAffinity:nil,PodAntiAffinity:&PodAntiAffinity{RequiredDuringSchedulingIgnoredDuringExecution:[]PodAffinityTerm{},PreferredDuringSchedulingIgnoredDuringExecution:[]WeightedPodAffinityTerm{WeightedPodAffinityTerm{Weight:40,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{LabelSelectorRequirement{Key:name,Operator:In,Values:[api-pod],},},},Namespaces:[],TopologyKey:lifeCycle,},},WeightedPodAffinityTerm{Weight:60,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{LabelSelectorRequirement{Key:name,Operator:In,Values:[api-pod],},},},Namespaces:[],TopologyKey:failure-domain.beta.kubernetes.io/zone,},},},},},SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-03 20:58:30 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-03 20:59:41 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-03 20:59:41 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-03 20:58:30 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.0.31.42,PodIP:172.26.82.12,StartTime:2020-03-03 20:58:30 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:api-con,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-03 20:58:38 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:quay.io/image,ImageID:docker-pullable://quay.io/image,ContainerID:docker://b13e39ac0ff1c613de689537e53eb7afaaf6b02bff76eb2cd3389defb0e0898c,Started:*true,},ContainerStatus{Name:statsd-prometheus-exporter-con,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-03 20:58:31 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:quay.io/image/,ImageID:docker-pullable://quay.io/image,ContainerID:docker://5f5b93888c49965666ab5f405aaa8c9a6339cd9e70cbe0a6fbfab599bc90fbbd,Started:*true,},},QOSClass:Burstable,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:172.26.82.12,},},EphemeralContainerStatuses:[]ContainerStatus{},},}\n",
    "timestamp": "2020-03-03T21:18:34.234Z"
}

[MaciekPytel]

This is clearly a part of pod object being printed. It can't possibly be the whole line emitted by CA - the start of the log line is obviously a tail of one of environment variables. I guess something in your logging pipeline trims lines that are too long (it's a shame it keeps tail rather than head though).

I tried to grep for log statements that may be printing pod, but I didn't find anything. I suspect the pod object is initially added in some fmt.Errorf() or similar and later logged down the line. Without the initial part of the log (most likely containing hard-coded string that could be found in code) the only way I see of finding this would be to audit the codebase. Frankly, I don't expect this to happen too soon.

[arukaen]

@MaciekPytel :

cluster-autoscaler-deployment-754c777db6-fsh9r cluster-autoscaler I0330 15:42:43.157618       9 scale_down.go:836] Scale-down: removing node ip.ec2.internal, utilization: {0.3071052631578947 0.12494579194988317 0 cpu 0.3071052631578947}, pods to reschedule: multi/craftapi-deployment-76d48787d5-5bfcl
cluster-autoscaler-deployment-754c777db6-fsh9r cluster-autoscaler I0330 15:42:43.166832       9 delete.go:102] Successfully added ToBeDeletedTaint on node ip.ec2.internal
cluster-autoscaler-deployment-754c777db6-fsh9r cluster-autoscaler E0330 15:42:43.257353       9 scale_down.go:1151] Not deleted yet &Pod{ObjectMeta:{craftapi-deployment-76d48787d5-5bfcl craftapi-deployment-76d48787d5- multi /api/v1/namespaces/multi/pods/craftapi-deployment-76d48787d5-5bfcl d511342d-556a-4b97-93e4-bbac8793da90 2126865880 0 2020-03-30 15:30:42 +0000 UTC 2020-03-30 15:43:13 +0000 UTC 0xc015405430 map[app.kubernetes.io/name:craftapi environment:production log_drain:loggly name:craftapi-pod owner:gold pod-template-hash:76d48787d5 strategy:rolling-update tier:multi-tenant] map[kubectl.kubernetes.io/restartedAt:2020-03-07T00:16:50-08:00] [{apps/v1 ReplicaSet craftapi-deployment-76d48787d5 40fa7683-f151-44ab-a247-0aa7bbc11fd4 0xc015405507 0xc015405508}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-q76j4,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-q76j4,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:statsd-splitter-con,Image:quay.io/image,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:8125,Protocol:UDP,HostIP:,},},Env:[]EnvVar{EnvVar{Name:UPSTREAM_HOSTS,Value:127.0.0.1:8126,statsd-collector-svc:8125,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{cpu: {{200 -3} {<nil>} 200m DecimalSI},memory: {{50331648 0} {<nil>}  BinarySI},},Requests:ResourceList{cpu: {{50 -3} {<nil>} 50m DecimalSI},memory: {{25165824 0} {<nil>}  BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-q76j4,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},Container{Name:statsd-prometheus-exporter-con,Image:quay.io/image,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:8126,Protocol:UDP,HostIP:,},},Env:[]EnvVar{EnvVar{Name:CACHE_SIZE,Value:100000,ValueFrom:nil,},EnvVar{Name:CLUSTER,Value:use1-prod-1,ValueFrom:nil,},EnvVar{Name:ENVIRONMENT,Value:production,ValueFrom:nil,},EnvVar{Name:METRICS_TTL,Value:30m,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:multi,ValueFrom:nil,},EnvVar{Name:REGION,Value:us-east-1,ValueFrom:nil,},EnvVar{Name:RIN,Value:QDE80621,ValueFrom:nil,},EnvVar{Name:TENANCY,Value:shared,ValueFrom:nil,},EnvVar{Name:TIER,Value:multi-tenant,ValueFrom:nil,},EnvVar{Name:STATSD_PORT,Value::8126,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{cpu: {{200 -3} {<nil>} 200m DecimalSI},memory: {{268435456 0} {<nil>}  BinarySI},},Requests:ResourceList{cpu: {{50 -3} {<nil>} 50m DecimalSI},memory: {{134217728 0} {<nil>}  BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-q76j4,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},Container{Name:craftapi-con,Image:quay.io/image,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:80,Protocol:TCP,HostIP:,},},Env:[]EnvVar{EnvVar{Name:ACCOUNTS_URL,Value:http://accountsapi-svc,ValueFrom:nil,},EnvVar{Name:AUTH_API_JWT_SHARED_SECRET,Value:secret,ValueFrom:nil,},EnvVar{Name:AUTH_API_URL,Value:http://authapi-svc,ValueFrom:nil,},EnvVar{Name:AUTH_URL,Value:http://auth-svc,ValueFrom:nil,},EnvVar{Name:AWS_PERM_BUCKET,Value:s3-bucket,ValueFrom:nil,},EnvVar{Name:CF_API_URL,Value:http://cfprojectsapi-svc,ValueFrom:nil,},EnvVar{Name:CF_PROJECTS_URL,Value:http://cfprojects-svc,ValueFrom:nil,},EnvVar{Name:CRAFT_API_STATSD_HOST,Value:127.0.0.1,ValueFrom:nil,},EnvVar{Name:CRAFT_API_STATSD_PORT,Value:8125,ValueFrom:nil,},EnvVar{Name:DISABLE_RUNTIME_METRICS,Value:false,ValueFrom:nil,},EnvVar{Name:DISCARD_INSPECT_DATA,Value:true,ValueFrom:nil,},EnvVar{Name:DOMAIN,Value:example.com,ValueFrom:nil,},EnvVar{Name:GET_HOSTS_FROM,Value:dns,ValueFrom:nil,},EnvVar{Name:LAUNCH_DARKLY_API_KEY,Value:secret,ValueFrom:nil,},EnvVar{Name:LOG_LEVEL,Value:info,ValueFrom:nil,},EnvVar{Name:METADATA_NAME,Value:k8s-cluster,ValueFrom:nil,},EnvVar{Name:MIN_REQUIRED_CLIENT_VERSION,Value:2.0.0,ValueFrom:nil,},EnvVar{Name:PERMISSIONS_V2_URL,Value:http://permissionsapiv2-svc,ValueFrom:nil,},EnvVar{Name:PORT,Value:80,ValueFrom:nil,},EnvVar{Name:PROTOTYPES_URL,Value:http://prototypesapi-svc,ValueFrom:nil,},EnvVar{Name:PROXY_ASSETS_HOST,Value:craft-assets.example.com,ValueFrom:nil,},EnvVar{Name:PROXY_BETA_HOST,Value:craft-beta.example.com,ValueFrom:nil,},EnvVar{Name:PROXY_MANAGER_HOST,Value:craft-manager-api.example.com,ValueFrom:nil,},EnvVar{Name:PROXY_REQUEST_SOURCES,Value:craftmanager_app,craft_sync_photoshop,craft_data_photoshop,craft_data_sketch,ValueFrom:nil,},EnvVar{Name:ROLLBAR_TOKEN,Value:secret,ValueFrom:nil,},EnvVar{Name:RUNTIME_METRICS_INTERVAL_SECONDS,Value:5,ValueFrom:nil,},EnvVar{Name:SERVER_READ_TIMEOUT_SECONDS,Value:600,ValueFrom:nil,},EnvVar{Name:SERVER_WRITE_TIMEOUT_SECONDS,Value:10,ValueFrom:nil,},EnvVar{Name:SERVICE_NAME,Value:craft-api,ValueFrom:nil,},EnvVar{Name:THUMBNAIL_URL,Value:http://thumb-svc,ValueFrom:nil,},EnvVar{Name:UNSPLASH_API_KEY,Value:,ValueFrom:nil,},EnvVar{Name:UPLOADER_URL,Value:http://upload.example.com,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{cpu: {{500 -3} {<nil>} 500m DecimalSI},memory: {{4294967296 0} {<nil>} 4Gi BinarySI},},Requests:ResourceList{cpu: {{200 -3} {<nil>} 200m DecimalSI},memory: {{134217728 0} {<nil>}  BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-q76j4,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthcheck,Port:{0 80 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:3,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthcheck,Port:{0 80 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:30,TimeoutSeconds:2,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{roleType: default,},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip.ec2.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:docker-quay-secret,},},Hostname:,Subdomain:,Affinity:&Affinity{NodeAffinity:nil,PodAffinity:nil,PodAntiAffinity:&PodAntiAffinity{RequiredDuringSchedulingIgnoredDuringExecution:[]PodAffinityTerm{},PreferredDuringSchedulingIgnoredDuringExecution:[]WeightedPodAffinityTerm{WeightedPodAffinityTerm{Weight:40,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{LabelSelectorRequirement{Key:name,Operator:In,Values:[craftapi-pod],},},},Namespaces:[],TopologyKey:lifeCycle,},},WeightedPodAffinityTerm{Weight:60,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{LabelSelectorRequirement{Key:name,Operator:In,Values:[craftapi-pod],},},},Namespaces:[],TopologyKey:failure-domain.beta.kubernetes.io/zone,},},},},},SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-30 15:30:42 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-30 15:32:14 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-30 15:32:14 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-30 15:30:42 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.10.10.151,PodIP:172.16.246.135,StartTime:2020-03-30 15:30:42 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:craftapi-con,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-30 15:31:44 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:quay.io/iimages,ImageID:docker-pullable://quay.io/image,ContainerID:docker://d85dbd2879bdb176603ccc1fb51cfec7616a0a83747fe97a1b2ea1b45bdbd924,Started:*true,},ContainerStatus{Name:statsd-prometheus-exporter-con,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-30 15:31:42 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:quay.io/images,ImageID:docker-pullable://quay.io/invision/statsd-prometheus-exporter@sha256:ad3321a11c45c9ad66129f194da79b29ae1488a1627c3c2efa724ea1ce28b2b5,ContainerID:docker://071ad73e9cfb0bb5e2fff8fa48dab7c9f1eba530ff15c02b7741518dcbae79e0,Started:*true,},ContainerStatus{Name:statsd-splitter-con,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-30 15:31:39 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:quay.io/image,ImageID:docker-pullable://quay.io/image,ContainerID:docker://f6883ddc0872f3cdb536cc22d106340a9a6b600530b65b3cdeb504e4b79d1224,Started:*true,},},QOSClass:Burstable,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:172.16.246.135,},},EphemeralContainerStatuses:[]ContainerStatus{},},}

[MaciekPytel]

That helps a lot, thanks. Turns out I wasn't able to find it with my grepping, because it was already fixed in #2476. Unfortunately this fix is currently only available in 1.17+.

[arukaen]

@MaciekPytel Any chance that this can be back ported to a 1.16 release? As mentioned earlier, my clusters are running k8s 1.16.3

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.