K8s Prow is limited to 1k concurrent pods, or podgc will fight with sinker #11594
Assignees
Labels
kind/bug
Categorizes issue or PR as related to a bug.
priority/important-soon
Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
sig/scalability
Categorizes an issue or PR as relevant to SIG Scalability.
sig/testing
Categorizes an issue or PR as relevant to SIG Testing.
Comments
|
/priority critical-urgent |
k8s-ci-robot
added
the
priority/critical-urgent
|
/sig testing |
k8s-ci-robot
added
the
sig/testing
could you list them? Strangely https://prow.k8s.io/?job=ci-kubernetes-e2e-gke-large-performance-regional has one entry at https://prow.k8s.io/rerun?prowjob=931c4290-3d8a-11e9-9c9a-0a580a6c0e78 apiVersion: prow.k8s.io/v1
kind: ProwJob
metadata:
annotations:
prow.k8s.io/job: ci-kubernetes-e2e-gke-large-performance-regional
creationTimestamp: null
labels:
created-by-prow: "true"
preset-k8s-ssh: "true"
preset-service-account: "true"
prow.k8s.io/id: 931c4290-3d8a-11e9-9c9a-0a580a6c0e78
prow.k8s.io/job: ci-kubernetes-e2e-gke-large-performance-regional
prow.k8s.io/type: periodic
name: e89951f0-3e98-11e9-b844-0a580a6c0923
spec:
agent: kubernetes
cluster: default
job: ci-kubernetes-e2e-gke-large-performance-regional
namespace: test-pods
pod_spec:
containers:
- args:
- --timeout=600
- --repo=k8s.io/kubernetes=master
- --repo=k8s.io/perf-tests=master
- --root=/go/src
- --scenario=kubernetes_e2e
- --
- --cluster=gke-regional-cluster
- --deployment=gke
- --extract=ci/latest-1.13
- --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging
- --gcp-node-image=gci
- --gcp-project=kubernetes-scale
- --gcp-region=us-east1
- --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork
name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19
- --gke-environment=staging
- --gke-node-locations=us-east1-b
- --gke-shape={"default":{"Nodes":1999,"MachineType":"n1-standard-1"},"heapster-pool":{"Nodes":1,"MachineType":"n1-standard-8"}}
- --provider=gke
- --test=false
- --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh
- --test-cmd-args=cluster-loader2
- --test-cmd-args=--nodes=2000
- --test-cmd-args=--provider=gke
- --test-cmd-args=--report-dir=/workspace/_artifacts
- --test-cmd-args=--testconfig=testing/density/config.yaml
- --test-cmd-args=--testconfig=testing/load/config.yaml
- --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml
- --test-cmd-name=ClusterLoaderV2
- --timeout=570m
- --use-logexporter
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/service-account/service-account.json
- name: E2E_GOOGLE_APPLICATION_CREDENTIALS
value: /etc/service-account/service-account.json
- name: USER
value: prow
- name: JENKINS_GCE_SSH_PRIVATE_KEY_FILE
value: /etc/ssh-key-secret/ssh-private
- name: JENKINS_GCE_SSH_PUBLIC_KEY_FILE
value: /etc/ssh-key-secret/ssh-public
image: gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master
name: ""
resources:
requests:
cpu: "6"
memory: 16Gi
volumeMounts:
- mountPath: /etc/service-account
name: service
readOnly: true
- mountPath: /etc/ssh-key-secret
name: ssh
readOnly: true
volumes:
- name: service
secret:
secretName: service-account
- name: ssh
secret:
defaultMode: 256
secretName: ssh-key-secret
type: periodic
status:
startTime: "2019-03-04T16:16:46Z"
state: triggeredhttps://testgrid.k8s.io/sig-scalability-gke#gke-large-performance-regional |
|
@BenTheElder I recently added more logging to |
|
All of the entries in testgrid are showing the same pod ID on deck, https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/161 |
|
the logs for horologium don't seem to show much so far (stackdriver export for horologium pod, [
{
"insertId": "1ulicp5fepvsxu",
"jsonPayload": {
"level": "info",
"should-trigger": true,
"name": "931c4290-3d8a-11e9-9c9a-0a580a6c0e78",
"component": "horologium",
"type": "periodic",
"job": "ci-kubernetes-e2e-gke-large-performance-regional",
"msg": "Triggering new run of cron periodic.",
"previous-found": true
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "horologium-78fb7b98f8-tn8sc",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "horologium",
"namespace_id": "default",
"instance_id": "5347705516640603225"
}
},
"timestamp": "2019-03-03T08:01:39Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "horologium-78fb7b98f8-tn8sc",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-blkqd"
},
"logName": "projects/k8s-prow/logs/horologium",
"receiveTimestamp": "2019-03-03T08:01:45.470497268Z"
},
{
"insertId": "15wtu9nfepwm0g",
"jsonPayload": {
"component": "horologium",
"msg": "Triggering cron job ci-kubernetes-e2e-gke-large-performance-regional.",
"client": "cron",
"level": "info"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "horologium-78fb7b98f8-tn8sc",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "horologium",
"namespace_id": "default",
"instance_id": "5347705516640603225"
}
},
"timestamp": "2019-03-03T08:01:00Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "horologium-78fb7b98f8-tn8sc",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-blkqd"
},
"logName": "projects/k8s-prow/logs/horologium",
"receiveTimestamp": "2019-03-03T08:01:06.402048467Z"
}
] |
|
plank for [
{
"insertId": "a85i6flj274m",
"jsonPayload": {
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189614343 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/165/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 165 map[github-reporter:pending]}})",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-04T13:08:23Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T13:08:28.783717431Z"
},
{
"insertId": "a85i6flj274l",
"jsonPayload": {
"client": "kube",
"level": "debug",
"component": "plank",
"msg": "GetProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78)"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-04T13:08:23Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T13:08:28.783717431Z"
},
{
"insertId": "a85i6flj2748",
"jsonPayload": {
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189487735 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/165/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 165 map[github-reporter:pending]}})",
"client": "kube",
"level": "debug",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921",
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow"
}
},
"timestamp": "2019-03-04T13:08:23Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T13:08:28.783717431Z"
},
{
"insertId": "a85i6flj2747",
"jsonPayload": {
"job": "ci-kubernetes-e2e-gke-large-performance-regional",
"msg": "Pod is missing, starting a new pod",
"level": "info",
"name": "931c4290-3d8a-11e9-9c9a-0a580a6c0e78",
"type": "periodic",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-04T13:08:23Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T13:08:28.783717431Z"
},
{
"insertId": "a85i6flj2746",
"jsonPayload": {
"component": "plank",
"msg": "CreatePod({{ } {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 created-by-prow:true prow.k8s.io/type:periodic prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional preset-k8s-ssh:true preset-service-account:true] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {[{service {nil nil nil nil nil &SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}] [] [{test gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil} {BUILD_ID 165 nil} {BUILD_NUMBER 165 nil} {JOB_NAME ci-kubernetes-e2e-gke-large-performance-regional nil} {JOB_SPEC {\"type\":\"periodic\",\"job\":\"ci-kubernetes-e2e-gke-large-performance-regional\",\"buildid\":\"165\",\"prowjobid\":\"931c4290-3d8a-11e9-9c9a-0a580a6c0e78\"} nil} {JOB_TYPE periodic nil} {PROW_JOB_ID 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}] Never <nil> <nil> map[] 0xc02e0680b8 false false false <nil> nil [] nil [] [] <nil> nil [] <nil>} { [] <nil> [] [] }})",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-04T13:08:23Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T13:08:28.783717431Z"
},
{
"insertId": "1vzsmovfk87wnw",
"jsonPayload": {
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189487735 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/164/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 164 map[github-reporter:pending]}})",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow"
}
},
"timestamp": "2019-03-04T05:43:23Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T05:43:28.556197331Z"
},
{
"insertId": "1vzsmovfk87wnv",
"jsonPayload": {
"component": "plank",
"msg": "GetProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78)",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow"
}
},
"timestamp": "2019-03-04T05:43:23Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T05:43:28.556197331Z"
},
{
"insertId": "1vzsmovfk87wnl",
"jsonPayload": {
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189363471 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/164/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 164 map[github-reporter:pending]}})",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921",
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow"
}
},
"timestamp": "2019-03-04T05:43:23Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T05:43:28.556197331Z"
},
{
"insertId": "1vzsmovfk87wnk",
"jsonPayload": {
"msg": "Pod is missing, starting a new pod",
"job": "ci-kubernetes-e2e-gke-large-performance-regional",
"level": "info",
"name": "931c4290-3d8a-11e9-9c9a-0a580a6c0e78",
"type": "periodic",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-04T05:43:23Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T05:43:28.556197331Z"
},
{
"insertId": "1vzsmovfk87wnj",
"jsonPayload": {
"msg": "CreatePod({{ } {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 created-by-prow:true prow.k8s.io/type:periodic prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {[{service {nil nil nil nil nil &SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}] [] [{test gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil} {BUILD_ID 164 nil} {BUILD_NUMBER 164 nil} {JOB_NAME ci-kubernetes-e2e-gke-large-performance-regional nil} {JOB_SPEC {\"type\":\"periodic\",\"job\":\"ci-kubernetes-e2e-gke-large-performance-regional\",\"buildid\":\"164\",\"prowjobid\":\"931c4290-3d8a-11e9-9c9a-0a580a6c0e78\"} nil} {JOB_TYPE periodic nil} {PROW_JOB_ID 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}] Never <nil> <nil> map[] 0xc010a48048 false false false <nil> nil [] nil [] [] <nil> nil [] <nil>} { [] <nil> [] [] }})",
"client": "kube",
"level": "debug",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-04T05:43:23Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-04T05:43:28.556197331Z"
},
{
"insertId": "10cniqefl3qiev",
"jsonPayload": {
"client": "kube",
"level": "debug",
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189363471 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic created-by-prow:true preset-k8s-ssh:true preset-service-account:true] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/163/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 163 map[github-reporter:pending]}})"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T22:26:54Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T22:27:01.550080143Z"
},
{
"insertId": "10cniqefl3qieu",
"jsonPayload": {
"client": "kube",
"level": "debug",
"component": "plank",
"msg": "GetProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78)"
},
"resource": {
"type": "container",
"labels": {
"namespace_id": "default",
"instance_id": "7000980459144515921",
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank"
}
},
"timestamp": "2019-03-03T22:26:54Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T22:27:01.550080143Z"
},
{
"insertId": "10cniqefl3qiee",
"jsonPayload": {
"client": "kube",
"level": "debug",
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189240028 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[prow.k8s.io/type:periodic created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[memory:{{17179869184 0} {<nil>} BinarySI} cpu:{{6 0} {<nil>} 6 DecimalSI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/163/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 163 map[github-reporter:pending]}})"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T22:26:54Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T22:27:01.550080143Z"
},
{
"insertId": "10cniqefl3qied",
"jsonPayload": {
"msg": "Pod is missing, starting a new pod",
"job": "ci-kubernetes-e2e-gke-large-performance-regional",
"level": "info",
"name": "931c4290-3d8a-11e9-9c9a-0a580a6c0e78",
"type": "periodic",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T22:26:54Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T22:27:01.550080143Z"
},
{
"insertId": "10cniqefl3qiec",
"jsonPayload": {
"msg": "CreatePod({{ } {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[created-by-prow:true prow.k8s.io/type:periodic prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {[{service {nil nil nil nil nil &SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}] [] [{test gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil} {BUILD_ID 163 nil} {BUILD_NUMBER 163 nil} {JOB_NAME ci-kubernetes-e2e-gke-large-performance-regional nil} {JOB_SPEC {\"type\":\"periodic\",\"job\":\"ci-kubernetes-e2e-gke-large-performance-regional\",\"buildid\":\"163\",\"prowjobid\":\"931c4290-3d8a-11e9-9c9a-0a580a6c0e78\"} nil} {JOB_TYPE periodic nil} {PROW_JOB_ID 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}] Never <nil> <nil> map[] 0xc012316018 false false false <nil> nil [] nil [] [] <nil> nil [] <nil>} { [] <nil> [] [] }})",
"client": "kube",
"level": "debug",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921",
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow"
}
},
"timestamp": "2019-03-03T22:26:54Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T22:27:01.550080143Z"
},
{
"insertId": "1o7ph7cfh465cc",
"jsonPayload": {
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189240028 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[prow.k8s.io/type:periodic created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/162/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 162 map[github-reporter:pending]}})",
"client": "kube",
"level": "debug",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T15:15:55Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T15:15:59.459168535Z"
},
{
"insertId": "1o7ph7cfh465cb",
"jsonPayload": {
"client": "kube",
"level": "debug",
"component": "plank",
"msg": "GetProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78)"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T15:15:55Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T15:15:59.459168535Z"
},
{
"insertId": "1o7ph7cfh4659z",
"jsonPayload": {
"client": "kube",
"level": "debug",
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189117035 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/162/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 162 map[github-reporter:pending]}})"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T15:15:55Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T15:15:59.459168535Z"
},
{
"insertId": "1o7ph7cfh4659y",
"jsonPayload": {
"job": "ci-kubernetes-e2e-gke-large-performance-regional",
"msg": "Pod is missing, starting a new pod",
"level": "info",
"name": "931c4290-3d8a-11e9-9c9a-0a580a6c0e78",
"component": "plank",
"type": "periodic"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T15:15:55Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T15:15:59.459168535Z"
},
{
"insertId": "1o7ph7cfh4659x",
"jsonPayload": {
"msg": "CreatePod({{ } {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 preset-k8s-ssh:true created-by-prow:true prow.k8s.io/type:periodic prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {[{service {nil nil nil nil nil &SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}] [] [{test gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil} {BUILD_ID 162 nil} {BUILD_NUMBER 162 nil} {JOB_NAME ci-kubernetes-e2e-gke-large-performance-regional nil} {JOB_SPEC {\"type\":\"periodic\",\"job\":\"ci-kubernetes-e2e-gke-large-performance-regional\",\"buildid\":\"162\",\"prowjobid\":\"931c4290-3d8a-11e9-9c9a-0a580a6c0e78\"} nil} {JOB_TYPE periodic nil} {PROW_JOB_ID 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}] Never <nil> <nil> map[] 0xc0121e6294 false false false <nil> nil [] nil [] [] <nil> nil [] <nil>} { [] <nil> [] [] }})",
"client": "kube",
"level": "debug",
"component": "plank"
},
"resource": {
"type": "container",
"labels": {
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow"
}
},
"timestamp": "2019-03-03T15:15:55Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T15:15:59.459168535Z"
},
{
"insertId": "1vpritkfhzolmx",
"jsonPayload": {
"client": "kube",
"level": "debug",
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189117014 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/161/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 161 map[github-reporter:pending]}})"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T08:01:56Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T08:02:01.492391989Z"
},
{
"insertId": "1vpritkfhzolmw",
"jsonPayload": {
"component": "plank",
"msg": "GetProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78)",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T08:01:56Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T08:02:01.492391989Z"
},
{
"insertId": "1vpritkfhzollp",
"jsonPayload": {
"component": "plank",
"msg": "ReplaceProwJob(931c4290-3d8a-11e9-9c9a-0a580a6c0e78, {{ProwJob prow.k8s.io/v1} {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 default /apis/prow.k8s.io/v1/namespaces/default/prowjobs/931c4290-3d8a-11e9-9c9a-0a580a6c0e78 933a4706-3d8a-11e9-898b-42010a80003a 189116931 1 2019-03-03 08:01:39 +0000 UTC <nil> <nil> map[prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 created-by-prow:true preset-k8s-ssh:true preset-service-account:true prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {periodic kubernetes default test-pods ci-kubernetes-e2e-gke-large-performance-regional <nil> [] false 0 false &PodSpec{Volumes:[{service {nil nil nil nil nil SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,} <nil> <nil>} {2019-03-03 08:01:39 +0000 UTC <nil> pending Job triggered. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/161/ 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 161 map[]}})",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"namespace_id": "default",
"instance_id": "7000980459144515921",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank"
}
},
"timestamp": "2019-03-03T08:01:55Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T08:02:01.492391989Z"
},
{
"insertId": "1vpritkfhzollo",
"jsonPayload": {
"msg": "Transitioning states.",
"job": "ci-kubernetes-e2e-gke-large-performance-regional",
"to": "pending",
"level": "info",
"name": "931c4290-3d8a-11e9-9c9a-0a580a6c0e78",
"component": "plank",
"type": "periodic",
"from": "triggered"
},
"resource": {
"type": "container",
"labels": {
"pod_id": "plank-9f6cb7fbb-4jdf2",
"zone": "us-central1-f",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T08:01:55Z",
"severity": "ERROR",
"labels": {
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default",
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T08:02:01.492391989Z"
},
{
"insertId": "1vpritkfhzolln",
"jsonPayload": {
"component": "plank",
"msg": "CreatePod({{ } {931c4290-3d8a-11e9-9c9a-0a580a6c0e78 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional preset-k8s-ssh:true preset-service-account:true prow.k8s.io/id:931c4290-3d8a-11e9-9c9a-0a580a6c0e78 created-by-prow:true prow.k8s.io/type:periodic] map[prow.k8s.io/job:ci-kubernetes-e2e-gke-large-performance-regional] [] nil [] } {[{service {nil nil nil nil nil &SecretVolumeSource{SecretName:service-account,Items:[],DefaultMode:nil,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {ssh {nil nil nil nil nil &SecretVolumeSource{SecretName:ssh-key-secret,Items:[],DefaultMode:*256,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}] [] [{test gcr.io/k8s-testimages/kubekins-e2e:v20190301-76bc03340-master [] [--timeout=600 --repo=k8s.io/kubernetes=master --repo=k8s.io/perf-tests=master --root=/go/src --scenario=kubernetes_e2e -- --cluster=gke-regional-cluster --deployment=gke --extract=ci/latest-1.13 --gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging --gcp-node-image=gci --gcp-project=kubernetes-scale --gcp-region=us-east1 --gke-create-command=container clusters create --quiet --enable-ip-alias --create-subnetwork name=ip-alias-subnet-regional --cluster-ipv4-cidr=/12 --services-ipv4-cidr=/19 --gke-environment=staging --gke-node-locations=us-east1-b --gke-shape={\"default\":{\"Nodes\":1999,\"MachineType\":\"n1-standard-1\"},\"heapster-pool\":{\"Nodes\":1,\"MachineType\":\"n1-standard-8\"}} --provider=gke --test=false --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh --test-cmd-args=cluster-loader2 --test-cmd-args=--nodes=2000 --test-cmd-args=--provider=gke --test-cmd-args=--report-dir=/workspace/_artifacts --test-cmd-args=--testconfig=testing/density/config.yaml --test-cmd-args=--testconfig=testing/load/config.yaml --test-cmd-args=--testoverrides=./testing/density/2000_nodes/override.yaml --test-cmd-name=ClusterLoaderV2 --timeout=570m --use-logexporter] [] [] [{GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {E2E_GOOGLE_APPLICATION_CREDENTIALS /etc/service-account/service-account.json nil} {USER prow nil} {JENKINS_GCE_SSH_PRIVATE_KEY_FILE /etc/ssh-key-secret/ssh-private nil} {JENKINS_GCE_SSH_PUBLIC_KEY_FILE /etc/ssh-key-secret/ssh-public nil} {BUILD_ID 161 nil} {BUILD_NUMBER 161 nil} {JOB_NAME ci-kubernetes-e2e-gke-large-performance-regional nil} {JOB_SPEC {\"type\":\"periodic\",\"job\":\"ci-kubernetes-e2e-gke-large-performance-regional\",\"buildid\":\"161\",\"prowjobid\":\"931c4290-3d8a-11e9-9c9a-0a580a6c0e78\"} nil} {JOB_TYPE periodic nil} {PROW_JOB_ID 931c4290-3d8a-11e9-9c9a-0a580a6c0e78 nil}] {map[] map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{17179869184 0} {<nil>} BinarySI}]} [{service true /etc/service-account <nil>} {ssh true /etc/ssh-key-secret <nil>}] [] nil nil nil nil false false false}] Never <nil> <nil> map[] 0xc006506a04 false false false <nil> nil [] nil [] [] <nil> nil [] <nil>} { [] <nil> [] [] }})",
"client": "kube",
"level": "debug"
},
"resource": {
"type": "container",
"labels": {
"zone": "us-central1-f",
"pod_id": "plank-9f6cb7fbb-4jdf2",
"project_id": "k8s-prow",
"cluster_name": "prow",
"container_name": "plank",
"namespace_id": "default",
"instance_id": "7000980459144515921"
}
},
"timestamp": "2019-03-03T08:01:55Z",
"severity": "ERROR",
"labels": {
"compute.googleapis.com/resource_name": "fluentd-gcp-v3.2.0-56c8g",
"container.googleapis.com/pod_name": "plank-9f6cb7fbb-4jdf2",
"container.googleapis.com/stream": "stderr",
"container.googleapis.com/namespace_name": "default"
},
"logName": "projects/k8s-prow/logs/plank",
"receiveTimestamp": "2019-03-03T08:02:01.492391989Z"
}
] |
|
Wait, so were there actually multiple versions of the test running at once? |
Thanks, @stevekuznetsov. Will keep that in mind for the future.
@BenTheElder, other examples name: ci-kubernetes-e2e-gce-scale-performance Job should be run once Mon-Fri, but recently there are days when it's run twice or thrice: name: ci-kubernetes-e2e-gke-large-performance Job is supposed to be run once every Sunday, but last yesterday was launched twice: There are probably a few more. Were you able to figure out what is going on? |
|
/cc |
We probably hit the OOMKilled again? |
@stevekuznetsov seems the pod finished and exited properly? Seems a bug in plank? |
|
Horologium triggered the job properly afaik also - no associated logs in sinker (so - who deleted the pod?) |
|
@krzyzacy what was the behavior? Plank will create a Pod if one does not exist and the ProwJob is not marked in some completed state, can you try to determine via logs how the pod exited and what the state of the prowjob was at the time? |
|
The pod exited with 1 (with I think the problem occurs after #11477? (Feb.26 according to @mm4tt 's screenshot) And I think the prowjob was still in pending state as I don't see any other state transition log: |
|
Edit: Just paste the full node log here: |
|
If the pod exited, are you seeing |
|
I don't see the clean up from sinker. How do I check the audit log? |
|
@wojtek-t by "separate build cluster" we mean a cluster that we schedule all scalaibility job pods there. They will still share the same prow controllers, but you will have control over your secrets/pods/nodes. (aka k8s prow can schedule pods onto your cluster) |
|
So I think I'm not following why this is supposed to help. If we would be sharing the same prow controllers, then the limits that we are affected by won't change, right [we're being hit by the concurrency limit at the prow level, right?] |
|
@wojtek-t nope - the podgc kicks in in the build cluster where all the workloads are running, not in the service cluster. |
|
Wait - I'm lost then. You're talking about Kubernetes podgc? This should never remove pods in state Running, which is happening in this case. |
|
@wojtek-t Prow has a logic that watches completed pod, and update prowjob status as passed/failed/etc. Right when a pod is finished, it's got deleted by K8s podgc, and plank does not using informer cache yet so it will miss the event, saw pod associated with the prowjob is missing, and restart the pod. (aka we tries to manage pod lifecycle by prow, not K8s podgc, and now they are fighting.) |
|
@krzyzacy - I'm not entirely sure it's our problem then, because unless I'm not missing anything we've seen cases where the test was restarted in the middle of the run, not after it was finished... But anyway - I'm fine with using a separate cluster, because it will mitigate at least some of the issues... |
|
@wojtek-t I think it shows up as 'R' in testgrid because the log failed to be uploaded, as testgrid is basically reading gcs. I can also take a closer look. |
This I know (and yes that happens). But we've seen cases that it is restarted in the meantime (though it's hard to tell why it was restarted and what happened in that first round) and the restarted attempt finishes on time and is reported correctly. Though I don't know if it was happening recently too.. |
|
@wojtek-t Or we might be hitting disk pressure again, I think we never solved that with the logexporter. Use dedicated nodes to run scalability jobs and give them more resources should help in general. I know you want to own your cluster, maybe hand us a cluster credential and we can help with carrying over the existing secrets? |
|
This has hit us again. I've set up us a dedicated cluster in which we'd like to run scalability jobs. @krzyzacy I'll follow up with you to pass the credentials. |
|
After switching to a dedicated build cluster scalability tests still suffers from the same problem (or we see the same symptoms). |
|
FYI: I've changed machines in the dedicated clusters to 16 cores, as I've seen there was ~100% CPU usage. We need to wait yet another day to check whether it helped. |
|
this smells like a different pattern, the job suppose to run daily? |
|
Since you guys have your own cluster now please use your credentials to investigate the test pods and pod events to see what is happening. |
|
this smells the pod was evicted The last few line of logs are like which indicates log-exporter was not working properly? And kubelet log: |
|
@oxddr 223620144Ki -> 229 GB, you're using more storage than the node can fit on disk ... sounds like we're pulling down logs to the job again, instead of up to GCS. That's just not going to work well. You could work around it by making nodes with very large disks and adding proper resource specs to the pods |
|
Thank you all for help and tips. Hopefully next time, I'll be able to debug it myself. I've increase disk size with the 16 core nodes added earlier today, but I am going to bump it even further to make sure our tests passes. I'll have a look tomorrow into why we are using so much disk space. |
|
It's strange that we've started using more storage, given that the only change to our job config was adding Maybe it's some kind of a permission issue, where due to misconfigured/missing secrets we cannot write to GCS and fallback to dumping the logs locally? |
|
This is clearly a separate issue. I've opened #12940, so this can be tracked there. |
|
(seems not happening anymore? feel free to reopen) |
|
Yes, since we switched to our own, dedicated build cluster it stopped happening. There are some "cleanups" we need to do to finalize this, but they're tracked in separate issues. We can close this one. Thanks, Sen! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Our scalability tests started behaving strangely over the weekend. The prow jobs running tests are scheduled when they shouldn't.
Example:
name: ci-kubernetes-e2e-gke-large-performance-regional
Config:
Test should be run once a week, but it has been scheduled 4 times over last weekend.
There are other jobs behaving similarly to this one, i.e. they are scheduled and run when they shouldn't be run.
This is wreaking havoc in our scalability tests.
Due to quota issues, the tests share the same gcp projects. Now, because they're run when shouldn't, they started interfering with each other causing multiple tests to fail.
The text was updated successfully, but these errors were encountered: