GCE cluster provision failure #14575

bparees · 2017-06-12T01:17:11Z

Failure summary:

  1. Host:     ci-prtest-5a37c28-2997-ig-n-fndk
     Play:     schedulable_nodes
     Task:     openshift-volume-quota : Create filesystem for /var/lib/origin/openshift.local.volumes
     Message:  Device /dev/sdc not found.

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/2997/consoleFull

The text was updated successfully, but these errors were encountered:

stevekuznetsov · 2017-06-12T14:01:02Z

@smarterclayton do we have a SOP for these?

stevekuznetsov · 2017-06-12T14:01:12Z

As in, are we creating tickets somewhere for the GCE API?

smarterclayton · 2017-06-12T14:35:45Z

We would create tickets under our account, yes. And you need to have enough info that they can reproduce. If you can't reproduce, they basically don't help much.

stevekuznetsov · 2017-06-16T14:54:05Z

Can't really prioritize this one

pweil- · 2017-06-19T09:39:36Z

Can't really prioritize this one

Adding p2 then since no priority will show up on the bug list to make sure it's triaged. Feel free to lower it further

0xmichalis · 2017-07-05T13:06:17Z

https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/1217/consoleFull#-191042845458b6e51eb7608a5981914356

smarterclayton · 2017-07-05T13:44:32Z

There's clearly something wrong with node setup in the installer, this is p0 until we know why. It looks similar to other racy startup failures, and it happens after networking is configured.

derekwaynecarr · 2017-07-05T19:05:58Z

@smarterclayton we have a bz that may or may not be similar but the cluster recovers:
https://bugzilla.redhat.com/show_bug.cgi?id=1466732

smarterclayton · 2017-07-05T20:46:20Z

Yeah, suspected something similar. I have not been able to manually reproduce it.

…

On Wed, Jul 5, 2017 at 3:06 PM, Derek Carr ***@***.***> wrote: @smarterclayton <https://github.com/smarterclayton> we have a bz that may or may not be similar but the cluster recovers: https://bugzilla.redhat.com/show_bug.cgi?id=1466732 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p_Dzh5pDmPBoYUSnDf-1feO-sPwiks5sK96YgaJpZM4N2kgI> .

0xmichalis · 2017-07-11T20:27:41Z

Another gce provisioning flake in the installer: https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/1262/consoleFull#68717269158b6e51eb7608a5981914356

TASK [dynamic-inventory : Templatize environment script] ***********************
Tuesday 11 July 2017  17:12:20 +0000 (0:00:01.010)       0:03:39.415 ********** 
ok: [localhost]

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
Tuesday 11 July 2017  17:12:22 +0000 (0:00:01.467)       0:03:40.882 ********** 
ok: [localhost]
ERROR! Attempted to execute "/usr/share/ansible/openshift-ansible-gce/inventory.sh" as inventory script: Inventory script (/usr/share/ansible/openshift-ansible-gce/inventory.sh) had an execution error: Traceback (most recent call last):
  File "/usr/share/ansible/openshift-ansible-gce/inventory/gce/hosts/gce.py", line 400, in <module>
    GceInventory()
  File "/usr/share/ansible/openshift-ansible-gce/inventory/gce/hosts/gce.py", line 132, in __init__
    print(self.json_format_dict(self.group_instances(zones),
  File "/usr/share/ansible/openshift-ansible-gce/inventory/gce/hosts/gce.py", line 320, in group_instances
    nodes = self.driver.list_nodes()
  File "/usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py", line 2419, in list_nodes
    use_disk_cache=ex_use_disk_cache)
  File "/usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py", line 8366, in _to_node
    bd['name'], bd['zone'], use_cache=use_disk_cache)
  File "/usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py", line 7109, in ex_get_volume
    return self._ex_lookup_volume(name, zone)
  File "/usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py", line 7405, in _ex_lookup_volume
    self._ex_populate_dict()
AttributeError: 'GCENodeDriver' object has no attribute '_ex_populate_dict'

smarterclayton · 2017-07-11T21:43:55Z

Ugh, @sdodson know else knows the gce py

bparees · 2017-07-12T17:13:03Z

another:

TASK [provision : Provision GCE resources] *************************************
Wednesday 12 July 2017  15:46:04 +0000 (0:00:00.111)       0:00:02.965 ******** 
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/tmp/provision.sh"], "delta": "0:00:41.080842", "end": "2017-07-12 15:46:45.500767", "failed": true, "rc": 1, "start": "2017-07-12 15:46:04.419925", "stderr": "ERROR: (gcloud.compute.networks.create) Could not fetch resource:\n - Quota 'ROUTES' exceeded.  Limit: 300.0\n - Quota 'SUBNETWORKS' exceeded.  Limit: 275.0", "stderr_lines": ["ERROR: (gcloud.compute.networks.create) Could not fetch resource:", " - Quota 'ROUTES' exceeded.  Limit: 300.0", " - Quota 'SUBNETWORKS' exceeded.  Limit: 275.0"], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=1   

Wednesday 12 July 2017  15:46:45 +0000 (0:00:41.202)       0:00:44.167 ******** 
=============================================================================== 
provision : Provision GCE resources ------------------------------------ 41.20s
provision : Provision GCE DNS domain ------------------------------------ 1.64s
provision : Templatize DNS script --------------------------------------- 0.69s
provision : Templatize provision script --------------------------------- 0.41s
provision : Ensure that DNS resolves to the hosted zone ----------------- 0.11s

Failure summary:

  1. Host:     localhost
     Play:     Ensure all cloud resources necessary for the cluster, including instances, have been started
     Task:     provision : Provision GCE resources
     Message:  ???

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/4241/console

bparees · 2017-07-12T17:13:22Z

(that one appears to be a quota exceeded issue)

smarterclayton · 2017-07-12T17:16:08Z

Fuck

…

On Wed, Jul 12, 2017 at 1:13 PM, Ben Parees ***@***.***> wrote: another: TASK [provision : Provision GCE resources] ************************************* Wednesday 12 July 2017 15:46:04 +0000 (0:00:00.111) 0:00:02.965 ******** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/tmp/provision.sh"], "delta": "0:00:41.080842", "end": "2017-07-12 15:46:45.500767", "failed": true, "rc": 1, "start": "2017-07-12 15:46:04.419925", "stderr": "ERROR: (gcloud.compute.networks.create) Could not fetch resource:\n - Quota 'ROUTES' exceeded. Limit: 300.0\n - Quota 'SUBNETWORKS' exceeded. Limit: 275.0", "stderr_lines": ["ERROR: (gcloud.compute.networks.create) Could not fetch resource:", " - Quota 'ROUTES' exceeded. Limit: 300.0", " - Quota 'SUBNETWORKS' exceeded. Limit: 275.0"], "stdout": "", "stdout_lines": []} PLAY RECAP ********************************************************************* localhost : ok=4 changed=3 unreachable=0 failed=1 Wednesday 12 July 2017 15:46:45 +0000 (0:00:41.202) 0:00:44.167 ******** =============================================================================== provision : Provision GCE resources ------------------------------------ 41.20s provision : Provision GCE DNS domain ------------------------------------ 1.64s provision : Templatize DNS script --------------------------------------- 0.69s provision : Templatize provision script --------------------------------- 0.41s provision : Ensure that DNS resolves to the hosted zone ----------------- 0.11s Failure summary: 1. Host: localhost Play: Ensure all cloud resources necessary for the cluster, including instances, have been started Task: provision : Provision GCE resources Message: ??? https://ci.openshift.redhat.com/jenkins/job/test_pull_ request_origin_extended_conformance_gce/4241/console — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p1YKU6w8Dvk9aiCWXtCtsJ5gSgCjks5sNP6jgaJpZM4N2kgI> .

stevekuznetsov · 2017-07-12T17:36:06Z

Working on a process to ensure cleanup in GCE so we don't hit quota.

smarterclayton · 2017-07-14T03:56:02Z

We have node logs now. What do they say? On Jul 13, 2017, at 10:17 PM, Ben Parees <[email protected]> wrote: next up: Failure summary: 1. Host: ci-prtest-5a37c28-4303-ig-m-rvst Play: Configure nodes Task: openshift_node : restart node Message: Unable to restart service origin-node: Job for origin-node.service failed because the control process exited with error code. See "systemctl status origin-node.service" and "journalctl -xe" for details. 2. Host: ci-prtest-5a37c28-4303-ig-n-x2bq Play: Configure nodes Task: openshift_node : restart node Message: Unable to restart service origin-node: Job for origin-node.service failed because the control process exited with error code. See "systemctl status origin-node.service" and "journalctl -xe" for details. 3. Host: ci-prtest-5a37c28-4303-ig-n-qs80 Play: Configure nodes Task: openshift_node : restart node Message: Unable to restart service origin-node: Job for origin-node.service failed because the control process exited with error code. See "systemctl status origin-node.service" and "journalctl -xe" for details. 4. Host: ci-prtest-5a37c28-4303-ig-n-hj1f Play: Configure nodes Task: openshift_node : restart node Message: Unable to restart service origin-node: Job for origin-node.service failed because the control process exited with error code. See "systemctl status origin-node.service" and "journalctl -xe" for details. https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/4303 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p4WdlwxypCXMp5S6exszCMAfCJP8ks5sNs_BgaJpZM4N2kgI> .

bparees · 2017-07-14T04:02:08Z

if they're gathered in s3 artifacts i don't see them. only journal item is docker.service

sdodson · 2017-07-14T12:36:54Z

Yeah, we need those. Really we need the full binary journal, no sense in gathering only specific journals when the ones we want are probably 90% of the total log volume anyway. I'll look at where that work got left off.

stevekuznetsov · 2017-07-14T13:29:50Z

Looks like we ensured the artifact gathering step could not fail but we still could have a single failure in the step skip executing of the rest of it, causing few or no artifacts to be gathered. I've fixed that in openshift-eng/aos-cd-jobs#433

0xmichalis · 2017-07-20T14:30:36Z

https://ci.openshift.redhat.com/jenkins/view/All/job/merge_pull_request_origin/1321/consoleFull#-10111959258b6e51eb7608a5981914356

enj · 2017-07-22T15:13:24Z

For #15348 we may need apache/libcloud@03df6a8

smarterclayton · 2017-07-22T16:53:38Z

Can you pick that into origin-gce? On Jul 22, 2017, at 11:13 AM, Mo Khan <[email protected]> wrote: For #15348 <#15348> we may need apache/libcloud@03df6a8 <apache/libcloud@03df6a8> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p8BWvMvq0DcGxrmlyYrDd3gqSCLpks5sQhGWgaJpZM4N2kgI> .

sdodson · 2017-07-26T19:56:12Z

So what's in origin-gce is just gce.py which is the ansible inventory script which relies on libcloud. Which version of python2-libcloud is installed on the host?

sdodson · 2017-07-26T20:04:45Z

Nevermind the latest version of the package doesn't have that fix.

stevekuznetsov · 2017-08-22T14:15:33Z

@smarterclayton @enj do we have resolution for this

enj · 2017-08-22T18:37:47Z

@stevekuznetsov that fix is tagged with v2.1.0-tentative but the highest RPM release I have seen is v2.0.

simo5 · 2017-08-23T17:28:52Z

Just hit this, not identical but seem closely related:

"ERROR: (gcloud.compute.instances.attach-disk) Could not fetch resource:", " - The resource 'projects/openshift-gce-devel-ci/zones/us-central1-a/instances/ci-prtest-5a37c28-6439-ig-n-38r7' was not found"], "stdout": "NAME                                MODE  IPV4_RANGE  GATEWAY_IPV4\nci-prtest-5a37c28-6439-ocp-network  auto\nNAME                                    NETWORK                             SRC_RANGES  RULES                                                                      SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-master-external  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   tcp:80,tcp:443,tcp:1936,tcp:8080,tcp:8443,tcp:30000-32000,udp:30000-32000            ocp-master\nNAME                         NETWORK                             SRC_RANGES  RULES  SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-icmp  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   icmp\nNAME                                 NETWORK                             SRC_RANGES  RULES   SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-ssh-internal  ci-prtest-5a37c28-6439-ocp-network              tcp:22  bastion\nNAME                                        NETWORK                             SRC_RANGES  RULES     SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-infra-node-internal  ci-prtest-5a37c28-6439-ocp-network              tcp:5000  ocp       ocp-infra-node\nNAME                                    NETWORK                             SRC_RANGES  RULES                                                                                                                                        SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-master-internal  ci-prtest-5a37c28-6439-ocp-network              tcp:2224,tcp:2379,tcp:2380,tcp:4001,udp:4789,udp:5404,udp:5405,tcp:8053,udp:8053,tcp:8444,tcp:10250,tcp:10255,udp:10255,tcp:24224,udp:24224  ocp       ocp-master\nNAME                                  NETWORK                             SRC_RANGES  RULES                                   SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-node-internal  ci-prtest-5a37c28-6439-ocp-network              udp:4789,tcp:10250,tcp:10255,udp:10255  ocp       ocp-node,ocp-infra-node\nNAME                                        NETWORK                             SRC_RANGES  RULES                                                    SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-infra-node-external  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   tcp:80,tcp:443,tcp:1936,tcp:30000-32000,udp:30000-32000            ocp-infra-node\nNAME                                 NETWORK                             SRC_RANGES  RULES   SRC_TAGS  TARGET_TAGS\nci-prtest-5a37c28-6439-ssh-external  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   tcp:22\nNAME                                             MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP\nci-prtest-5a37c28-6439-instance-template-master  n1-standard-2               2017-08-23T09:58:48.326-07:00\nNAME                                           MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP\nci-prtest-5a37c28-6439-instance-template-node  n1-standard-2               2017-08-23T09:58:48.262-07:00\nNAME                                                 MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP\nci-prtest-5a37c28-6439-instance-template-node-infra  n1-standard-2               2017-08-23T09:58:48.210-07:00\nNAME                         LOCATION       SCOPE  BASE_INSTANCE_NAME           SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                              AUTOSCALED\nci-prtest-5a37c28-6439-ig-n  us-central1-a  zone   ci-prtest-5a37c28-6439-ig-n  0     3            ci-prtest-5a37c28-6439-instance-template-node  no\nNAME                         LOCATION       SCOPE  BASE_INSTANCE_NAME           SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                                AUTOSCALED\nci-prtest-5a37c28-6439-ig-m  us-central1-a  zone   ci-prtest-5a37c28-6439-ig-m  0     1            ci-prtest-5a37c28-6439-instance-template-master  no\nNAME                         LOCATION       SCOPE  BASE_INSTANCE_NAME           SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                                    AUTOSCALED\nci-prtest-5a37c28-6439-ig-i  us-central1-a  zone   ci-prtest-5a37c28-6439-ig-i  0     0            ci-prtest-5a37c28-6439-instance-template-node-infra  no\nNAME                                     ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-n-38r7-docker  us-central1-a  75       pd-ssd  READY\nNAME                                     ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-m-4644-docker  us-central1-a  75       pd-ssd  READY\nNAME                                     ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-n-t26c-docker  us-central1-a  75       pd-ssd  READY\nNAME                                     ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-n-9ctw-docker  us-central1-a  75       pd-ssd  READY\nNAME                                        ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-n-38r7-openshift  us-central1-a  50       pd-ssd  READY\nNAME                                        ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-m-4644-openshift  us-central1-a  50       pd-ssd  READY\nNAME                                        ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-n-9ctw-openshift  us-central1-a  50       pd-ssd  READY\nNAME                                        ZONE           SIZE_GB  TYPE    STATUS\nci-prtest-5a37c28-6439-ig-n-t26c-openshift  us-central1-a  50       pd-ssd  READY", "stdout_lines": ["NAME                                MODE  IPV4_RANGE  GATEWAY_IPV4", "ci-prtest-5a37c28-6439-ocp-network  auto", "NAME                                    NETWORK                             SRC_RANGES  RULES                                                                      SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-master-external  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   tcp:80,tcp:443,tcp:1936,tcp:8080,tcp:8443,tcp:30000-32000,udp:30000-32000            ocp-master", "NAME                         NETWORK                             SRC_RANGES  RULES  SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-icmp  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   icmp", "NAME                                 NETWORK                             SRC_RANGES  RULES   SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-ssh-internal  ci-prtest-5a37c28-6439-ocp-network              tcp:22  bastion", "NAME                                        NETWORK                             SRC_RANGES  RULES     SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-infra-node-internal  ci-prtest-5a37c28-6439-ocp-network              tcp:5000  ocp       ocp-infra-node", "NAME                                    NETWORK                             SRC_RANGES  RULES                                                                                                                                        SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-master-internal  ci-prtest-5a37c28-6439-ocp-network              tcp:2224,tcp:2379,tcp:2380,tcp:4001,udp:4789,udp:5404,udp:5405,tcp:8053,udp:8053,tcp:8444,tcp:10250,tcp:10255,udp:10255,tcp:24224,udp:24224  ocp       ocp-master", "NAME                                  NETWORK                             SRC_RANGES  RULES                                   SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-node-internal  ci-prtest-5a37c28-6439-ocp-network              udp:4789,tcp:10250,tcp:10255,udp:10255  ocp       ocp-node,ocp-infra-node", "NAME                                        NETWORK                             SRC_RANGES  RULES                                                    SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-infra-node-external  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   tcp:80,tcp:443,tcp:1936,tcp:30000-32000,udp:30000-32000            ocp-infra-node", "NAME                                 NETWORK                             SRC_RANGES  RULES   SRC_TAGS  TARGET_TAGS", "ci-prtest-5a37c28-6439-ssh-external  ci-prtest-5a37c28-6439-ocp-network  0.0.0.0/0   tcp:22", "NAME                                             MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP", "ci-prtest-5a37c28-6439-instance-template-master  n1-standard-2               2017-08-23T09:58:48.326-07:00", "NAME                                           MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP", "ci-prtest-5a37c28-6439-instance-template-node  n1-standard-2               2017-08-23T09:58:48.262-07:00", "NAME                                                 MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP", "ci-prtest-5a37c28-6439-instance-template-node-infra  n1-standard-2               2017-08-23T09:58:48.210-07:00", "NAME                         LOCATION       SCOPE  BASE_INSTANCE_NAME           SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                              AUTOSCALED", "ci-prtest-5a37c28-6439-ig-n  us-central1-a  zone   ci-prtest-5a37c28-6439-ig-n  0     3            ci-prtest-5a37c28-6439-instance-template-node  no", "NAME                         LOCATION       SCOPE  BASE_INSTANCE_NAME           SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                                AUTOSCALED", "ci-prtest-5a37c28-6439-ig-m  us-central1-a  zone   ci-prtest-5a37c28-6439-ig-m  0     1            ci-prtest-5a37c28-6439-instance-template-master  no", "NAME                         LOCATION       SCOPE  BASE_INSTANCE_NAME           SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                                    AUTOSCALED", "ci-prtest-5a37c28-6439-ig-i  us-central1-a  zone   ci-prtest-5a37c28-6439-ig-i  0     0            ci-prtest-5a37c28-6439-instance-template-node-infra  no", "NAME                                     ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-n-38r7-docker  us-central1-a  75       pd-ssd  READY", "NAME                                     ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-m-4644-docker  us-central1-a  75       pd-ssd  READY", "NAME                                     ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-n-t26c-docker  us-central1-a  75       pd-ssd  READY", "NAME                                     ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-n-9ctw-docker  us-central1-a  75       pd-ssd  READY", "NAME                                        ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-n-38r7-openshift  us-central1-a  50       pd-ssd  READY", "NAME                                        ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-m-4644-openshift  us-central1-a  50       pd-ssd  READY", "NAME                                        ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-n-9ctw-openshift  us-central1-a  50       pd-ssd  READY", "NAME                                        ZONE           SIZE_GB  TYPE    STATUS", "ci-prtest-5a37c28-6439-ig-n-t26c-openshift  us-central1-a  50       pd-ssd  READY"]}

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=1   

Wednesday 23 August 2017  16:59:46 +0000 (0:02:17.379)       0:02:20.148 ****** 
=============================================================================== 
provision : Provision GCE resources ----------------------------------- 137.38s
provision : Provision GCE DNS domain ------------------------------------ 1.52s
provision : Templatize DNS script --------------------------------------- 0.51s
provision : Templatize provision script --------------------------------- 0.39s
provision : Ensure that DNS resolves to the hosted zone ----------------- 0.08s

Failure summary:

  1. Host:     localhost
     Play:     Ensure all cloud resources necessary for the cluster, including instances, have been started
     Task:     provision : Provision GCE resources
     Message:  ???

smarterclayton · 2017-08-23T19:40:43Z

Opened a ticket with GCE support.

…

On Wed, Aug 23, 2017 at 1:28 PM, Simo Sorce ***@***.***> wrote: Just hit this, not identical but seem closely related: "ERROR: (gcloud.compute.instances.attach-disk) Could not fetch resource:", " - The resource 'projects/openshift-gce-devel-ci/zones/us-central1-a/instances/ci-prtest-5a37c28-6439-ig-n-38r7' was not found"], "stdout": "NAME MODE IPV4_RANGE GATEWAY_IPV4\nci-prtest-5a37c28-6439-ocp-network auto\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-master-external ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 tcp:80,tcp:443,tcp:1936,tcp:8080,tcp:8443,tcp:30000-32000,udp:30000-32000 ocp-master\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-icmp ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 icmp\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-ssh-internal ci-prtest-5a37c28-6439-ocp-network tcp:22 bastion\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-infra-node-internal ci-prtest-5a37c28-6439-ocp-network tcp:5000 ocp ocp-infra-node\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-master-internal ci-prtest-5a37c28-6439-ocp-network tcp:2224,tcp:2379,tcp:2380,tcp:4001,udp:4789,udp:5404,udp:5405,tcp:8053,udp:8053,tcp:8444,tcp:10250,tcp:10255,udp:10255,tcp:24224,udp:24224 ocp ocp-master\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-node-internal ci-prtest-5a37c28-6439-ocp-network udp:4789,tcp:10250,tcp:10255,udp:10255 ocp ocp-node,ocp-infra-node\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-infra-node-external ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 tcp:80,tcp:443,tcp:1936,tcp:30000-32000,udp:30000-32000 ocp-infra-node\nNAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS\nci-prtest-5a37c28-6439-ssh-external ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 tcp:22\nNAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP\nci-prtest-5a37c28-6439-instance-template-master n1-standard-2 2017-08-23T09:58:48.326-07:00\nNAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP\nci-prtest-5a37c28-6439-instance-template-node n1-standard-2 2017-08-23T09:58:48.262-07:00\nNAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP\nci-prtest-5a37c28-6439-instance-template-node-infra n1-standard-2 2017-08-23T09:58:48.210-07:00\nNAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED\nci-prtest-5a37c28-6439-ig-n us-central1-a zone ci-prtest-5a37c28-6439-ig-n 0 3 ci-prtest-5a37c28-6439-instance-template-node no\nNAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED\nci-prtest-5a37c28-6439-ig-m us-central1-a zone ci-prtest-5a37c28-6439-ig-m 0 1 ci-prtest-5a37c28-6439-instance-template-master no\nNAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED\nci-prtest-5a37c28-6439-ig-i us-central1-a zone ci-prtest-5a37c28-6439-ig-i 0 0 ci-prtest-5a37c28-6439-instance-template-node-infra no\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-n-38r7-docker us-central1-a 75 pd-ssd READY\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-m-4644-docker us-central1-a 75 pd-ssd READY\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-n-t26c-docker us-central1-a 75 pd-ssd READY\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-n-9ctw-docker us-central1-a 75 pd-ssd READY\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-n-38r7-openshift us-central1-a 50 pd-ssd READY\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-m-4644-openshift us-central1-a 50 pd-ssd READY\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-n-9ctw-openshift us-central1-a 50 pd-ssd READY\nNAME ZONE SIZE_GB TYPE STATUS\nci-prtest-5a37c28-6439-ig-n-t26c-openshift us-central1-a 50 pd-ssd READY", "stdout_lines": ["NAME MODE IPV4_RANGE GATEWAY_IPV4", "ci-prtest-5a37c28-6439-ocp-network auto", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-master-external ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 tcp:80,tcp:443,tcp:1936,tcp:8080,tcp:8443,tcp:30000-32000,udp:30000-32000 ocp-master", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-icmp ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 icmp", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-ssh-internal ci-prtest-5a37c28-6439-ocp-network tcp:22 bastion", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-infra-node-internal ci-prtest-5a37c28-6439-ocp-network tcp:5000 ocp ocp-infra-node", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-master-internal ci-prtest-5a37c28-6439-ocp-network tcp:2224,tcp:2379,tcp:2380,tcp:4001,udp:4789,udp:5404,udp:5405,tcp:8053,udp:8053,tcp:8444,tcp:10250,tcp:10255,udp:10255,tcp:24224,udp:24224 ocp ocp-master", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-node-internal ci-prtest-5a37c28-6439-ocp-network udp:4789,tcp:10250,tcp:10255,udp:10255 ocp ocp-node,ocp-infra-node", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-infra-node-external ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 tcp:80,tcp:443,tcp:1936,tcp:30000-32000,udp:30000-32000 ocp-infra-node", "NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS", "ci-prtest-5a37c28-6439-ssh-external ci-prtest-5a37c28-6439-ocp-network 0.0.0.0/0 tcp:22", "NAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP", "ci-prtest-5a37c28-6439-instance-template-master n1-standard-2 2017-08-23T09:58:48.326-07:00", "NAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP", "ci-prtest-5a37c28-6439-instance-template-node n1-standard-2 2017-08-23T09:58:48.262-07:00", "NAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP", "ci-prtest-5a37c28-6439-instance-template-node-infra n1-standard-2 2017-08-23T09:58:48.210-07:00", "NAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED", "ci-prtest-5a37c28-6439-ig-n us-central1-a zone ci-prtest-5a37c28-6439-ig-n 0 3 ci-prtest-5a37c28-6439-instance-template-node no", "NAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED", "ci-prtest-5a37c28-6439-ig-m us-central1-a zone ci-prtest-5a37c28-6439-ig-m 0 1 ci-prtest-5a37c28-6439-instance-template-master no", "NAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED", "ci-prtest-5a37c28-6439-ig-i us-central1-a zone ci-prtest-5a37c28-6439-ig-i 0 0 ci-prtest-5a37c28-6439-instance-template-node-infra no", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-n-38r7-docker us-central1-a 75 pd-ssd READY", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-m-4644-docker us-central1-a 75 pd-ssd READY", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-n-t26c-docker us-central1-a 75 pd-ssd READY", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-n-9ctw-docker us-central1-a 75 pd-ssd READY", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-n-38r7-openshift us-central1-a 50 pd-ssd READY", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-m-4644-openshift us-central1-a 50 pd-ssd READY", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-n-9ctw-openshift us-central1-a 50 pd-ssd READY", "NAME ZONE SIZE_GB TYPE STATUS", "ci-prtest-5a37c28-6439-ig-n-t26c-openshift us-central1-a 50 pd-ssd READY"]} PLAY RECAP ********************************************************************* localhost : ok=4 changed=3 unreachable=0 failed=1 Wednesday 23 August 2017 16:59:46 +0000 (0:02:17.379) 0:02:20.148 ****** =============================================================================== provision : Provision GCE resources ----------------------------------- 137.38s provision : Provision GCE DNS domain ------------------------------------ 1.52s provision : Templatize DNS script --------------------------------------- 0.51s provision : Templatize provision script --------------------------------- 0.39s provision : Ensure that DNS resolves to the hosted zone ----------------- 0.08s Failure summary: 1. Host: localhost Play: Ensure all cloud resources necessary for the cluster, including instances, have been started Task: provision : Provision GCE resources Message: ??? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pzGOYrjciBmKz-sRrVR5o3YP-wbDks5sbGFagaJpZM4N2kgI> .

enj · 2017-08-23T20:05:43Z

Seen in #15907 https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/6456/

smarterclayton · 2017-08-23T22:24:56Z

Fixed by openshift/origin-gce#42 (because we don't attach anymore) which is going to merge O(soon)

…

On Wed, Aug 23, 2017 at 4:05 PM, Mo Khan ***@***.***> wrote: Seen in #15907 <#15907> https://ci.openshift.redhat.com/jenkins/job/test_pull_ request_origin_extended_conformance_gce/6456/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pzZg0VygvGLInIuUuo2mM5OhflyXks5sbIYbgaJpZM4N2kgI> .

smarterclayton · 2017-08-24T13:16:59Z

The change was delivered to origin-gce and a new image was built, but I'm not seeing new jobs use the image yet for some reason. On Aug 24, 2017, at 8:39 AM, Scott Dodson <[email protected]> wrote: Assigned #14575 <#14575> to @smarterclayton <https://github.com/smarterclayton>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p9bl-O3oupOhw6P25iA8z0HOs8BJks5sbW7_gaJpZM4N2kgI> .

smarterclayton · 2017-08-24T17:41:36Z

/close Fixed On Thu, Aug 24, 2017 at 9:16 AM, Clayton Coleman <[email protected]> wrote:

…

The change was delivered to origin-gce and a new image was built, but I'm not seeing new jobs use the image yet for some reason. On Aug 24, 2017, at 8:39 AM, Scott Dodson ***@***.***> wrote: Assigned #14575 <#14575> to @smarterclayton <https://github.com/smarterclayton>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p9bl-O3oupOhw6P25iA8z0HOs8BJks5sbW7_gaJpZM4N2kgI> .

simo5 · 2017-08-31T13:43:25Z

Still seeing this:

TASK [gce-provision : Provision GCE resources] *********************************
Wednesday 30 August 2017  19:11:04 +0000 (0:00:00.076)       0:00:02.751 ****** 
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/tmp/provision.sh"], "delta": "0:00:51.272819", "end": "2017-08-30 19:11:55.974629", "failed": true, "rc": 1, "start": "2017-08-30 19:11:04.701810", "stderr": "ERROR: (gcloud.compute.networks.create) Could not fetch resource:\n - Quota 'ROUTES' exceeded.  Limit: 300.0", "stderr_lines": ["ERROR: (gcloud.compute.networks.create) Could not fetch resource:", " - Quota 'ROUTES' exceeded.  Limit: 300.0"], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=1   

Wednesday 30 August 2017  19:11:55 +0000 (0:00:51.389)       0:00:54.140 ****** 
=============================================================================== 
gce-provision : Provision GCE resources -------------------------------- 51.39s
gce-provision : Provision GCE DNS domain -------------------------------- 1.59s
gce-provision : Templatize DNS script ----------------------------------- 0.51s
gce-provision : Templatize provision script ----------------------------- 0.38s
gce-provision : Ensure that DNS resolves to the hosted zone ------------- 0.08s

Failure summary:

  1. Host:     localhost
     Play:     Ensure all cloud resources necessary for the cluster, including instances, have been started
     Task:     gce-provision : Provision GCE resources
     Message:  ???
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace

stevekuznetsov · 2017-08-31T13:44:38Z

Quota 'ROUTES' exceeded

We had some issues. Resolved today.

bparees added kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1 labels Jun 12, 2017

bparees assigned stevekuznetsov Jun 12, 2017

bparees mentioned this issue Jun 12, 2017

Support valueFrom syntax for build env vars #14143

Merged

stevekuznetsov removed the priority/P1 label Jun 16, 2017

pweil- added the priority/P2 label Jun 19, 2017

soltysh mentioned this issue Jun 27, 2017

Update default allowed registries #14850

Merged

This was referenced Jun 29, 2017

more env, server side debug for pipeline ext tests #14949

Merged

flake: GCE cluster provision failure (2) #14964

Closed

0xmichalis mentioned this issue Jul 5, 2017

Break up os::cleanup::all #14982

Merged

smarterclayton added priority/P0 and removed priority/P2 labels Jul 5, 2017

coreydaley mentioned this issue Jul 5, 2017

Throw error using --context-dir with a template #15019

Merged

pweil- assigned sdodson and unassigned stevekuznetsov Jul 12, 2017

stevekuznetsov mentioned this issue Jul 20, 2017

[3.6] Fix issues related to the migration to RBAC and the Kubernetes authorizer #15348

Merged

stevekuznetsov mentioned this issue Jul 26, 2017

flake: Failed to deprovision cluster: GCENodeDriver object has no attribute '_ex_populate_dict' #15451

Closed

simo5 mentioned this issue Aug 23, 2017

Ensure AddPostStartHook always succeeds, or die #15881

Merged

enj mentioned this issue Aug 23, 2017

Clarify router cmd ignoreError comments #15907

Merged

simo5 mentioned this issue Aug 23, 2017

move more RBAC to legacy package #15936

Merged

sdodson assigned smarterclayton and unassigned sdodson Aug 24, 2017

mrogers950 mentioned this issue Aug 24, 2017

Add a router cmd test to exercise ignoreError() locally #15940

Merged

openshift-ci-robot closed this as completed Aug 24, 2017

simo5 mentioned this issue Aug 31, 2017

Move project creation to use RBAC objects #15973

Merged

simo5 mentioned this issue Aug 31, 2017

Remove usless test and resolved comments #16000

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCE cluster provision failure #14575

GCE cluster provision failure #14575

bparees commented Jun 12, 2017

stevekuznetsov commented Jun 12, 2017

stevekuznetsov commented Jun 12, 2017

smarterclayton commented Jun 12, 2017

stevekuznetsov commented Jun 16, 2017

pweil- commented Jun 19, 2017

0xmichalis commented Jul 5, 2017

smarterclayton commented Jul 5, 2017

derekwaynecarr commented Jul 5, 2017

smarterclayton commented Jul 5, 2017 via email

0xmichalis commented Jul 11, 2017

smarterclayton commented Jul 11, 2017 via email

bparees commented Jul 12, 2017

bparees commented Jul 12, 2017

smarterclayton commented Jul 12, 2017 via email

stevekuznetsov commented Jul 12, 2017

smarterclayton commented Jul 14, 2017 via email

bparees commented Jul 14, 2017

sdodson commented Jul 14, 2017

stevekuznetsov commented Jul 14, 2017 •

edited

Loading

0xmichalis commented Jul 20, 2017

enj commented Jul 22, 2017

smarterclayton commented Jul 22, 2017 via email

sdodson commented Jul 26, 2017

sdodson commented Jul 26, 2017

stevekuznetsov commented Aug 22, 2017

enj commented Aug 22, 2017

simo5 commented Aug 23, 2017

smarterclayton commented Aug 23, 2017 via email

enj commented Aug 23, 2017

smarterclayton commented Aug 23, 2017 via email

smarterclayton commented Aug 24, 2017 via email

smarterclayton commented Aug 24, 2017 via email

simo5 commented Aug 31, 2017

stevekuznetsov commented Aug 31, 2017

GCE cluster provision failure #14575

GCE cluster provision failure #14575

Comments

bparees commented Jun 12, 2017

stevekuznetsov commented Jun 12, 2017

stevekuznetsov commented Jun 12, 2017

smarterclayton commented Jun 12, 2017

stevekuznetsov commented Jun 16, 2017

pweil- commented Jun 19, 2017

0xmichalis commented Jul 5, 2017

smarterclayton commented Jul 5, 2017

derekwaynecarr commented Jul 5, 2017

smarterclayton commented Jul 5, 2017 via email

0xmichalis commented Jul 11, 2017

smarterclayton commented Jul 11, 2017 via email

bparees commented Jul 12, 2017

bparees commented Jul 12, 2017

smarterclayton commented Jul 12, 2017 via email

stevekuznetsov commented Jul 12, 2017

smarterclayton commented Jul 14, 2017 via email

bparees commented Jul 14, 2017

sdodson commented Jul 14, 2017

stevekuznetsov commented Jul 14, 2017 • edited Loading

0xmichalis commented Jul 20, 2017

enj commented Jul 22, 2017

smarterclayton commented Jul 22, 2017 via email

sdodson commented Jul 26, 2017

sdodson commented Jul 26, 2017

stevekuznetsov commented Aug 22, 2017

enj commented Aug 22, 2017

simo5 commented Aug 23, 2017

smarterclayton commented Aug 23, 2017 via email

enj commented Aug 23, 2017

smarterclayton commented Aug 23, 2017 via email

smarterclayton commented Aug 24, 2017 via email

smarterclayton commented Aug 24, 2017 via email

simo5 commented Aug 31, 2017

stevekuznetsov commented Aug 31, 2017

stevekuznetsov commented Jul 14, 2017 •

edited

Loading