[WIP] [DO NOT MERGE] Only run fan-in fan-out test #377

bobcatfish · 2019-01-09T19:28:24Z

Trying to gather more info for #375 - does the test fail when it's run
on it's own? (not 100% sure i got the syntax right haha just winging it)

@shashwathi @pivotal-nader-ziada @adrcunha

Trying to gather more info for tektoncd#375 - does the test fail when it's run on it's own? (not 100% sure i got the syntax right haha just winging it)

bobcatfish · 2019-01-09T19:40:59Z

Boo that didn't work:

I0109 19:37:14.793] === RUN   TestPipelineRun
I0109 19:37:14.793] --- PASS: TestPipelineRun (0.00s)
I0109 19:37:14.794] testing: warning: no tests to run
I0109 19:37:14.794] PASS
I0109 19:37:15.798] ok  	github.com/knative/build-pipeline/test	1.309s [no tests to run]

I'll try this locally before pushing again XD

nader-ziada · 2019-01-09T19:47:04Z

the integration tests job looks like it passed, but it didn't actually run any tests

bobcatfish · 2019-01-09T20:06:44Z

Looks like it fails when it's run on it's own and the error I thought was ominous doesn't seem to be there:

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_build-pipeline/377/pull-knative-build-pipeline-integration-tests/1083087926100758530/

This looks interesting tho:

I0109 20:04:48.199] ---
I0109 20:04:48.199] apiVersion: pipeline.knative.dev/v1alpha1
I0109 20:04:48.199] kind: TaskRun
I0109 20:04:48.199] metadata:
I0109 20:04:48.199]   creationTimestamp: "2019-01-09T19:54:48Z"
I0109 20:04:48.199]   generation: 1
I0109 20:04:48.199]   labels:
I0109 20:04:48.200]     pipeline.knative.dev/pipeline: helloworld-pipeline1
I0109 20:04:48.200]     pipeline.knative.dev/pipelineRun: helloworld-pipelinerun1
I0109 20:04:48.200]   name: helloworld-pipelinerun1-create-file-kritis
I0109 20:04:48.200]   namespace: arendelle-gniwfnkv
I0109 20:04:48.200]   ownerReferences:
I0109 20:04:48.200]   - apiVersion: pipeline.knative.dev/v1alpha1
I0109 20:04:48.200]     blockOwnerDeletion: true
I0109 20:04:48.200]     controller: true
I0109 20:04:48.201]     kind: PipelineRun
I0109 20:04:48.201]     name: helloworld-pipelinerun1
I0109 20:04:48.201]     uid: 6b32ed53-1448-11e9-9c86-42010a800104
I0109 20:04:48.201]   resourceVersion: "1505"
I0109 20:04:48.201]   selfLink: /apis/pipeline.knative.dev/v1alpha1/namespaces/arendelle-gniwfnkv/taskruns/helloworld-pipelinerun1-create-file-kritis
I0109 20:04:48.201]   uid: 6b3a19b4-1448-11e9-9c86-42010a800104
I0109 20:04:48.201] spec:
I0109 20:04:48.201]   generation: 1
I0109 20:04:48.201]   inputs:
I0109 20:04:48.201]     resources:
I0109 20:04:48.202]     - name: workspace
I0109 20:04:48.202]       paths: null
I0109 20:04:48.202]       resourceRef:
I0109 20:04:48.202]         name: kritis-resource-git
I0109 20:04:48.202]     - name: gcsbucket
I0109 20:04:48.202]       paths: null
I0109 20:04:48.202]       resourceRef:
I0109 20:04:48.202]         name: gcs-resource
I0109 20:04:48.202]   outputs:
I0109 20:04:48.202]     resources:
I0109 20:04:48.202]     - name: workspace
I0109 20:04:48.203]       paths:
I0109 20:04:48.203]       - /pvc/create-file-kritis/workspace
I0109 20:04:48.203]       resourceRef:
I0109 20:04:48.203]         name: kritis-resource-git
I0109 20:04:48.203]   taskRef:
I0109 20:04:48.203]     kind: Task
I0109 20:04:48.203]     name: create-file
I0109 20:04:48.203]   trigger:
I0109 20:04:48.203]     type: ""
I0109 20:04:48.203] status:
I0109 20:04:48.203]   conditions:
I0109 20:04:48.204]   - lastTransitionTime: "2019-01-09T19:54:58Z"
I0109 20:04:48.204]     message: 'pod status "PodScheduled":"False"; message: "0/3 nodes are available:
I0109 20:04:48.204]       3 node(s) had no available volume zone."'
I0109 20:04:48.204]     reason: Pending
I0109 20:04:48.204]     status: Unknown
I0109 20:04:48.204]     type: Succeeded
I0109 20:04:48.204]   podName: helloworld-pipelinerun1-create-file-kritis-pod-84e1d2
I0109 20:04:48.204]   startTime: "2019-01-09T19:54:48Z"

shashwathi · 2019-01-09T20:10:45Z

@bobcatfish : @pivotal-nader-ziada and I were suspecting if no available volume zone means that test is running across multiple zones and PVCs cannot be shared.

Why is it happening only in this test.
This test is the only with 4 tasks but test fails in running the first task itself so its not because of parallel accessing of PVC.
Uses GCS resource but for something to fail with execution pod should be initialized in first place.
There are other tests that run PipelineRun so it is not creation of PVCs.

bobcatfish · 2019-01-09T20:53:06Z

This test is the only with 4 tasks but test fails in running the first task itself so its not because of parallel accessing of PVC.

I'll try reducing the test to only one task just to see if the failure still occurs!

It looks like the test fails during the first Task so I want to try simplifying the test to see if it will fail with that Task only, or if it's the interaction between tasks that causes it.

bobcatfish · 2019-01-09T21:18:18Z

oooo looks like it still fails, which is interesting:

I0109 21:15:29.426]   taskRuns:
I0109 21:15:29.426]     helloworld-pipelinerun1-create-file-kritis:
I0109 21:15:29.426]       conditions:
I0109 21:15:29.427]       - lastTransitionTime: "2019-01-09T21:05:39Z"
I0109 21:15:29.427]         message: 'pod status "PodScheduled":"False"; message: "0/3 nodes are available:
I0109 21:15:29.427]           3 node(s) had no available volume zone."'
I0109 21:15:29.427]         reason: Pending
I0109 21:15:29.427]         status: Unknown
I0109 21:15:29.427]         type: Succeeded
I0109 21:15:29.427]       podName: helloworld-pipelinerun1-create-file-kritis-pod-442c92
I0109 21:15:29.427]       startTime: "2019-01-09T21:05:29Z"

I noticed this changed in tektoncd@5f28380 and the error we are getting is related to zones so just gonna wing it and see what happens. There was some other interesting stuff that changed too about service accounts that might be interesting to look into (or might have to change with this)

knative-prow-robot · 2019-01-09T21:22:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bobcatfish

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [bobcatfish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bobcatfish · 2019-01-09T21:27:11Z

I0109 21:23:49.413] ---- Checking autogenerated code is up-to-date ----

curses, hacks foiled

Since I'm intentionally trying to mess with this code manually, I need to disable the check that makes sure I haven't messed with it manually XD

bobcatfish · 2019-01-09T22:10:42Z

I0109 21:29:35.339] ERROR: (gcloud.container.get-server-config) One of [--zone, --region] must be supplied: Please specify location.

hm weird. well i think just changing it in one place probably isnt enough after looking at knative/test-infra@c237e79

…al1-a`" That didn't work at all - I think I have to change a bunch of things if I want to go back to how it was (after looking at knative/test-infra@c237e79). This reverts commit daba8cf.

I want to try to simplify the repro case as much as I can, I'm guessing the problem is related to the output.

bobcatfish · 2019-01-09T22:25:06Z

lol well that wasn't nearly clever enough:

I0109 22:22:50.646]   - lastTransitionTime: "2019-01-09T22:22:49Z"
I0109 22:22:50.646]     message: 'invalid output resources: didn''t need these resources but they were
I0109 22:22:50.646]       provided anyway: [workspace]'
I0109 22:22:50.646]     reason: PipelineValidationFailed
I0109 22:22:50.647]     status: "False"
I0109 22:22:50.647]     type: Succeeded
I0109 22:22:50.647]

lol jokes on me I wonder though about the fact that we're still binding all kinds of Tasks that aren't in teh Pipeline - maybe tektoncd#341 will catch that now that it's merged? maybe not tho!

bobcatfish · 2019-01-09T22:42:21Z

Okay looks like it passed without the output! So that (i think) should imply that i can simplify this test down to only having the output and it should fail...

The output is linked to the failure

Looks like we can repro this by just having an output (verified locally with us-central1 zone cluster instead of us-central-1a region cluster)

The pod gets into the bad state immediately, so we should see that right away in the taskrun status. We could make this even shorter, however the presubmit tests take like 10 min on their own anyway so *shrug* XD

knative-prow-robot · 2019-01-09T23:22:39Z

@bobcatfish: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-knative-build-pipeline-integration-tests	`a80baa1`	link	`/test pull-knative-build-pipeline-integration-tests`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

bobcatfish · 2019-01-10T01:34:35Z

Okay I think I figured out what's going on, closing this PR and will update #375.

[WIP] [DO NOT MERGE] Only run fan-in fan-out test

3033e41

Trying to gather more info for tektoncd#375 - does the test fail when it's run on it's own? (not 100% sure i got the syntax right haha just winging it)

knative-prow-robot requested review from ericKlawitter and srinivashegde86 January 9, 2019 19:28

Re-enable the offending test so we can see what happens

7f23174

knative-prow-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 9, 2019

Reduce fan-in fan-out to just one Task

ac337d7

It looks like the test fails during the first Task so I want to try simplifying the test to see if it will fail with that Task only, or if it's the interaction between tasks that causes it.

knative-prow-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 9, 2019

Disable autogenerated code check

ab6d227

Since I'm intentionally trying to mess with this code manually, I need to disable the check that makes sure I haven't messed with it manually XD

bobcatfish added 2 commits January 9, 2019 14:10

Revert "Try changing back from region us-central1 to zone `us-centr…

10575f5

…al1-a`" That didn't work at all - I think I have to change a bunch of things if I want to go back to how it was (after looking at knative/test-infra@c237e79). This reverts commit daba8cf.

See if the output is triggering the problem

658b33d

I want to try to simplify the repro case as much as I can, I'm guessing the problem is related to the output.

Don't try to bind the output I just removed

7abac4f

lol jokes on me I wonder though about the fact that we're still binding all kinds of Tasks that aren't in teh Pipeline - maybe tektoncd#341 will catch that now that it's merged? maybe not tho!

Put the output back

2247481

The output is linked to the failure

bobcatfish mentioned this pull request Jan 9, 2019

TestPipelineRun/fan-in_and_fan-out is broken #375

Closed

bobcatfish added 2 commits January 9, 2019 15:04

Remove stuff not needed to repro

02b6981

Looks like we can repro this by just having an output (verified locally with us-central1 zone cluster instead of us-central-1a region cluster)

Decrease timeout

a80baa1

The pod gets into the bad state immediately, so we should see that right away in the taskrun status. We could make this even shorter, however the presubmit tests take like 10 min on their own anyway so *shrug* XD

knative-prow-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 9, 2019

bobcatfish closed this Jan 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] [DO NOT MERGE] Only run fan-in fan-out test #377

[WIP] [DO NOT MERGE] Only run fan-in fan-out test #377

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

nader-ziada commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

shashwathi commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

knative-prow-robot commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

knative-prow-robot commented Jan 9, 2019

bobcatfish commented Jan 10, 2019

[WIP] [DO NOT MERGE] Only run fan-in fan-out test #377

[WIP] [DO NOT MERGE] Only run fan-in fan-out test #377

Conversation

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

nader-ziada commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

shashwathi commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

knative-prow-robot commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

bobcatfish commented Jan 9, 2019

knative-prow-robot commented Jan 9, 2019

bobcatfish commented Jan 10, 2019