[scalability testing] large number of jobs (100?) running concurrently?

[jlewi]

As a step towards reaching v1, I think we should consider doing a scalability test to see if we can correctly handle a large number 100-1000 of jobs running concurrently.

I think what we want to look at is:

1. CPU/MEM usage of the operator
2. What is the latency with which the operator responds to events
   • e.g. when a pod dies, does the operator still process it in a timely fashion.

For the latter part we might need to add instrumentation to TFOperator to report metrics about event processing to Prometheus. Would be good to sync with folks in K8s community to see what they do.

/cc @richardsliu
/cc @johnugeorge

[chrisheecho]

/priority p1

[chrisheecho]

/priority p2

Don't have large users hitting this as far as I'm aware

[jbottum]

/remove-priority p1

[xyhuang]

/assign

[richardsliu]

Anything else left to do here? Otherwise we can close this.

[richardsliu]

Results can be found at https://bit.ly/2CtWFn3. Closing this for now.
/close

[k8s-ci-robot]

@richardsliu: Closing this issue.

In response to this:

Results can be found at https://bit.ly/2CtWFn3. Closing this for now.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.