[scalability testing] large number of replicas (100)

[jlewi]

As a step towards reaching v1, I think we should consider doing a scalability test to see if we can correctly handle a large number 100-1000 of jobs running concurrently.

I think what we want to look at is:

1. CPU/MEM usage of the operator
2. What is the latency with which the operator responds to events
   e.g. when a pod dies, does the operator still process it in a timely fashion.

For the latter part we might need to add instrumentation to TFOperator to report metrics about event processing to Prometheus. Would be good to sync with folks in K8s community to see what they do.

See also:
#829 scalability testing for large number of concurrent jobs.

/cc @richardsliu
/cc @johnugeorge

[johnugeorge]

@gaocegege how should we plan this ?

[richardsliu]

@johnugeorge I think we should first try to enhance the test code so it becomes easier to add an E2E test. Then we can extend the infrastructure to support stress testing and collect metrics.

[johnugeorge]

Are you planning to add this as a E2E tests for every PR presubmit? I don't know if it is feasible. Is Kubernetes doing the same way?

[richardsliu]

Not on pre or post submits. But we can do this periodically on release (stable) branches.

[jlewi]

Downgrading to P1
/priority p1
/cancel-priority p0

[xyhuang]

/assign

[richardsliu]

Anything else left to do here? Otherwise we can close this.

[richardsliu]

Results can be found at https://bit.ly/2CtWFn3. Closing this for now.
/close

[k8s-ci-robot]

@richardsliu: Closing this issue.

In response to this:

Results can be found at https://bit.ly/2CtWFn3. Closing this for now.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.