Add e2e test for train API

[helenxie-bit]

What this PR does / why we need it:
Add an e2e test in the test_e2e_train_api.py for the train API.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

[coveralls]

Pull Request Test Coverage Report for Build 11021210555

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 10963743211:	0.0%
Covered Lines:	66
Relevant Lines:	66

---

💛 - Coveralls

[google-oss-prow]

New changes are detected. LGTM label has been removed.

[andreyvelich]

I think, as part of this E2Es we need to build and verify that Trainer and Storage Initializer images are functional with the PR changes.
Since we don't allow to override those image right now, maybe we can solve this problem by implementing logic that I described here: #2247 (comment)

WDYT @deepanker13 @helenxie-bit @kubeflow/wg-training-leads ?

[tenzen-y]

I think, as part of this E2Es we need to build and verify that Trainer and Storage Initializer images are functional with the PR changes. Since we don't allow to override those image right now, maybe we can solve this problem by implementing logic that I described here: #2247 (comment)

WDYT @deepanker13 @helenxie-bit @kubeflow/wg-training-leads ?

That sounds reasonable. Actually, #2247 has already been merged. So, we can do it in this PR.
@helenxie-bit Could you update this PR to verify the image every for PR?

[helenxie-bit]

I think, as part of this E2Es we need to build and verify that Trainer and Storage Initializer images are functional with the PR changes. Since we don't allow to override those image right now, maybe we can solve this problem by implementing logic that I described here: #2247 (comment)
WDYT @deepanker13 @helenxie-bit @kubeflow/wg-training-leads ?

That sounds reasonable. Actually, #2247 has already been merged. So, we can do it in this PR. @helenxie-bit Could you update this PR to verify the image every for PR?

Sure, I attempted to add the code for building and verifying the Storage Initializer and Trainer images, but the e2e tests failed due to a 'no space left on device' error when loading the images to the kind cluster. I’ve already tried pruning unused images and build caches, but there still isn’t enough space. Do you have any suggestions on how to resolve this? @andreyvelich @tenzen-y

[andreyvelich]

@tenzen-y Any thoughts on this ? Do we need to migrate to Minikube similar to Katib ?
For Katib we load a lot of images to the CI cluster without any issues: https://github.com/kubeflow/katib/blob/master/test/e2e/v1beta1/scripts/gh-actions/build-load.sh#L55

[helenxie-bit]

@andreyvelich @tenzen-y According to today's discussion, I have tried "docker system prune -a -f" but it still cannot fix the problem. Should we try to build a smaller image specifically for e2e test? Do you have any suggestions about the base image that we can use?