KEP-2170: Strictly verify the CRD marker validation and defaulting in the integration testings

[tenzen-y]

What this PR does / why we need it:
I refactored the CRD marker integration testing so that we could more strictly verify the expected behavior.
Additionally, I made the .spec.runtimeRef field as immutable field to reduce unexpected errors when actual Jobs are built.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #2293
Relates to #2170

Checklist:

• Docs included if any changes are user facing

[tenzen-y]

/hold
/assign @kubeflow/wg-training-leads
/cc @akshaychitneni

[google-oss-prow]

@tenzen-y: GitHub didn't allow me to request PR reviews from the following users: akshaychitneni.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/hold
/assign @kubeflow/wg-training-leads
/cc @akshaychitneni

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[coveralls]

Pull Request Test Coverage Report for Build 11489327226

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 11482479049:	0.0%
Covered Lines:	77
Relevant Lines:	77

---

💛 - Coveralls

[andreyvelich]

When we could return this error ?

[tenzen-y]

We will use this in webhook impl similar to

training-operator/pkg/webhooks/pytorch/pytorchjob_webhook.go

Line 102 in ab6938c

allErrs = append(allErrs, field.Forbidden(elasticNProcPerNodePath, fmt.Sprintf("must not be used with %s", nprocPerNodePath)))

[andreyvelich]

Thank you @tenzen-y
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

/hold cancel