Set the volume mount's readonly annotation based on the ISVC annotation

[hdefazio]

What this PR does / why we need it:
Fixes kserve#3687
https://issues.redhat.com/browse/RHOAIENG-8288

Type of changes
Bug fix (non-breaking change which fixes an issue)

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test Setup:

• Created a cluster
• Installed Service Mesh, Serverless, Opendatahub operators (in that order)
• Created a DSCI and waited for it to be Ready
• Created a DSC and waited for it to be Ready
• Scaled deployments -> opendatahub-operator-controller-manager down to zero in the openshift-operators namespace
• Replaced the image in deployments -> kserve-controller with my local image in the opendatahub namespace
• Created a serving runtime via yaml config
• Created an ISVC via a yaml config

Test Cases:
storage.kserve.io/readonly: "false"

storage.kserve.io/readonly: "true"

storage.kserve.io/readonly was not provided

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

• Have you added unit/e2e tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Release note:

Re-running failed tests

• /rerun-all - rerun all failed workflows.
• /rerun-workflow <workflow name> - rerun a specific failed workflow. Only one workflow name can be specified. Multiple /rerun-workflow commands are allowed per comment.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hdefazio
Once this PR has been reviewed and has the lgtm label, please assign xaenalt for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[spolti]

@terrytangyuan this changes looks ok, do you want to take another look?
And, @harshad16 before we propose it upstream, do you mind building the kserve image and test with ODH?

[harshad16]

before we propose it upstream, do you mind building the kserve image and test with ODH?

@spolti do you mean to tag me here? not sure if this relate to the work i do.

[terrytangyuan]

This PR is still in draft. Is it ready for review?

[terrytangyuan]

I mentioned previously that we'll need the upstream to review this as well. Do we have a PR in upstream?

[hdefazio]

@spolti Just curious - does the readonly annotation key get dropped when the pod is created? Or should that field be here too? (the test isn't failing as-is)

[spolti]

before we propose it upstream, do you mind building the kserve image and test with ODH?

@spolti do you mean to tag me here? not sure if this relate to the work i do.

Sry, wrong tag =/

[hdefazio]

This PR is still in draft. Is it ready for review?

@terrytangyuan This is ready for review now

[hdefazio]

Closing in favor of kserve#3885