test: Add TorchServe FVT

[rafvasq]

Motivation

Support for TorchServe was added in #250 and kserve/modelmesh-runtime-adapter#34. A test should be added for it as well.

Modifications

• Adds basic FVT for load/inference with a TorchServe MAR model using the native TorchServe gRPC API
• Disables OVMS runtime and tests to allow TorchServe to be tested due to resource constraints

Result

Closes #280

[rafvasq]

To unblock this PR, will work with @ckadner to add the TorchServe model used for the new FVT test to the modelmesh-minio-dev-examples image. From there, I can confirm whether the PR passes the FVTs as expected.

[ckadner]

To unblock this PR, will work with @ckadner to add the TorchServe model used for the new FVT test to the modelmesh-minio-dev-examples image. From there, I can confirm whether the PR passes the FVTs as expected.

@rafvasq -- I pushed a new modelmesh-minio-dev-examples:latest image with the FVT models you added in PR #300

https://hub.docker.com/r/kserve/modelmesh-minio-dev-examples/tags

[ckadner]

Even though the FVTs are failing, I no longer see the pytorch-mar predictor tests fail, like this one 2 weeks ago

Summarizing 2 Failures:
  [FAIL] Predictor update pytorch-mar predictor [BeforeEach] should successfully update and reload the model
  /home/runner/work/modelmesh-serving/modelmesh-serving/fvt/helpers.go:290
  [FAIL] Predictor create pytorch-mar predictor [It] should successfully load a model
  /home/runner/work/modelmesh-serving/modelmesh-serving/fvt/helpers.go:290

Ran 42 of 136 Specs in 200.237 seconds
FAIL! - Interrupted by Other Ginkgo Process -- 40 Passed | 2 Failed | 0 Pending | 94 Skipped

So I assume the new modelmesh-minio-dev-examples image works okay, no?

[rafvasq]

@ckadner, that's right. I can see that the pytorch-mar predictor is being loaded in the test:

1.6742378737294984e+09	INFO	Watcher got event with object	{"name": "pytorch-mar-predictor-4vnh9", "status.available": true, "status.activeModelState": "Loaded", "status.targetModelState": "", "status.transitionStatus": "UpToDate", "status.lastFailureInfo": null}

I did have to update .github/workflows/run-fvt.yml to load the TorchServe runtime too since it was disabled. I'm aware that there could be resource issues related to loaded all 4 runtimes at once, but I'm debugging to figure out if it's failing because of that.

Seems like it's getting past the TorchServe tests though so I believe the new modelmesh-minio-dev-examples image is working just fine. :)

[ckadner]

Hi @rafvasq -- I see you are still battling FVT errors. When I checked yesterday, I saw this error which I assume was caused by resource restriction.

https://github.com/kserve/modelmesh-serving/actions/runs/4002549941/jobs/6869875571#step:13:3630

  Predictor pytorch-mar-predictor-46866 state should not be changed from 'Loading' to 'FailedToLoad'
  Expected
      <bool>: false
  to be true
  In [It] at: /home/runner/work/modelmesh-serving/modelmesh-serving/fvt/helpers.go:286

Should we try to increase the resources on the FVT cluster to accommodate the additional workload?

[rafvasq]

I've created #320 to address FVT errors going forward and as @ckadner noted there, the Github constraints are separate from the nightly e2e-pipeline on IBM Cloud.

Re: this PR, TorchServe FVTs have been proven to pass and will stay disabled for now due to resource-constraints.

[ckadner]

/lgtm

[njhill]

Thanks @rafvasq LGTM, just a small comment.

[njhill]

Maybe add the reason here?

[kserve-oss-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner, njhill, rafvasq

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [njhill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment