Private Endpoint Tasks

[Jooho]

This is tracking GH issue.

• odh-model-controller (Feature: Create cert Secret and update KServe local gateway odh-model-controller#221, Internal hostname gateway follow up odh-model-controller#229)

  • annotate serving cert for runtime service and copy it to istio-system namespace.
  • update gateway.
  • Sync to 0.12.0 (Sync 0.12.0 with main branch for rhoai 2.11 odh-model-controller#231)

• upstream/kserve

  • create a new Gateway/Service in doc
  • update VS local gateway name (Allow KServe to have its own local gateways for Serverless mode kserve/kserve#3737)

• opendatahub-operator ([RHOAIENG-7919] add kserve-local-gateway Gateway and Service opendatahub-operator#1056)

  • create a new Gateway/Service for kserve

• odh/kserve

  • update inferenceservice configmap
    • master - update local gateway information #372
    • 0.12.1 - [Cherry-pick] update local gateway information #373
  • cherry-pick creating VS PR
    • master - [cherry-pick] Prevent the PassthroughCluster for clients in the service mesh #369
    • 0.12.1 - cherry-pick(#369) and fix conflict #370)
  • update VS local gateway name (Allow KServe to have its own local gateways for Serverless mode #376)

[Jooho]

Full test

Setup Test Environment to verify all PRs
PR list

• kserve - 0.12.1 branch
  • Allow KServe to have its own local gateways for Serverless mode
  • update local gateway information
  • Prevent the PassthroughCluster for clients/workloads in the service mesh (kserve#3711)
• opendatahub operator - incubating branch
  • [RHOAIENG-7919] add kserve-local-gateway Gateway and Service
• odh model controller - master branch
  • Feature: Create cert Secret and update KServe local gateway

images

• quay.io/jooholee/odh-model-controller:private
• quay.io/jooholee/kserve-controller:private
• quay.io/jooholee/opendatahub-operator:private
• quay.io/jooholee/opendatahub-operator-catalog:0.0.0

Manual way

Install custom odh operator/kserve operator/odh-model-controller operator

• Manifests

  • CUSTOM_KSERVE_MANIFESTS=https://github.com/jooho/kserve/tarball/test-manifest
  • CUSTOM_ODH_MODEL_CONTROLLER_MANIFESTS=https://github.com/jooho/odh-model-controller/tarball/test-manifests
  • CUSTOM_ODH_OPERATOR_CATALOGSOURCE_IMG=quay.io/jooholee/opendatahub-operator-catalog:v0.0.0

• Deploy test sklearn iris model

  • ServingRuntime: https://raw.githubusercontent.com/Jooho/loopy/main/src/roles/kserve/samples/sklearn/manifests/sklearn/sklearn-runtime.yaml
  • ISVC: https://raw.githubusercontent.com/Jooho/loopy/main/src/roles/kserve/samples/sklearn/manifests/sklearn/sklearn-isvc-iris-v2-storageuri-rest.yaml

---

Script way

Loopy Setup
refer this doc

git clone   [email protected]:Jooho/loopy.git
cd loopy
make init
 
or

docker run -it quay.io/jooholee/loopy

cat cluster_info.sh
CLUSTER_CONSOLE_URL=https://console-openshift-console.XXXX
CLUSTER_API_URL=https://api.XXX:443
CLUSTER_ADMIN_ID=admin
CLUSTER_ADMIN_PW=password
CLUSTER_TOKEN=sha256~XXX
CLUSTER_TYPE=ROSA

Install Kserve with a new manifests

loopy playbooks run private-endpoint-poc -p CUSTOM_KSERVE_MANIFESTS=https://github.com/jooho/kserve/tarball/test-manifest  -p CUSTOM_ODH_MODEL_CONTROLLER_MANIFESTS=https://github.com/jooho/odh-model-controller/tarball/test-manifests -vvv -i ./cluster_info.sh

Deploy sample sklearn model

loopy units run test-kserve-sklearn-v2-iris-role -vvv -i ./cluster_info.sh 

---

Test from the same namespace

1. Running test pods

TEST_NAMESPACE=kserve-demo

oc label namespace ${TEST_NAMESPACE} pod-security.kubernetes.io/enforce=baseline --overwrite
oc label namespace ${TEST_NAMESPACE} pod-security.kubernetes.io/warn=baseline --overwrite
oc label namespace ${TEST_NAMESPACE} pod-security.kubernetes.io/audit=baseline  --overwrite

oc run test-with-istio  -n ${TEST_NAMESPACE} --image=registry.access.redhat.com/rhel7/rhel-tools --annotations=sidecar.istio.io/inject="true" -- sleep infinity 2> /dev/null
oc run test  -n ${TEST_NAMESPACE} --image=registry.access.redhat.com/rhel7/rhel-tools -- sleep infinity 2> /dev/null

2. curl from a pod with istio-proxy from the same cluster

oc exec test-with-istio -n ${TEST_NAMESPACE} -- curl -s sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live

3. curl from a pod without istio-proxy from the same cluster

oc exec test -n ${TEST_NAMESPACE}  -- curl -sk https://sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live

Test from another namespace

1. Running test pods

ANOTHER_NAMESPACE=curl-test
oc new-project ${ANOTHER_NAMESPACE}
oc label namespace ${ANOTHER_NAMESPACE} pod-security.kubernetes.io/enforce=baseline --overwrite
oc label namespace ${ANOTHER_NAMESPACE} pod-security.kubernetes.io/warn=baseline --overwrite
oc label namespace ${ANOTHER_NAMESPACE} pod-security.kubernetes.io/audit=baseline  --overwrite
cat << EOF |oc create -f -
kind: ServiceMeshMember
apiVersion: maistra.io/v1
metadata:
  name: default
  namespace:  ${ANOTHER_NAMESPACE} 
spec:
  controlPlaneRef:
    name: data-science-smcp
    namespace: istio-system
EOF


oc run test-with-istio  -n  ${ANOTHER_NAMESPACE}  --image=registry.access.redhat.com/rhel7/rhel-tools -n ${ANOTHER_NAMESPACE}  --annotations=sidecar.istio.io/inject="true" -- sleep infinity 2> /dev/null
oc run test  -n  ${ANOTHER_NAMESPACE}  --image=registry.access.redhat.com/rhel7/rhel-tools -n ${ANOTHER_NAMESPACE} -- sleep infinity 2> /dev/null

2. curl from a pod with istio-proxy from another cluster

oc exec test-with-istio  -n ${ANOTHER_NAMESPACE} -- curl -s sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live

3. curl from a pod without istio-proxy from another cluster

oc exec test -n ${ANOTHER_NAMESPACE} -- curl -sk https://sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live 

[Jooho]

all done