Task/APPENG-2738: NIM Cypress E2E tests #3298

dmartinol · 2024-10-04T10:03:24Z

Description

Adding Cypress UI tests to cover the NIM Model use cases from all the supported pages and tabs:

Show/Hide NVIDIA NIM model serving platform
Deploy NVIDIA NIM models
List NVIDIA NIM models
Delete NVIDIA NIM model

How Has This Been Tested?

Running npm run test:cypress-ci command.

Test Impact

Increases code coverage of UI code for all the components impacted by the NIM models.

Request review criteria:

Self checklist (all need to be checked):

The developer has manually tested the changes and verified that the changes work
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has added tests or explained why testing cannot be added (unit or cypress tests for related changes)

After the PR is posted & before it merges:

The developer has tested their solution on a cluster by using the image produced by the PR to main

openshift-ci · 2024-10-04T10:03:36Z

Hi @dmartinol. Thanks for your PR.

I'm waiting for a opendatahub-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lokeshrangineni · 2024-10-04T15:48:19Z

Below check on the PR is currently failing, but it seems to be passing in our local environment. It might be worth re-running the check to determine if it's a legitimate issue.

https://github.com/opendatahub-io/odh-dashboard/actions/runs/11182131290/job/31087701260?pr=3298

faling cypress file as per above check - applications/notebookServer.cy.ts

on local it is successful

andrewballantyne · 2024-10-04T19:17:20Z

/ok-to-test

codecov · 2024-10-04T20:46:03Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.88%. Comparing base (f3cb0a4) to head (a9007d5).
Report is 7 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3298      +/-   ##
==========================================
+ Coverage   85.13%   85.88%   +0.75%     
==========================================
  Files        1338     1338              
  Lines       30118    30123       +5     
  Branches     8269     8270       +1     
==========================================
+ Hits        25641    25872     +231     
+ Misses       4477     4251     -226

see 29 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3cb0a4...a9007d5. Read the comment docs.

andrewballantyne

Some initial feedback. I'll finish up the review on Monday.

frontend/src/__mocks__/mockNimResource.ts

frontend/src/__tests__/cypress/cypress.config.ts

frontend/src/__tests__/cypress/cypress/utils/nimUtils.ts

andrewballantyne

More comments -- I tried to avoid making the same comment multiple times, but I may have missed the mark on that. General themes should be applied to all tests.

frontend/src/__tests__/cypress/cypress/tests/mocked/projects/modelServingNim.cy.ts

frontend/src/__tests__/cypress/cypress/utils/nimUtils.ts

frontend/src/__tests__/cypress/cypress/tests/mocked/projects/modelServingNim.cy.ts

frontend/src/__tests__/cypress/cypress/utils/nimUtils.ts

frontend/src/__tests__/cypress/cypress/tests/mocked/projects/modelServingNim.cy.ts

TomerFi · 2024-10-23T21:10:13Z

@andrewballantyne , shall I rebase to resolve the new conflicts?

Signed-off-by: Tomer Figenblat <[email protected]>

…s standalone cypress test suite.

…#2) Signed-off-by: Daniele Martinoli <[email protected]>

…render nim and all other cards on overview and model tab when there are no ServingRuntimes configured.

Signed-off-by: Daniele Martinoli <[email protected]>

* when combination of configurations to enable nim models are not configured properly.

Signed-off-by: Tomer Figenblat <[email protected]>

…here are deployments and no deployments. (#9) * Moved the utility method to nimUtils.ts

Signed-off-by: Tomer Figenblat <[email protected]>

Signed-off-by: Daniele Martinoli <[email protected]>

Signed-off-by: Tomer Figenblat <[email protected]>

- Use JSON object instead of string in mocking of NIM images configmap - Remove experiementalStudio cypres config param - Add mock for NIM project Co-authored-by: Daniele Martinoli <[email protected]> Co-authored-by: lokeshrangineni <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

- Add nim-serving interception to odh intercept type safety mechanism - Add accelerators interception to odh intercept type safety mechanism Co-authored-by: Daniele Martinoli <[email protected]> Co-authored-by: lokeshrangineni <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

Co-authored-by: Daniele Martinoli <[email protected]> Co-authored-by: lokeshrangineni <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

…odelServingNim.cy.ts Co-authored-by: Andrew Ballantyne <[email protected]>

Signed-off-by: Tomer Figenblat <[email protected]>

frontend/src/__tests__/cypress/cypress/pages/components/NIMDeployModal.ts

andrewballantyne · 2024-10-29T14:14:00Z

frontend/src/__tests__/cypress/cypress/pages/components/NIMDeployModal.ts

+  }
+
+  findNimModelReplicas() {
+    return cy.get('[id="model-server-replicas"]');


Same here -- another gap we already had.

I'll log a bug for this... this is a common component we failed to properly annotate, shouldn't be one you folks have to make use of. Leave as-is.

andrewballantyne · 2024-10-29T14:16:58Z

frontend/src/__tests__/cypress/cypress/pages/components/NIMDeployModal.ts

+  findNimModelReplicasMinusButton() {
+    return this.find().find('button[aria-label="Minus"]').eq(1);
+  }
+
+  findNimModelReplicasPlusButton() {
+    return this.find().find('button[aria-label="Plus"]').eq(1);
+  }


I wonder if these have a chance of conflicting with the StorageSize +/-

Hopefully in the improvement issues I'll log we can make sure this is covered too.

I'll log a bug for this... this is a common component we failed to properly annotate, shouldn't be one you folks have to make use of. Leave as-is.

andrewballantyne · 2024-10-29T14:18:21Z

frontend/src/__tests__/cypress/cypress/pages/modelServing.ts

+  findServingRuntime() {
+    return this.find().find(`[data-label="Serving Runtime"]`);
+  }
+
+  findProject() {
+    return this.find().find(`[data-label=Project]`);
+  }


Although these are not likely to change, they feel like they might be brittle based on how the selectors are written.

No action needed atm, but I wonder if we can't log something to improve these gaps in the future.

andrewballantyne · 2024-10-29T14:26:25Z

frontend/src/__tests__/cypress/cypress/pages/projects.ts

+  findDeployedModelServingRuntime(name: string) {
+    return cy
+      .findByTestId('section-overview')
+      .get('div')
+      .contains(name)
+      .parents('.odh-type-bordered-card .model-server')
+      .get('dd');
+  }


I don't think this will select anything -- I can't find section-overview test id in the code outside of this selector.

Looking for classes and going through divs is definitely a fast-track to brittle tests. If you're trying to select an element specifically on a page, we need a better data-testid on it.

I does selects, the test works:

it('should list the deployed model in Overview tab', () => { initInterceptsToEnableNim({ hasAllModels: false }); cy.interceptK8sList(InferenceServiceModel, mockK8sResourceList([mockNimInferenceService()])); cy.interceptK8sList(ServingRuntimeModel, mockK8sResourceList([mockNimServingRuntime()])); projectDetails.visit('test-project'); // Card is visible projectDetailsOverviewTab .findDeployedModelServingRuntime('Test Name') .should('have.text', 'NVIDIA NIM'); });

I couldn't find a testid on the deployed model card, using the parent based on the calsses was the only way I was able to select the card. Can you suggest something else please?

andrewballantyne · 2024-10-29T14:29:02Z

frontend/src/__tests__/cypress/cypress/pages/projects.ts

+  }
+}
+
+class KserveTableRow extends TableRow {


This feels like it doesn't belong here -- KServe Specific stuff in a projects file? Hmmm... I wonder if this doesn't belong better in the modelServing.ts page component file?

Maybe we named it incorrectly, but we use it to select deployed model records from the models table in the Models tab. See test called should list the deployed model in Models tab.

@andrewballantyne there's a decent amount of model serving related duplication between the projects page tests and the model serving page tests, since we have some of the same functionality in 3 places (project details overview tab, project details models tab, model serving page). It maybe should be cleaned up at the same time as some of the refactoring @lucferbux and @emilys314 have mentioned doing as part of the KServe Raw Deployment work.

frontend/src/__tests__/cypress/cypress/tests/mocked/modelServing/modelServingNim.cy.ts

andrewballantyne · 2024-10-29T15:28:11Z

frontend/src/__tests__/cypress/cypress/tests/mocked/projects/modelServingNim.cy.ts

+      initInterceptsToDeployModel(nimInferenceService);
+
+      projectDetails.visitSection('test-project', 'model-server');
+      cy.get('button[data-testid=deploy-button]').click();


This feels like a gap in the page object -- you shouldn't need selectors in the test itself.

In the project's models tab, when no models are deployed:

If only one type of platform is enabled, i.e. nim for our tests, the one and only deploy button's testid is deploy-button.

If multiple platform are enabled, each button get's a its own testid, for nim it's nim-serving-deploy-button.

Do you want to add a "findNim..", "findSingle...", "findMulti...", "findOnlyOne..."?

Hmmm -- there are probably better ways to handle this. I'd need to take a look at the situation more closely to know what the best call is. Having a page selector for each seems a bit much; especially when only 1 will work at a given moment.

andrewballantyne · 2024-10-29T15:29:14Z

frontend/src/__tests__/cypress/cypress/tests/mocked/projects/modelServingNim.cy.ts

+          disableNIMModelServing: true,
+        });
+        projectDetailsOverviewTab.visit('test-project');
+        cy.findByTestId('model-serving-platform-button').should('not.exist');


The initial part should be in a page object -- you should be doing the .should off page object methods.

Yeah, the reason I did that here, in the comment below (and in a bunch of other places in this file 😃 ), is because both deploy buttons for single-serving and nim-serving have the same testid; in our test cases, only the nim-platform-card is available, so this works.

We can add object funcs like "findNimServingDeployButton" and "findSingleServingDeployButton", but it won't be pretty. Unless we can change the testid of one of the buttons...

The parent => parent div has nvidia-nim-platform-card

andrewballantyne · 2024-10-29T15:29:35Z

frontend/src/__tests__/cypress/cypress/tests/mocked/projects/modelServingNim.cy.ts

+        });
+        projectDetailsOverviewTab.visit('test-project');
+        projectDetailsOverviewTab.findModelServingPlatform('nvidia-nim').should('not.exist');
+        cy.findByTestId('model-serving-platform-button').should('not.exist');


Most definitely if you're repeating the selector.

co-authored-by: Andrew Ballantyne <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

TomerFi · 2024-10-30T16:55:06Z

@andrewballantyne - the changes to the infereceservice mock for supporting custom labels, as we talked about offline are here: 80af75e.

co-authored-by: Andrew Ballantyne <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

andrewballantyne

/hold

Waiting to see if Tomer has something he wants to do here... there are other changes we can do but this is a lingering PR that I would like merged so we can keep building on it.

openshift-ci · 2024-10-30T19:06:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewballantyne

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~frontend/OWNERS~~ [andrewballantyne]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andrewballantyne · 2024-10-30T19:12:27Z

/unhold

openshift-ci bot requested review from alexcreasy and christianvogt October 4, 2024 10:03

openshift-ci bot added the needs-ok-to-test The openshift bot needs to label PRs from non members to avoid strain on the CI label Oct 4, 2024

TomerFi force-pushed the nim-cypress-e2e-tests branch from 8480168 to e8be6e5 Compare October 4, 2024 14:51

openshift-ci bot added ok-to-test The openshift bot needs `ok-to-test` to allow non member PRs to run the tests. and removed needs-ok-to-test The openshift bot needs to label PRs from non members to avoid strain on the CI labels Oct 4, 2024

TomerFi force-pushed the nim-cypress-e2e-tests branch from ff4c204 to b1c2492 Compare October 4, 2024 19:57

andrewballantyne requested changes Oct 4, 2024

View reviewed changes

openshift-ci bot assigned andrewballantyne Oct 4, 2024

lokeshrangineni mentioned this pull request Oct 4, 2024

incorporating code review fix. TomerFi/odh-dashboard#16

Merged

dmartinol mentioned this pull request Oct 7, 2024

Adding mockNimEnabledProject mocking function TomerFi/odh-dashboard#17

Closed

andrewballantyne requested changes Oct 7, 2024

View reviewed changes

TomerFi and others added 13 commits October 29, 2024 10:00

test: added mocks and interceptors for nim cypress tests

82dc694

Signed-off-by: Tomer Figenblat <[email protected]>

Merging the base test cases of Daniel and Lokesh's code to the Tomer'…

777fbf1

…s standalone cypress test suite.

Reverting the mock changes which are not needed.

88fe97b

nim-specific classes and functions, partial rework of deployment test (…

dddb3fc

…#2) Signed-off-by: Daniele Martinoli <[email protected]>

Checking in the cypress test covers to validate all the scenarios to …

683bf47

…render nim and all other cards on overview and model tab when there are no ServingRuntimes configured.

Added mockNimServingRuntimeTemplate mock function

353ae78

Signed-off-by: Daniele Martinoli <[email protected]>

mocking deploy respurces (WIP)

f44825c

Signed-off-by: Daniele Martinoli <[email protected]>

Completed deployment tests

534222b

Signed-off-by: Daniele Martinoli <[email protected]>

Added new test cases to validate if the nim is not enabled

da51470

* when combination of configurations to enable nim models are not configured properly.

test: added test cases for deleting model from the project models tab

1a0b155

Signed-off-by: Tomer Figenblat <[email protected]>

* Added new test cases to validate the model serving menu item when t…

a7826d2

…here are deployments and no deployments. (#9) * Moved the utility method to nimUtils.ts

test: cleanups

5dc4c2c

Signed-off-by: Tomer Figenblat <[email protected]>

adding const for modal dialog title

5a87a50

Signed-off-by: Daniele Martinoli <[email protected]>

TomerFi and others added 13 commits October 29, 2024 10:01

test: fix linting errors

a5d6858

Signed-off-by: Tomer Figenblat <[email protected]>

test: create page objects for verifying nim model tabels

3e2b3c2

Co-authored-by: Daniele Martinoli <[email protected]> Co-authored-by: lokeshrangineni <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

Update frontend/src/__tests__/cypress/cypress/tests/mocked/projects/m…

a955014

…odelServingNim.cy.ts Co-authored-by: Andrew Ballantyne <[email protected]>

test: fix linting errors

0c519b9

Signed-off-by: Tomer Figenblat <[email protected]>

test: rewrite test case for global model list nim

19d20f6

Signed-off-by: Tomer Figenblat <[email protected]>

test: fix linting errors

26195c1

Signed-off-by: Tomer Figenblat <[email protected]>

test: rewrite test case for project overview list nim

98eabbe

Signed-off-by: Tomer Figenblat <[email protected]>

test: rewrite test case for project models tab list nim

abdd860

Signed-off-by: Tomer Figenblat <[email protected]>

test: cleanups

bd9fa01

Signed-off-by: Tomer Figenblat <[email protected]>

test: rewrite test cases nim enablement

b01fe1b

Signed-off-by: Tomer Figenblat <[email protected]>

test: final touchups

06baa5e

Signed-off-by: Tomer Figenblat <[email protected]>

TomerFi force-pushed the nim-cypress-e2e-tests branch from bd71dc4 to 06baa5e Compare October 29, 2024 14:09

test: fixing 1 of 2 failing tests after rebase

9dea313

Signed-off-by: Tomer Figenblat <[email protected]>

andrewballantyne requested changes Oct 29, 2024

View reviewed changes

test: fix mockInfereceService should take labels

80af75e

co-authored-by: Andrew Ballantyne <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

TomerFi force-pushed the nim-cypress-e2e-tests branch from 5684898 to 80af75e Compare October 30, 2024 16:53

TomerFi force-pushed the nim-cypress-e2e-tests branch from db79682 to 80af75e Compare October 30, 2024 17:02

test: fix review suggestion for global model edit button

a9007d5

co-authored-by: Andrew Ballantyne <[email protected]> Signed-off-by: Tomer Figenblat <[email protected]>

TomerFi force-pushed the nim-cypress-e2e-tests branch from 368e747 to a9007d5 Compare October 30, 2024 17:08

andrewballantyne approved these changes Oct 30, 2024

View reviewed changes

openshift-ci bot added do-not-merge/hold This PR is hold for some reason lgtm labels Oct 30, 2024

openshift-ci bot added the approved label Oct 30, 2024

openshift-ci bot removed the do-not-merge/hold This PR is hold for some reason label Oct 30, 2024

openshift-merge-bot bot merged commit 23c82ac into opendatahub-io:main Oct 30, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task/APPENG-2738: NIM Cypress E2E tests #3298

Task/APPENG-2738: NIM Cypress E2E tests #3298

dmartinol commented Oct 4, 2024

openshift-ci bot commented Oct 4, 2024

lokeshrangineni commented Oct 4, 2024 •

edited

Loading

andrewballantyne commented Oct 4, 2024

codecov bot commented Oct 4, 2024 •

edited

Loading

andrewballantyne left a comment

andrewballantyne left a comment

TomerFi commented Oct 23, 2024

andrewballantyne Oct 29, 2024

andrewballantyne Oct 30, 2024

andrewballantyne Oct 29, 2024

andrewballantyne Oct 30, 2024

andrewballantyne Oct 29, 2024 •

edited

Loading

andrewballantyne Oct 29, 2024

TomerFi Oct 30, 2024

andrewballantyne Oct 29, 2024

TomerFi Oct 30, 2024

mturley Oct 30, 2024

andrewballantyne Oct 29, 2024

TomerFi Oct 30, 2024

andrewballantyne Oct 30, 2024 •

edited

Loading

andrewballantyne Oct 29, 2024

TomerFi Oct 30, 2024 •

edited

Loading

andrewballantyne Oct 30, 2024

andrewballantyne Oct 29, 2024

TomerFi commented Oct 30, 2024

andrewballantyne left a comment

openshift-ci bot commented Oct 30, 2024

andrewballantyne commented Oct 30, 2024

Task/APPENG-2738: NIM Cypress E2E tests #3298

Task/APPENG-2738: NIM Cypress E2E tests #3298

Conversation

dmartinol commented Oct 4, 2024

Description

How Has This Been Tested?

Test Impact

Request review criteria:

openshift-ci bot commented Oct 4, 2024

lokeshrangineni commented Oct 4, 2024 • edited Loading

andrewballantyne commented Oct 4, 2024

codecov bot commented Oct 4, 2024 • edited Loading

Codecov Report

andrewballantyne left a comment

Choose a reason for hiding this comment

andrewballantyne left a comment

Choose a reason for hiding this comment

TomerFi commented Oct 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewballantyne Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewballantyne Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomerFi Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomerFi commented Oct 30, 2024

andrewballantyne left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Oct 30, 2024

andrewballantyne commented Oct 30, 2024

lokeshrangineni commented Oct 4, 2024 •

edited

Loading

codecov bot commented Oct 4, 2024 •

edited

Loading

andrewballantyne Oct 29, 2024 •

edited

Loading

andrewballantyne Oct 30, 2024 •

edited

Loading

TomerFi Oct 30, 2024 •

edited

Loading