feat(e2e): Smoke tests task & initial entrypoint COMPASS-8531 #6571

lerouxb · 2024-12-17T13:36:11Z

This is also documented by yargs and in comments, but for convenience I'm adding a summary here.

For running locally and testing with compass you packaged yourself:

HADRON_DISTRIBUTION=compass IS_OSX=true npm run smoketest -w compass-e2e-tests -- --skip-download

For running locally and testing against compass that was packaged by evergreen for a pull request:

# DEV_VERSION_IDENTIFIER should be printed out by the package task and in the
# format yy-mm-dd-dev.hms (leave it out for testing with the version specified
# in packages/compass/package.json, ie. for locally packaged or beta or stable)
export DEV_VERSION_IDENTIFIER=24.12.14-dev.5046

# EVERGREEN_BUCKET_NAME is currently always the same
export EVERGREEN_BUCKET_NAME=mciuploads

# EVERGREEN_BUCKET_KEY_PREFIX should be printed out by the package task and is
# in the format ${project}/${revision}_${revision_order_id}
export EVERGREEN_BUCKET_KEY_PREFIX=10gen-compass-main/4754c3d04086f2473560d1d8b14f7d83f68e1017_nikola.irinchev_586

# HADRON_DISTRIBUTION can be compass, compass-readonly or compass-isolated
export HADRON_DISTRIBUTION=compass

# ARCH will pick up platform.arch if not set. Useful for setting it to something
# else if you want to download/test different packages
export ARCH=arm64

# IS_OSX could also be IS_UBUNTU, IS_RHEL or IS_WINDOWS. That's to mirror our
# existing variables used in evergreen. You have to specify one of them
export IS_OSX=true 

npm run smoketest -w compass-e2e-tests

lerouxb · 2024-12-17T13:36:44Z

.evergreen/buildvariants-and-tasks.in.yml

+    depends_on: 'package-ubuntu',
+  },
+
+//  {


For now I have most platforms commented out because this is a lot of new tasks that will do nothing for the moment.

lerouxb · 2024-12-17T13:37:20Z

.evergreen/buildvariants-and-tasks.in.yml

@@ -180,6 +214,19 @@ buildvariants:
    <% } %>
 <% } %>

+<% for (const buildVariant of SMOKETEST_BUILD_VARIANTS) { %>
+<% for (const distribution of ['compass']) { %>


Looping through just compass here rather than all of COMPASS_DISTRIBUTIONS. For now. Until we want to test readonly and isolated.

lerouxb · 2024-12-17T13:37:34Z

.evergreen/buildvariants-and-tasks.in.yml

@@ -424,6 +471,22 @@ tasks:
      - func: save-all-artifacts
        vars:
          compass_distribution: <%= distribution %>
+  <% } %>
+
+<% for (const distribution of ['compass']) { %>


Also not looping through COMPASS_DISTRIBUTIONS here yet.

lerouxb · 2024-12-17T13:38:57Z

.evergreen/preinstall.sh

@@ -12,6 +12,11 @@ echo "NPM_VERSION: $NPM_VERSION"
 echo "APPDATA: $APPDATA"
 echo "PATH: $PATH"

+# these are super useful if you want to run the smoke tests locally


We're going to need these when we want to test with the files produced by a PR or beta or ga build, so I'm just adding it next to other useful/essential variables being output. They aren't secrets so that's fine.

lerouxb · 2024-12-17T13:40:09Z

packages/compass-e2e-tests/smoke-test.ts

+      }
+    }
+
+    if (!(argv.isWindows || argv.isOSX || argv.isUbuntu || argv.isRHEL)) {


We could detect these as a default, but I'm not sure we want to duplicate the logic?

I spent way too much time thinking about this and going back and forth.

Should we also check to ensure these are mutually exclusive?

I've been thinking about that. Either make them mutually exclusive or allow you to "OR" them together. Then you can use this to download everything if you want to or at least test multiple .zip files at a time. Sure it will break later, but much of the code could run on mac for testing.

Actually I just tested this and it won't work because hadron-build info can only do one at a time.

I just count them now as a poor man's logical XOR.

lerouxb · 2024-12-17T13:40:41Z

packages/compass-e2e-tests/tsconfig.json

@@ -1,7 +1,4 @@
 {
  "extends": "@mongodb-js/tsconfig-compass/tsconfig.common.json",
-  "compilerOptions": {


Driveby from uncommitted changes.

lerouxb · 2024-12-17T13:42:29Z

packages/hadron-build/commands/download.js

+
+const handler = function handler(argv) {
+  cli.argv = argv;
+  run(argv).catch(abortIfError);


In the end I'm not using download because as is it would download all the packages and therefore mean we have to depend on every package task rather than just the relevant one for the platform.

I could modify download more to filter but we already have the info we need from hadron-build info anyway so I just download straight in the script.

But I think the change here is harmless and a very minor improvement anyway.

lerouxb · 2024-12-17T13:43:00Z

packages/compass-e2e-tests/smoke-test.ts

+    verifyPackagesExist(packages);
+  }
+
+  // TODO(COMPASS-8533): extract or install each package and then test the Compass binary


The rest of the epic goes here :)

lerouxb · 2024-12-17T13:45:13Z

packages/compass-e2e-tests/smoke-test.ts

+  const extension = config.extension;
+
+  return names
+    .filter((name) => !extension || name.endsWith(extension))


By optionally filtering the files we'll test we can easily run this on just one file at a time during development and we can also change the evergreen config to add one task per file if we wanted to at some point.

kraenhansen

I've got a few minor suggestions, but I like the approach.

The only bigger thing is: I have a feeling "smoke test" is too generic for what we're doing with this project. I mean, this will specifically test "installs" and "updating", right?

kraenhansen · 2024-12-18T12:05:59Z

packages/compass-e2e-tests/smoke-test.ts

+        const fileWriter = createWriteStream(targetFile).on('finish', () => {
+          resolve();
+        });


Perhaps you'd want to register an 'error' listener here as well (calling reject), in case the write stream fails?

I suppose that won't hurt. I find streams such a footgun. There really should be builtin utilities somewhere for these cases. We use the download npm package in a few other places but it hasn't been published to in 5 years, supports no ESM, no TypeScript.. with no obvious perfect alternative I could quickly find.

packages/compass-e2e-tests/smoke-test.ts

kraenhansen · 2024-12-18T12:10:44Z

packages/compass-e2e-tests/smoke-test.ts

+      }
+    }
+
+    if (!(argv.isWindows || argv.isOSX || argv.isUbuntu || argv.isRHEL)) {


Should we also check to ensure these are mutually exclusive?

packages/compass-e2e-tests/smoke-test.ts

lerouxb added 8 commits December 11, 2024 17:24

WIP

b37ec80

wip

77fb9aa

WIP

db7d752

WIP

a52b299

WIP

1e72a54

wip

92bdb85

initial smoketest task & entrypoint

fcb5ad6

limit to just ubuntu and just the compass distribution for now

c8f0a6b

lerouxb added the no release notes Fix or feature not for release notes label Dec 17, 2024

github-actions bot added the feat label Dec 17, 2024

lerouxb commented Dec 17, 2024

View reviewed changes

lerouxb added 3 commits December 17, 2024 14:19

correct package

24b4665

fine. no shebang

a712bfc

Merge branch 'main' into smoke-tests-task

fd6396d

kraenhansen approved these changes Dec 18, 2024

View reviewed changes

lerouxb added 3 commits December 18, 2024 12:36

xor the IS_* platform options

2badcfd

verify once

17f4152

add error callback

47e2816

lerouxb merged commit 31cf841 into main Dec 18, 2024
25 of 28 checks passed

lerouxb deleted the smoke-tests-task branch December 18, 2024 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(e2e): Smoke tests task & initial entrypoint COMPASS-8531 #6571

feat(e2e): Smoke tests task & initial entrypoint COMPASS-8531 #6571

lerouxb commented Dec 17, 2024 •

edited

Loading

lerouxb Dec 17, 2024

lerouxb Dec 17, 2024

lerouxb Dec 17, 2024

lerouxb Dec 17, 2024

lerouxb Dec 17, 2024

kraenhansen Dec 18, 2024

lerouxb Dec 18, 2024

lerouxb Dec 18, 2024

lerouxb Dec 18, 2024

lerouxb Dec 17, 2024

lerouxb Dec 17, 2024

lerouxb Dec 17, 2024

lerouxb Dec 17, 2024

kraenhansen left a comment

kraenhansen Dec 18, 2024

lerouxb Dec 18, 2024

kraenhansen Dec 18, 2024

feat(e2e): Smoke tests task & initial entrypoint COMPASS-8531 #6571

feat(e2e): Smoke tests task & initial entrypoint COMPASS-8531 #6571

Conversation

lerouxb commented Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kraenhansen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lerouxb commented Dec 17, 2024 •

edited

Loading