Let me specify the parallelism, executor and more on the jobs

[Peter-Darton-i2]

Describe Request:

The node orb provides two really useful jobs ... but they could really do with some extension hooks.

1. Parallelism
   One of CircleCI's killer features is that it's easy to split large workloads across multiple build agents.
   You do this by specifying a parallelism: <number> field in the job.
   ... but neither the run nor the test jobs support this - there is no "pass through" parameter for it.
2. Executor
   While the orb's default executor is good, it's not suitable for everything.
   e.g. I recently had to implement a job that ran "playwright" tests and not even the lts-browsers tag came with everything it needed pre-installed. I needed to use the official playwright docker image instead ... but this orb's jobs do not allow for that kind of flexibility.
3. Multiple npm run commands
   It'd be lovely if I could tell it to "do rpm run this, then npm run that" just through parameters ... although that could also be achieved via the following...
4. Add an "after node/install-packages has run" set of steps as an optional parameter.
   That way folks could add "and cache this too" functionality by having a restore_cache in the "steps" parameter and a save_cache in this new "after the packages have been installed" parameter.

Examples:

I wanted to use playwright's --shard functionality to split tests between multiple agents.
... so I had to copy/paste this orb's test job and edit it:

  node-test:
    description: |
      Copy of https://github.com/CircleCI-Public/node-orb/blob/master/src/jobs/test.yml
      with added parallelism and executor parameter and resource_class and version removed.
    executor: << parameters.executor >>
    parallelism: << parameters.parallelism >>
    parameters:
      parallelism:
        type: integer
        default: 1
      executor:
        type: executor
...

Supporting Documentation Links:

Example of using playwright, and having to have a copy/pasted version of the test job: https://app.circleci.com/pipelines/github/i2group/ias-client/386/workflows/ab3cbacf-e8f3-4d3b-83ac-d8296c6fb413/jobs/1666

[marboledacci]

I don't understand very well the 3rd and 4th point. How different would this be to use the post and pre steps of a job?

[Peter-Darton-i2]

Re: 4th point

This would happen between the "install" and the "run-test" stages.
A pre-step step happens before the "install".
A post-step step happens after the "run-test".

Having looked at the code, I think doing (3) would be really hard to do "properly" across all modes of operation (yarn vs npm etc) ... whereas (4) would be "just" a new parameter of type "steps" ... and is the option I went for locally.

FYI this is "the kind of thing I was thinking of" - search for "i2group" in this text to see where the changes are.

# i2group extension of the official CircleCI node orb's test job,
# https://github.com/CircleCI-Public/node-orb/blob/master/src/jobs/test.yml
#
# This is a copy/paste but with added parallelism and executor parameter,
# added post-install-steps parameter, and with the syntax errors fixed
# ... and then all the kebab-case changed to snake_case.

description: |
    Simple drop-in job to test your Node.js application automatically.

parallelism: << parameters.parallelism >> # i2group: added

parameters:
    parallelism: # i2group: added
        type: integer
        default: 1
        description: The number of parallel jobs to run. See https://circleci.com/docs/parallelism-faster-jobs/
    executor: # i2group: added
        type: executor
        default: ccinode/default
        description: The executor to use for this job. e.g. the node orb's default executor.
    pkg_manager:
        type: enum
        enum: ['npm', 'yarn', 'yarn-berry', 'pnpm']
        default: 'npm'
        description: Select the default node package manager to use.
    cache_version:
        type: string
        default: v1
        description: Change the default cache version if you need to clear the cache for any reason.
    setup:
        type: steps
        description: Provide any optional steps you would like to run prior to installing the node dependencies. This is a good place to install global modules.
        default: []
    post_install_steps: # i2group: added
        type: steps
        description: Provide any optional steps you would like to run after installing the node dependencies.
        default: []
    run_command:
        type: string
        default: 'test'
        description: The name of the script within your package.json which will run your tests.
    app_dir:
        type: string
        default: '~/project'
        description: Path to the directory containing your package.json file. Not needed if package.json lives in the root.
    override_ci_command:
        description: |
            By default, packages will be installed with "npm ci" or "yarn install --frozen-lockfile".
            Optionally supply a custom package installation command, with any additional flags needed.
        type: string
        default: ''
    test_results_path:
        type: string
        default: ''
        description: |
            If you use test_results_for, this will be configured automatically.
            If you configure test results yourself (for example in your node config files) set this to the location that you write them to, and they will be uploaded.
    test_results_for:
        default: other
        type: enum
        enum: ['jest', 'mocha', 'other']
        description: |
            Testing framework your project uses.
            If this is set to jest or mocha, test results will be automatically produced. When using jest, the jest-junit package is required as a dev dependency. See https://github.com/jest-community/jest-junit for more information.
            When using mocha, the mocha-junit-reporter and mocha-multi packages are required as dev dependency. See https://github.com/michaelleeallen/mocha-junit-reporter and https://github.com/glenjamin/mocha-multi for more information.
    test_coverage_path:
        type: string
        default: ''
        description: |
            Set this to the location where code coverage files will be placed after running tests.
            The code coverage files will be uploaded to artifacts for this workflow.
    # i2group: removed resource_class parameter - use executor parameter instead
    include_branch_in_cache_key:
        type: boolean
        default: false # i2group: changed from true to false
        description: >
            If true, this cache bucket will only apply to jobs within the same branch.
    cache_path:
        description: |
            By default, this orb will utilize 'npm ci' and cache the '~/.npm' directory. Override which path to cache with this parameter.
            The cache will be ignored when using npm ci, as the command doesn't benefit from cache.
        type: string
        default: ''
    cache_only_lockfile:
        type: boolean
        default: true
        description: |
            If true, package.json will be ignored in the cache key. Useful for projects where package.json changes do not always invalidate dependencies.
            Note: package.json will still be the final fallback key incase a project is not configured with a lock file.
    with_cache:
        type: boolean
        default: true
        description: |
            Cache your node packages automatically for faster install times.
            Cache will be ignored when using npm ci.
    check_cache:
        type: enum
        default: 'never'
        enum: ['never', 'always', 'detect']
        description: |
            Yarn berry only for Zero install support -
            Use 'always' to always --check-cache argument to yarn install.
            Use 'detect' to enable caching of yarn.lock and to only add when required.
executor: << parameters.executor >> # i2group: no longer restricted to the node orb's default executor

steps:
    - checkout
    - steps: << parameters.setup >>
    - ccinode/install-packages:
          app-dir: <<parameters.app_dir>>
          pkg-manager: <<parameters.pkg_manager>>
          cache-version: <<parameters.cache_version>>
          override-ci-command: <<parameters.override_ci_command>>
          include-branch-in-cache-key: <<parameters.include_branch_in_cache_key>>
          cache-path: <<parameters.cache_path>>
          cache-only-lockfile: <<parameters.cache_only_lockfile>>
          with-cache: <<parameters.with_cache>>
          check-cache: <<parameters.check_cache>>
    - steps: << parameters.post_install_steps >> # i2group: added
    - when: # Run tests for NPM, without test results
          condition:
              and:
                  - equal: [npm, << parameters.pkg_manager >>]
                  - equal: [other, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Run NPM Tests
                    working_directory: <<parameters.app_dir>>
                    command: npm run <<parameters.run_command>>
    - when: # Run tests for NPM, with jest test results
          condition:
              and:
                  - equal: [npm, << parameters.pkg_manager >>]
                  - equal: [jest, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Check for test reporter
                    working_directory: <<parameters.app_dir>>
                    command: npm list jest-junit || (echo "Add the package jest-junit to the dev dependencies of your project with `npm install --save-dev jest-junit`" && exit 1)
              - run:
                    name: Run NPM Tests
                    working_directory: <<parameters.app_dir>>
                    command: npm run <<parameters.run_command>> -- --reporters=default --reporters=jest-junit
    - when: # Run tests for NPM, with mocha test results
          condition:
              and:
                  - equal: [npm, << parameters.pkg_manager >>]
                  - equal: [mocha, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Check for test reporter
                    working_directory: <<parameters.app_dir>>
                    command: npm list mocha-junit-reporter mocha-multi || (echo "Add the packages mocha-junit-reporter & mocha-multi to the dev dependencies of your project with `npm install --save-dev mocha-junit-reporter mocha-multi`" && exit 1)
              - run:
                    name: Run NPM Tests
                    working_directory: <<parameters.app_dir>>
                    command: npm run <<parameters.run_command>> -- --reporter mocha-multi --reporter-options spec=-,mocha-junit-reporter=-
    - when: # Run tests for YARN, without test results
          condition:
              and:
                  - equal: [yarn, << parameters.pkg_manager >>]
                  - equal: [other, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Run YARN Tests
                    working_directory: <<parameters.app_dir>>
                    command: yarn run <<parameters.run_command>>
    - when: # Run tests for YARN, with jest test results
          condition:
              and:
                  - equal: [yarn, << parameters.pkg_manager >>]
                  - equal: [jest, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Check for test reporter
                    working_directory: <<parameters.app_dir>>
                    command: yarn list --pattern jest-junit | grep " jest-junit@" || (echo "Add the package jest-junit to your projects dev dependencies with `yarn add --dev jest-junit`" && exit 1)
              - run:
                    name: Run YARN Tests
                    working_directory: <<parameters.app_dir>>
                    command: yarn run <<parameters.run_command>> --reporters=default --reporters=jest-junit
    - when: # Run tests for YARN, with mocha test results
          condition:
              and:
                  - equal: [yarn, << parameters.pkg_manager >>]
                  - equal: [mocha, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Check for test reporter
                    working_directory: <<parameters.app_dir>>
                    command: yarn list --pattern "mocha-junit-reporter|mocha-multi" | grep "mocha-junit-reporter@\|mocha-multi@" || (echo "Add the packages mocha-junit-reporter & mocha-multi to the dev dependencies of your project with `yarn add --dev mocha-junit-reporter mocha-multi`" && exit 1)
              - run:
                    name: Run YARN Tests
                    working_directory: <<parameters.app_dir>>
                    command: yarn run <<parameters.run_command>> --reporter mocha-multi --reporter-options spec=-,mocha-junit-reporter=-
    - when: # Run tests for YARN 2.x, without test results
          condition:
              and:
                  - equal: [yarn-berry, << parameters.pkg_manager >>]
                  - equal: [other, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Run YARN 2.x Tests
                    working_directory: <<parameters.app_dir>>
                    command: yarn run <<parameters.run_command>>
    - when: # Run tests for YARN 2.x, with jest test results
          condition:
              and:
                  - equal: [yarn-berry, << parameters.pkg_manager >>]
                  - equal: [jest, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Run YARN 2.x Tests
                    working_directory: <<parameters.app_dir>>
                    command: yarn run <<parameters.run_command>> --reporters=default --reporters=jest-junit
    - when: # Run tests for YARN 2.x, with mocha test results
          condition:
              and:
                  - equal: [yarn-berry, << parameters.pkg_manager >>]
                  - equal: [mocha, << parameters.test_results_for >>]
          steps:
              - run:
                    name: Check for test reporter
                    working_directory: <<parameters.app_dir>>
                    command: yarn list --pattern "mocha-junit-reporter|mocha-multi" | grep "mocha-junit-reporter@\|mocha-multi@" || (echo "Add the packages mocha-junit-reporter & mocha-multi to the dev dependencies of your project with `yarn add --dev mocha-junit-reporter mocha-multi`" && exit 1)
              - run:
                    name: Run YARN 2.x Tests
                    working_directory: <<parameters.app_dir>>
                    command: yarn run <<parameters.run_command>> --reporter mocha-multi --reporter-options spec=-,mocha-junit-reporter=-
    - when: # Run tests for pnpm, without test results
        condition:
          and:
            - equal: [ pnpm, << parameters.pkg_manager >> ]
            - equal: [ other, << parameters.test_results_for >> ]
        steps:
          - run:
              name: Run pnpm Tests
              working_directory: <<parameters.app_dir>>
              command: pnpm run <<parameters.run_command>>
    - when: # Run tests for pnpm, with jest test results
        condition:
          and:
            - equal: [ pnpm, << parameters.pkg_manager >> ]
            - equal: [ jest, << parameters.test_results_for >> ]
        steps:
          - run:
              name: Check for test reporter
              working_directory: <<parameters.app_dir>>
              command: pnpm list "jest-junit@*" | grep "jest-junit" || (echo "Add the package jest-junit to your projects dev dependencies with `pnpm add -D jest-junit`" && exit 1)
          - run:
              name: Run pnpm Tests
              working_directory: <<parameters.app_dir>>
              command: pnpm run <<parameters.run_command>> --reporters=default --reporters=jest-junit
    - when: # Run tests for pnpm, with mocha test results
        condition:
          and:
            - equal: [ pnpm, << parameters.pkg_manager >> ]
            - equal: [ mocha, << parameters.test_results_for >> ]
        steps:
          - run:
              name: Check for test reporter
              working_directory: <<parameters.app_dir>>
              command: pnpm list "mocha-junit-reporter@*" "mocha-multi@*" | grep "mocha-junit-reporter\|mocha-multi" || (echo "Add the packages mocha-junit-reporter & mocha-multi to the dev dependencies of your project with `pnpm add -D mocha-junit-reporter mocha-multi`" && exit 1)
          - run:
              name: Run pnpm Tests
              working_directory: <<parameters.app_dir>>
              command: pnpm run <<parameters.run_command>> --reporter mocha-multi --reporter-options spec=-,mocha-junit-reporter=-
    - when: # upload test results if a path was given
          condition: << parameters.test_results_path >>
          steps:
              # i2group: Original was syntactically invalid; was missing the - before store_test_results
              - store_test_results:
                  path: << parameters.test_results_path >>
    - when: # upload test results if jest was automatically configured and test_results_path is not specified
          condition:
              and:
                  - equal: [jest, << parameters.test_results_for >>]
                  - equal: ['', << parameters.test_results_path >>]
          steps:
              # i2group: Original was syntactically invalid; was missing the - before store_test_results
              - store_test_results:
                  path: <<parameters.app_dir>>/junit.xml
    - when: # upload test results if mocha was automatically configured and test_results_path is not specified
          condition:
              and:
                  - equal: [mocha, << parameters.test_results_for >>]
                  - equal: ['', << parameters.test_results_path >>]
          steps:
              # i2group: Original was syntactically invalid; was missing the - before store_test_results
              - store_test_results:
                  path: <<parameters.app_dir>>/test-results.xml
    - when: # upload coverage files if test_coverage_path was configured
          condition: <<parameters.test_coverage_path>>
          steps:
              - store_artifacts:
                    path: <<parameters.test_coverage_path>>