aws-crypto 3.0.0 dependency causes breakage with npm 9 pack

[PaulCarlucciStratascale]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

npm 9 fixed a bug/behavior regarding globbing for what goes into npm pack. Unfortunately there are a few things out there that depended on that bug/behavior in order to continue working, one such example is @aws-sdk/util-utf8-browser. As a result attempting to run a package built with npm 9 results in errors such as

Error: Cannot find module '/home/node/package/node_modules/@aws-sdk/util-utf8-browser/dist-cjs/index.js'. Please verify that the package.json has a valid "main" entry

The resolution is to use aws-crypto 5.1.0 which will pull in @smithy/util-utf8 instead of @aws-sdk/util-utf8-browser where it is fixed. Unfortunately aws-sdk-js-v3 is still pulls in aws-crypto 3.0.0 and as a result we have to run the following override in package.json as a workaround in order to build a working pacakge against npm 9.x

  "overrides": {
    "@aws-sdk/util-utf8-browser": {
      "@smithy/util-utf8": "^2.0.0"
    }
  },

Hopefully the above is sufficient to help other searching for how to deal with this.

SDK version number

@aws-sdk/client-sesv2 3.423.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node v18.18

Reproduction Steps

Run npm pack with npm 9.x series when @aws-sdk/util-utf8-browser is included as a dependency. Then untar that package into a clean runtime environment such as a container and attempt to run. It will fail with the error:

Error: Cannot find module '/home/node/package/node_modules/@aws-sdk/util-utf8-browser/dist-cjs/index.js'. Please verify that the package.json has a valid "main" entry

Observed Behavior

See above

Expected Behavior

Run as proper

Possible Solution

Bump from @aws-crypto 3.0.0 to @aws-crypto 5.1.0 or above.

Additional Information/Context

No response

[kuhe]

Please continue to use the workaround, since we cannot upgrade to 5.1.0 yet due to the AWS SDKs and Tools maintenance policy.

Specifically, the SDK supports end-of-life runtimes (Node.js 14.x) for at least 6 months after announcing our end of support, which hasn't happened yet.

[jkelley-godaddy]

Hey @kuhe, our use case is similar to this issue on the jsii repository where we use jsii and jsii-pacmak to package a module (with executable commands) for use in multiple target languages. jsii-pacmak uses bundleDependencies to bundle node packages so executable commands can be used even when trans-piled to other languages.

As described in the linked issue above, due to jsii-pacmak using npm pack under the hood, any node version that uses NPM 9+ creates a package without required files due to the breaking changes with npm pack, which results in this Cannot find module error.

We cannot use overrides as a work-around as these do not affect npm pack in any way, and the resulting package for another language (ie. python) has the issue with missing files.

For now we are using a deprecated version of Node (16.20.2) that uses NPM 8 as a work-around, as it appears the only resolution for NPM versions 9+ will be when the aws-sdk dependencies are updated upstream.

Specifically, the SDK supports end-of-life runtimes (Node.js 14.x) for at least 6 months after announcing our end of support, which hasn't happened yet.

I was unable to find any announcement on the AWS SDK Blog for Node.js 14.x. The AWS SDKs and Tools maintenance policy says the following:

Language Runtime: Examples include Java 7, Java 8, Java 11, .NET Core, .NET Standard, .NET PCL, etc.
Our policy is to continue supporting SDK dependencies for at least 6 months after the community or vendor ends support for the dependency. This policy, however, could vary depending on the specific dependency.

It appears that official Security Support for Node.js 14 ended Ended 5 months and 3 weeks ago(30 Apr 2023).

Does this mean that the aws-sdk library would be eligible for update 30 Oct 2023 or is this change is waiting on a different (unannounced?) date from AWS?

If it's the second case, would it be possible to create a separate "beta" or "preview" version of the @aws-sdk/client-sts package to unblock usage of this package with currently supported LTS Node runtimes (that use NPM 9+), while not making breaking changes to older versions?

Please let us know if there is anything we can clarify further, and thank you for your assistance!

[kuhe]

Our Node.js 14.x end of support date is not announced yet. The date will be on or after May 1, 2024.

For a fix, could to make a request to the aws-crypto owners to release a version that is compatible with Node.js 14 but also compatible with NPM 9? https://github.com/aws/aws-sdk-js-crypto-helpers/issues

It could be a branched patch of the 3.0.0 version with the dependency updated.

[jkelley-godaddy]

Thank you for the clarification on the end of support date.

As requested I've submitted a new issue to @aws-crypto for a version with both Node.js 14.x and NPM 9+ support: aws/aws-sdk-js-crypto-helpers#761

If the aws-crypto team is unable to resolve the issue would it be possible to publish a branch version of the @aws-sdk/client-sts package ie. 3.433.0b or 3.433.0-npm9 based on 3.433.0 except with the package.json dependencies for @aws-crypto set to latest ("*") ?

[kuhe]

working on updating aws-crypto now that we dropped Node.js 14 support last month

[kuhe]

expected in https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.598.0 later today

[aBurmeseDev]

Dependencies were updated in https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.598.0. Thank you again for reaching out here and let us know if you have any other SDK related questions! Closing issue for now.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.