Schematics Migrations after update to 2211.19 are not executed

[janwidmer]

Describe the bug
When Updating an existing Spartacus Project from Version 6.8.0 from a selfhosted Spartacus Package Registry by following the update instructions on https://github.com/SAP/spartacus/blob/develop/docs/self-publishing-spartacus-libraries.md, the only changes are done in the file package.json.

No schematics update migrations are executed.

When updating from the official SAP Private Repository, the schematics work as expected.

Tell us the version of Spartacus

• Library version: 2211.19

To Reproduce
Steps to reproduce the behavior:

Create Spartacus in version 6.8.0
Upgrade it to version 2211.19.0 (actual current spartacus version when building the lib manually)

Expected behavior
The ng update migrations schould be run

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Mac OS Sonoma, 14.2.1

Additional context
Running the command ng update @spartacus/[email protected] / ng update @spartacus/[email protected] it just stops after updating the package.json:

According to this article it says:

"The command ng update @my/lib@version downloads the specified version of the library and launches all schematics listed in its migrations.json that have an appropriate property of "version" within the semver range from >installedVersion to <=targetVersion."

• installedVersion: 6.8.0
• targetVersion: 2211.19.0
• migrations version: 2211.19.0

[timdk]

Do you have a trailing .0 that doesn't need to be there? The command in the documentation is

ng update @spartacus/[email protected]

I have a separate issue that the migration fails due to a missing file under node_modules directory

~ ng update @spartacus/[email protected]

Using package manager: npm
Collecting installed dependencies...
Found 137 dependencies.
Fetching dependency metadata from registry...
                  Package "@spartacus/styles" has a missing peer dependency of "@fontsource/open-sans" @ "^4.5.14".
    Updating package.json with dependency @spartacus/schematics @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/setup @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/cart @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/checkout @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/core @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/order @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/pdf-invoices @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/smartedit @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/storefront @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/styles @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/tracking @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency @spartacus/user @ "2211.19.0" (was "6.8.0")...
    Updating package.json with dependency angular-oauth2-oidc @ "17.0.1" (was "15.0.1")...
    Updating package.json with dependency i18next @ "23.10.0" (was "21.10.0")...
    Updating package.json with dependency i18next-http-backend @ "2.5.0" (was "1.4.5")...
    Updating package.json with dependency parse5 @ "7.1.2" (was "6.0.1")...
    Updating package.json with dependency rxjs @ "7.8.1" (was "6.6.7")...
UPDATE package.json (7326 bytes)
✔ Packages successfully installed.
** Executing migrations of package '@spartacus/schematics' **

❯ Updates the SSR setup.
DELETE server.ts.bak
DELETE src/express.tokens.ts
UPDATE server.ts (16747 bytes)
UPDATE src/app/serverRequestProviders.ts (1299 bytes)
UPDATE src/app/debugging/debug.interceptor.ts (1919 bytes)
UPDATE src/app/spartacus/services/default-language.server.service.ts (493 bytes)
  Migration completed (6 files modified).

❯ Handle deprecated configuration properties.
  Migration completed (No changes made).

❯ Add or remove constructor parameters.
    Checking constructor deprecations...
  Migration completed (No changes made).

❯ Handle deprecated CSS.
    For a CSS migration guide, please visit this URL: https://help.sap.com/docs/SAP_COMMERCE_COMPOSABLE_STOREFRONT. Under the Installation and Upgrade header, select “Updating Composable Storefront” for version 2211, and then open “Changes to Styles in Composable Storefront 2211.
  Migration completed (No changes made).

❯ Update dependencies.
    🩹 Upgrading 'bootstrap' to ^4.6.2 (was 4.6.1)
    🩹 Upgrading 'i18next-resources-to-backend' to ^1.2.0 (was 1.1.4)
    ✅️ Added '@fontsource/open-sans' into dependencies
UPDATE package.json (7366 bytes)
✖ Migration failed: Cannot find module '/Users/xx/dev/yy/storefront-upgrade/js-storefront/zz/node_modules/@angular-devkit/schematics/node_modules/rxjs/dist/cjs/index.js'
  See "/private/var/folders/p6/cstqq4hn0fx88_mrp3h2cdmw0000gn/T/ng-0Nwm1R/angular-errors.log" for further details.

The directory node_modules/@angular-devkit/schematics/node_modules/rxjs is deleted when packages are installed after updating to rxjs v7.8.1 in the first step.

If I upgrade rxjs to 7.8.1 prior to running the migration, it isn't listed in the package.json upgrades but it still fails on the same step.

Full package.json dependencies and devDependencies are as follows

  "dependencies": {
    "@angular/animations": "17.2.2",
    "@angular/common": "17.2.2",
    "@angular/compiler": "17.2.2",
    "@angular/core": "17.2.2",
    "@angular/forms": "17.2.2",
    "@angular/localize": "17.2.2",
    "@angular/platform-browser": "17.2.2",
    "@angular/platform-browser-dynamic": "17.2.2",
    "@angular/platform-server": "17.2.2",
    "@angular/router": "17.2.2",
    "@angular/service-worker": "17.2.2",
    "@angular/ssr": "^17.2.1",
    "@fortawesome/angular-fontawesome": "~0.14.0",
    "@fortawesome/fontawesome-free": "~6.5.1",
    "@fortawesome/fontawesome-svg-core": "~6.5.1",
    "@fortawesome/free-brands-svg-icons": "~6.5.1",
    "@fortawesome/free-regular-svg-icons": "~6.5.1",
    "@fortawesome/free-solid-svg-icons": "~6.5.1",
    "@ng-bootstrap/ng-bootstrap": "^16.0.0",
    "@ng-select/ng-select": "^12.0.7",
    "@ngrx/effects": "^17.1.1",
    "@ngrx/router-store": "^17.1.1",
    "@ngrx/store": "^17.1.1",
    "@popperjs/core": "^2.11.8",
    "@rx-angular/cdk": "^1.0.0",
    "@rx-angular/template": "^1.0.0",
    "@schematics/angular": "^17.2.1",
    "@spartacus/cart": "~6.8.0",
    "@spartacus/checkout": "~6.8.0",
    "@spartacus/core": "~6.8.0",
    "@spartacus/order": "~6.8.0",
    "@spartacus/pdf-invoices": "~6.8.0",
    "@spartacus/smartedit": "~6.8.0",
    "@spartacus/storefront": "~6.8.0",
    "@spartacus/styles": "~6.8.0",
    "@spartacus/tracking": "~6.8.0",
    "@spartacus/user": "~6.8.0",
    "angular-csv-ext": "^1.0.5",
    "angular-oauth2-oidc": "^15.0.1",
    "angular-responsive-carousel3": "1.0.0",
    "bootstrap": "^4.6.1",
    "cookie-parser": "^1.4.6",
    "express": "^4.15.2",
    "express-basic-auth": "^1.2.1",
    "file-saver": "^2.0.5",
    "helmet": "^5.1.1",
    "http-proxy-middleware": "^2.0.6",
    "i18next": "^21.10.0",
    "i18next-http-backend": "^1.4.1",
    "i18next-resources-to-backend": "^1.1.4",
    "jquery": "^3.6.0",
    "jsonc-parser": "^3.0.0",
    "latest": "^0.2.0",
    "libphonenumber-js": "^1.9.53",
    "lodash-es": "^4.17.21",
    "md5": "^2.3.0",
    "ng-recaptcha": "^11.0.0",
    "ngx-clipboard": "^15.0.0",
    "ngx-cookie-service-ssr": "^15.0.0",
    "ngx-infinite-scroll": "^17.0.0",
    "ngx-owl-carousel-o": "^15.0.0",
    "node-fetch": "^3.3.2",
    "parse5": "^6.0.1",
    "rx-angular": "^1.1.3",
    "rxjs": "~6.6.0",
    "svg-parser": "^2.0.4",
    "ts-loader": "^6.0.4",
    "tslib": "^2.2.0",
    "xlsx": "^0.18.0",
    "zone.js": "~0.14.4"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "17.2.1",
    "@angular-devkit/core": "17.2.1",
    "@angular-devkit/schematics": "17.2.1",
    "@angular-eslint/builder": "17.2.1",
    "@angular-eslint/eslint-plugin": "17.2.1",
    "@angular-eslint/eslint-plugin-template": "17.2.1",
    "@angular-eslint/schematics": "17.2.1",
    "@angular-eslint/template-parser": "17.2.1",
    "@angular/cli": "17.2.1",
    "@angular/compiler-cli": "17.2.2",
    "@babel/core": "^7.21.3",
    "@babel/runtime": "^7.23.2",
    "@compodoc/compodoc": "1.1.19",
    "@ngrx/store-devtools": "^17.1.1",
    "@spartacus/schematics": "~6.8.0",
    "@spartacus/setup": "~6.8.0",
    "@storybook/addon-actions": "^7.4.6",
    "@storybook/addon-designs": "^7.0.9",
    "@storybook/addon-essentials": "^7.4.6",
    "@storybook/addon-interactions": "^7.4.6",
    "@storybook/addon-links": "^7.4.6",
    "@storybook/angular": "^7.4.6",
    "@storybook/testing-library": "^0.2.2",
    "@types/express": "^4.17.0",
    "@types/file-saver": "^2.0.5",
    "@types/jasmine": "~3.8.0",
    "@types/lodash-es": "^4.17.6",
    "@types/node": "^12.20.47",
    "@types/scriptjs": "0.0.2",
    "@typescript-eslint/eslint-plugin": "4.28.2",
    "@typescript-eslint/parser": "4.28.2",
    "babel-loader": "^8.3.0",
    "browser-sync": "^3.0.0",
    "chromatic": "^6.18.0",
    "compression": "^1.7.4",
    "cross-env": "^7.0.3",
    "css-loader": "^6.7.1",
    "eslint": "^7.32.0",
    "eslint-config-prettier": "^9.1.0",
    "eslint-plugin-import": "2.27.5",
    "eslint-plugin-jsdoc": "40.1.0",
    "eslint-plugin-prefer-arrow": "1.2.3",
    "eslint-plugin-storybook": "^0.6.14",
    "express": "^4.15.2",
    "husky": "^8.0.3",
    "jasmine-core": "~3.8.0",
    "karma": "~6.3.0",
    "karma-chrome-launcher": "~3.1.0",
    "karma-coverage": "^2.2.1",
    "karma-coverage-istanbul-reporter": "^3.0.3",
    "karma-jasmine": "~4.0.0",
    "karma-jasmine-html-reporter": "~1.7.0",
    "lint-staged": "^15.2.0",
    "postcss": "^8.4.32",
    "prettier": "3.0.3",
    "prettier-plugin-organize-attributes": "^1.0.0",
    "puppeteer": "^21.1.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "scriptjs": "^2.5.9",
    "sonar-scanner": "^3.1.0",
    "storybook": "^7.4.6",
    "stylelint": "^15.11.0",
    "stylelint-config-recess-order": "^4.4.0",
    "stylelint-config-sass-guidelines": "^10.0.0",
    "stylelint-config-standard-scss": "^11.1.0",
    "ts-loader": "^6.0.4",
    "typescript": "5.3.3",
    "webpack-bundle-analyzer": "^4.6.1"
  },

Edit: I was able to resolve this issue by removing rxjs from package.json dependencies prior to running the migration. It is added again by the migration where it was previously failing.

❯ Update dependencies.
    🩹 Upgrading 'bootstrap' to ^4.6.2 (was 4.6.1)
    🩹 Upgrading 'i18next-resources-to-backend' to ^1.2.0 (was 1.1.4)
    ✅️ Added '@fontsource/open-sans' into dependencies
    ✅️ Added 'rxjs' into dependencies
UPDATE package.json (7366 bytes)
✔ Packages installed successfully.
  Migration completed (1 file modified).

[janwidmer]

@timdk I also tried with ng update @spartacus/[email protected], but it does not make a difference..

Here, the logic is explained about the versions and when a migration instruction is being executed (even from a member of the spartacus core team..).

I have enhanced the ticket description with my assumption, why it does not work

[timdk]

You are right, also in migrations.json it has a patch version.

I was able to resolve my issue by removing rxjs from package.json prior to running the migration. I updated my original comment.

[Platonn]

Reason of the problem: The ending .0-4 (or .0-whateverNumber) is not a valid Semantic Versioning. Therefore ng update command has problems to deal with it. The problem doesn't exist, if you're using a release with a normal ending .0.

Workaround 1: Please find-replace all occurrences of .0-4 to .0 in the repository. And only then build and publish libraries on your own.

Workaround 2: (didn't check it but I guess it should work): Alternatively you can jsut run ng upgrade with passing a specific version containing .0-someNumber, e.g ng update @spartacus/[email protected] (with exact .0-4 at the end).

Note: The ending .0-someNumber is used only for internal purposes in the core team. It doesn't comply with the official SemVer specification, therefore it may behave strange when running ng update scripts, which under the hood rely on the semver specs. I'm sorry for inconveniences that it causes for you when building and publishing packages on your own.

[janwidmer]

@Platonn thanks for your input. My Assumption was also, that the version could be the problem.. but it does not seem to be:

I have built and deployed spartacus 2211.19.0 to our package registry:

When I now run the command ng update @spartacus/[email protected], it still only changes the package json and does NOT run any migrations (even though, the version in the migrations.json matches the package version):

By the way, 2211.19.0-4 still seems to be a valid version according to https://jubianchi.github.io/semver-check/#/version/2211.19.0-4

Which sub version of the released version am I supposed to use?

In the github repo, there are several sub versions for 2211.19 and 2211.20:

In the docs, there is no reference, which sub version is actually released to the SAP NPM Repo:
https://help.sap.com/whats-new/c1c8cba6c45a4283bbfa44c84cb9e74d?locale=en-US

[RadhepS]

Hi @janwidmer,

You should be using the version that has nothing appended to it. Those are our official releases.
Version 2211.19.0-4 is actually a pre-release and 2211.19.0 is the actual released version.

The best way to know what is the latest version of a release is to use the tags:
https://github.com/SAP/spartacus/tags
https://github.com/SAP/spartacus/releases/tag/2211.19.0

You can checkout a commit associated to a tag, build the libraries, publish it using verdaccio

[kpawelczak]

We tested migrations with Verdaccio on the release branch release-2211.19.0, and they worked successfully.

When looking at 2211.19.0 and 2211.19.0-4, the suffix "-4" indicates a pre-release build.
When executing the command ng update @spartacus/[email protected], it will update to the highest version matching the specified major and minor versions, which in this case is 2211.19.x. As 2211.19.0 is the latest stable version, it will update to that version instead of the pre-release version 2211.19.0-4.

I would advise you to update @angular-devkit/schematics before updating Spartacus (https://github.com/SAP/spartacus/blob/develop/docs/migration/2211_19/2211-migration.md). The angular team made some changes to that package and it could affect migrations.

If you go to the node_modules -> @spartacus/schematics/src/migrations do you see 2211_0 there?

[janwidmer]

@RadhepS Thanks for the explanations

@kpawelczak I have updated @angular-eslint/schematics to 17.2.1 prior running the update command. I have tried with the command ng update @spartacus/[email protected] but same behaviour, it does not execute the migrations.

Yes, I see that folder:

Regarding Verdacchio, I tried following the instructions on how to publish the package, but I always receive an error. At the end, it should not matter, how the packages where published, as long as the packages are available in the correct version, right?

We publish it by running the following commands:

npm run build:libs

cd ${CI_PROJECT_DIR}/dist
npm publish

cd ../projects/schematics
npm publish

cd ../storefrontstyles
npm publish

From what I have seen, these are also the commands which are being used in verdacchio to publish

Here is the zip file of the package, that I have published to our private repo:
@spartacus_schematics-2211.19.0.tgz

I have also tried comparing this zip file with the one from the SAP Private Repo of spartacus to see if there are any differences in the files, but havent seen any..