ESMification

[Timvde]

I want to inform you about the ESMification that was recently completed in Firefox: the code was migrated away from Mozilla's JSM modules to ES6 modules.

In about a year (Firefox 136, planned for March 2025), Firefox will completely remove support for ChromeUtils.import and similar APIs.

All information about migrating to ES6 modules can be found in this migration document: https://docs.google.com/document/d/14FqYX749nJkCSL_GknCDZyQnuqkXNc9KoTuXSV3HtMg/edit

Some more information (the documentation about the migration in the Firefox codebase itself) is here: https://docs.google.com/document/d/1cpzIK-BdP7u6RJSar-Z955GV--2Rj8V4x2vl34m36Go/edit

[onemen]

I am using ChromeUtils.importESModule and ChromeUtils.defineESModuleGetters for Fireofx files that ends with .sys.mjs

see https://github.com/onemen/TabMixPlus/blob/main/addon/modules/ChromeUtils.jsm

Do you see any problem with Tab Mix Plus?

[117649]

I am using ChromeUtils.importESModule and ChromeUtils.defineESModuleGetters for Fireofx files that ends with .sys.mjs

see https://github.com/onemen/TabMixPlus/blob/main/addon/modules/ChromeUtils.jsm

Do you see any problem with Tab Mix Plus?

@onemen Well I know something:

TabMixPlus/addon/modules/TabmixSvc.jsm

Lines 425 to 428 in c4c1c2a

	// Don't ChromeUtils.import here it can not import variables that
	// are not in EXPORTED_SYMBOLS
	// eslint-disable-next-line mozilla/use-chromeutils-import
	return Cu.import("resource:///modules/sessionstore/SessionStore.jsm");

And it is very bad.

[Timvde]

Do you see any problem with Tab Mix Plus?

No, sorry for being unclear about that :) I just did a search on the code and noticed that some of the affected APIs were still used. I filed this issue only to inform you ahead of time in case you weren't aware. If this is not applicable to TMP because you are already using ES6 modules everywhere and the JSM modules are only there for compatibility with older versions, that's great :D

[onemen]

Thank you

[117649]

@onemen Just realize this will cause bootstraploader script stop working.
I've reviewed loader script again I think we can have a solution by set legacy addon's sign state to 'not required' and freeze appDisable to false during install or we can just set it to 'privileged'.

Plus do we have a way to changeCode() an ES6 module obj?

[onemen]

We will have to change all Tab Mix Plus and Firefox scripts jsm files to be es6 modules

[117649]

@onemen

We will have to change all Tab Mix Plus and Firefox scripts jsm files to be es6 modules

First attempt of BootstrapLoader.jsm .

Left some that are not yet es6 un touched.

Have freeze 'appDisabled' and 'signedState' on addon internal.

/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

'use strict';

let EXPORTED_SYMBOLS = [];

const { XPCOMUtils } = ChromeUtils.importESModule('resource://gre/modules/XPCOMUtils.sys.mjs');
const Services = globalThis.Services;

XPCOMUtils.defineLazyModuleGetters(this, {
  Blocklist: 'resource://gre/modules/Blocklist.jsm',
  ConsoleAPI: 'resource://gre/modules/Console.jsm',
  InstallRDF: 'chrome://userchromejs/content/RDFManifestConverter.jsm',
});

Services.obs.addObserver(doc => {
  if (doc.location.protocol + doc.location.pathname === 'about:addons' ||
      doc.location.protocol + doc.location.pathname === 'chrome:/content/extensions/aboutaddons.html') {
    const win = doc.defaultView;
    let handleEvent_orig = win.customElements.get('addon-card').prototype.handleEvent;
    win.customElements.get('addon-card').prototype.handleEvent = function (e) {
      if (e.type === 'click' &&
          e.target.getAttribute('action') === 'preferences' &&
          this.addon.__AddonInternal__.optionsType == 1/*AddonManager.OPTIONS_TYPE_DIALOG*/ && !!this.addon.optionsURL) {
        var windows = Services.wm.getEnumerator(null);
        while (windows.hasMoreElements()) {
          var win2 = windows.getNext();
          if (win2.closed) {
            continue;
          }
          if (win2.document.documentURI == this.addon.optionsURL) {
            win2.focus();
            return;
          }
        }
        var features = 'chrome,titlebar,toolbar,centerscreen';
        win.docShell.rootTreeItem.domWindow.openDialog(this.addon.optionsURL, this.addon.id, features);
      } else {
        handleEvent_orig.apply(this, arguments);
      }
    }
    let update_orig = win.customElements.get('addon-options').prototype.update;
    win.customElements.get('addon-options').prototype.update = function (card, addon) {
      update_orig.apply(this, arguments);
      if (addon.__AddonInternal__.optionsType == 1/*AddonManager.OPTIONS_TYPE_DIALOG*/ && !!addon.optionsURL)
        this.querySelector('panel-item[data-l10n-id="preferences-addon-button"]').hidden = false;
    }
  }
}, 'chrome-document-loaded');

const {AddonManager} = ChromeUtils.importESModule('resource://gre/modules/AddonManager.sys.mjs');
const {XPIDatabase, AddonInternal} = ChromeUtils.importESModule('resource://gre/modules/addons/XPIDatabase.sys.mjs');

// const { defineAddonWrapperProperty } = Cu.import('resource://gre/modules/addons/XPIDatabase.jsm');
// defineAddonWrapperProperty('optionsType', function optionsType() {
//   if (!this.isActive) {
//     return null;
//   }

//   let addon = this.__AddonInternal__;
//   let hasOptionsURL = !!this.optionsURL;

//   if (addon.optionsType) {
//     switch (parseInt(addon.optionsType, 10)) {
//       case 1/*AddonManager.OPTIONS_TYPE_DIALOG*/:
//       case AddonManager.OPTIONS_TYPE_TAB:
//       case AddonManager.OPTIONS_TYPE_INLINE_BROWSER:
//         return hasOptionsURL ? addon.optionsType : null;
//     }
//     return null;
//   }

//   return null;
// });

XPIDatabase.isDisabledLegacy = () => false;

ChromeUtils.defineLazyGetter(this, 'BOOTSTRAP_REASONS', () => {
  const {XPIProvider} = ChromeUtils.importESModule('resource://gre/modules/addons/XPIProvider.sys.mjs');
  return XPIProvider.BOOTSTRAP_REASONS;
});

const {Log} = ChromeUtils.importESModule('resource://gre/modules/Log.sys.mjs');
var logger = Log.repository.getLogger('addons.bootstrap');

/**
 * Valid IDs fit this pattern.
 */
var gIDTest = /^(\{[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\}|[a-z0-9-\._]*\@[a-z0-9-\._]+)$/i;

// Properties that exist in the install manifest
const PROP_METADATA      = ['id', 'version', 'type', 'internalName', 'updateURL',
                            'optionsURL', 'optionsType', 'aboutURL', 'iconURL'];
const PROP_LOCALE_SINGLE = ['name', 'description', 'creator', 'homepageURL'];
const PROP_LOCALE_MULTI  = ['developers', 'translators', 'contributors'];

// Map new string type identifiers to old style nsIUpdateItem types.
// Retired values:
// 32 = multipackage xpi file
// 8 = locale
// 256 = apiextension
// 128 = experiment
// theme = 4
const TYPES = {
  extension: 2,
  dictionary: 64,
};

const COMPATIBLE_BY_DEFAULT_TYPES = {
  extension: true,
  dictionary: true,
};

const hasOwnProperty = Function.call.bind(Object.prototype.hasOwnProperty);

function isXPI(filename) {
  let ext = filename.slice(-4).toLowerCase();
  return ext === '.xpi' || ext === '.zip';
}

/**
 * Gets an nsIURI for a file within another file, either a directory or an XPI
 * file. If aFile is a directory then this will return a file: URI, if it is an
 * XPI file then it will return a jar: URI.
 *
 * @param {nsIFile} aFile
 *        The file containing the resources, must be either a directory or an
 *        XPI file
 * @param {string} aPath
 *        The path to find the resource at, '/' separated. If aPath is empty
 *        then the uri to the root of the contained files will be returned
 * @returns {nsIURI}
 *        An nsIURI pointing at the resource
 */
function getURIForResourceInFile(aFile, aPath) {
  if (!isXPI(aFile.leafName)) {
    let resource = aFile.clone();
    if (aPath)
      aPath.split('/').forEach(part => resource.append(part));

    return Services.io.newFileURI(resource);
  }

  return buildJarURI(aFile, aPath);
}

/**
 * Creates a jar: URI for a file inside a ZIP file.
 *
 * @param {nsIFile} aJarfile
 *        The ZIP file as an nsIFile
 * @param {string} aPath
 *        The path inside the ZIP file
 * @returns {nsIURI}
 *        An nsIURI for the file
 */
function buildJarURI(aJarfile, aPath) {
  let uri = Services.io.newFileURI(aJarfile);
  uri = 'jar:' + uri.spec + '!/' + aPath;
  return Services.io.newURI(uri);
}

var BootstrapLoader = {
  name: 'bootstrap',
  manifestFile: 'install.rdf',
  async loadManifest(pkg) {
    /**
     * Reads locale properties from either the main install manifest root or
     * an em:localized section in the install manifest.
     *
     * @param {Object} aSource
     *        The resource to read the properties from.
     * @param {boolean} isDefault
     *        True if the locale is to be read from the main install manifest
     *        root
     * @param {string[]} aSeenLocales
     *        An array of locale names already seen for this install manifest.
     *        Any locale names seen as a part of this function will be added to
     *        this array
     * @returns {Object}
     *        an object containing the locale properties
     */
    function readLocale(aSource, isDefault, aSeenLocales) {
      let locale = {};
      if (!isDefault) {
        locale.locales = [];
        for (let localeName of aSource.locales || []) {
          if (!localeName) {
            logger.warn('Ignoring empty locale in localized properties');
            continue;
          }
          if (aSeenLocales.includes(localeName)) {
            logger.warn('Ignoring duplicate locale in localized properties');
            continue;
          }
          aSeenLocales.push(localeName);
          locale.locales.push(localeName);
        }

        if (locale.locales.length == 0) {
          logger.warn('Ignoring localized properties with no listed locales');
          return null;
        }
      }

      for (let prop of [...PROP_LOCALE_SINGLE, ...PROP_LOCALE_MULTI]) {
        if (hasOwnProperty(aSource, prop)) {
          locale[prop] = aSource[prop];
        }
      }

      return locale;
    }

    let manifestData = await pkg.readString('install.rdf');
    let manifest = InstallRDF.loadFromString(manifestData).decode();

    let addon = new AddonInternal();
    for (let prop of PROP_METADATA) {
      if (hasOwnProperty(manifest, prop)) {
        addon[prop] = manifest[prop];
      }
    }

    if (!addon.type) {
      addon.type = 'extension';
    } else {
      let type = addon.type;
      addon.type = null;
      for (let name in TYPES) {
        if (TYPES[name] == type) {
          addon.type = name;
          break;
        }
      }
    }

    if (!(addon.type in TYPES))
      throw new Error('Install manifest specifies unknown type: ' + addon.type);

    if (!addon.id)
      throw new Error('No ID in install manifest');
    if (!gIDTest.test(addon.id))
      throw new Error('Illegal add-on ID ' + addon.id);
    if (!addon.version)
      throw new Error('No version in install manifest');

    addon.strictCompatibility = (!(addon.type in COMPATIBLE_BY_DEFAULT_TYPES) ||
                                 manifest.strictCompatibility == 'true');

    // Only read these properties for extensions.
    if (addon.type == 'extension') {
      if (manifest.bootstrap != 'true') {
        throw new Error('Non-restartless extensions no longer supported');
      }

      if (addon.optionsType &&
          addon.optionsType != 1/*AddonManager.OPTIONS_TYPE_DIALOG*/ &&
          addon.optionsType != AddonManager.OPTIONS_TYPE_INLINE_BROWSER &&
          addon.optionsType != AddonManager.OPTIONS_TYPE_TAB) {
            throw new Error('Install manifest specifies unknown optionsType: ' + addon.optionsType);
      }

      if (addon.optionsType)
        addon.optionsType = parseInt(addon.optionsType);
    }

    addon.defaultLocale = readLocale(manifest, true);

    let seenLocales = [];
    addon.locales = [];
    for (let localeData of manifest.localized || []) {
      let locale = readLocale(localeData, false, seenLocales);
      if (locale)
        addon.locales.push(locale);
    }

    let dependencies = new Set(manifest.dependencies);
    addon.dependencies = Object.freeze(Array.from(dependencies));

    let seenApplications = [];
    addon.targetApplications = [];
    for (let targetApp of manifest.targetApplications || []) {
      if (!targetApp.id || !targetApp.minVersion ||
          !targetApp.maxVersion) {
            logger.warn('Ignoring invalid targetApplication entry in install manifest');
            continue;
      }
      if (seenApplications.includes(targetApp.id)) {
        logger.warn('Ignoring duplicate targetApplication entry for ' + targetApp.id +
                    ' in install manifest');
        continue;
      }
      seenApplications.push(targetApp.id);
      addon.targetApplications.push(targetApp);
    }

    // Note that we don't need to check for duplicate targetPlatform entries since
    // the RDF service coalesces them for us.
    addon.targetPlatforms = [];
    for (let targetPlatform of manifest.targetPlatforms || []) {
      let platform = {
        os: null,
        abi: null,
      };

      let pos = targetPlatform.indexOf('_');
      if (pos != -1) {
        platform.os = targetPlatform.substring(0, pos);
        platform.abi = targetPlatform.substring(pos + 1);
      } else {
        platform.os = targetPlatform;
      }

      addon.targetPlatforms.push(platform);
    }

    addon.userDisabled = false;
    addon.softDisabled = addon.blocklistState == Blocklist.STATE_SOFTBLOCKED;
    addon.applyBackgroundUpdates = AddonManager.AUTOUPDATE_DEFAULT;

    addon.userPermissions = null;

    addon.icons = {};
    if (await pkg.hasResource('icon.png')) {
      addon.icons[32] = 'icon.png';
      addon.icons[48] = 'icon.png';
    }

    if (await pkg.hasResource('icon64.png')) {
      addon.icons[64] = 'icon64.png';
    }

    Object.defineProperty(addon, 'appDisabled', {
      value: false,
      writable: false
    });

    Object.defineProperty(addon, 'signedState', {
      value: AddonManager.SIGNEDSTATE_NOT_REQUIRED,
      writable: false
    });

    return addon;
  },

  loadScope(addon) {
    let file = addon.file || addon._sourceBundle;
    let uri = getURIForResourceInFile(file, 'bootstrap.js').spec;
    let principal = Services.scriptSecurityManager.getSystemPrincipal();

    let sandbox = new Cu.Sandbox(principal, {
      sandboxName: uri,
      addonId: addon.id,
      wantGlobalProperties: ['ChromeUtils'],
      metadata: { addonID: addon.id, URI: uri },
    });

    try {
      Object.assign(sandbox, BOOTSTRAP_REASONS);

      XPCOMUtils.defineLazyGetter(sandbox, 'console', () =>
        new ConsoleAPI({ consoleID: `addon/${addon.id}` }));

      Services.scriptloader.loadSubScript(uri, sandbox);
    } catch (e) {
      logger.warn(`Error loading bootstrap.js for ${addon.id}`, e);
    }

    function findMethod(name) {
      if (sandbox[name]) {
        return sandbox[name];
      }

      try {
        let method = Cu.evalInSandbox(name, sandbox);
        return method;
      } catch (err) { }

      return () => {
        logger.warn(`Add-on ${addon.id} is missing bootstrap method ${name}`);
      };
    }

    let install = findMethod('install');
    let uninstall = findMethod('uninstall');
    let startup = findMethod('startup');
    let shutdown = findMethod('shutdown');

    return {
      install(...args) {
        install(...args);
        // Forget any cached files we might've had from this extension.
        Services.obs.notifyObservers(null, 'startupcache-invalidate');
      },

      uninstall(...args) {
        uninstall(...args);
        // Forget any cached files we might've had from this extension.
        Services.obs.notifyObservers(null, 'startupcache-invalidate');
      },

      startup(...args) {
        if (addon.type == 'extension') {
          logger.debug(`Registering manifest for ${file.path}\n`);
          Components.manager.addBootstrappedManifestLocation(file);
        }
        return startup(...args);
      },

      shutdown(data, reason) {
        try {
          return shutdown(data, reason);
        } catch (err) {
          throw err;
        } finally {
          if (reason != BOOTSTRAP_REASONS.APP_SHUTDOWN) {
            logger.debug(`Removing manifest for ${file.path}\n`);
            Components.manager.removeBootstrappedManifestLocation(file);
          }
        }
      },
    };
  },
};

AddonManager.addExternalExtensionLoader(BootstrapLoader);

if (AddonManager.isReady) {
  AddonManager.getAllAddons().then(addons => {
    addons.forEach(addon => {
      if (addon.type == 'extension' && !addon.isWebExtension && !addon.userDisabled) {
        addon.reload();
      };
    });
  });
}

[onemen]

@117649,

try to turn BootstrapLoader.jsm to BootstrapLoader.sys.mjs and use ChromeUtils.importESModule in config.js

[117649]

@117649,

try to turn BootstrapLoader.jsm to BootstrapLoader.sys.mjs and use ChromeUtils.importESModule in config.js

NO.

BootstrapLoader.jsm does not export any object so I'll try Services.scriptloader.loadSubScript first.
But before any of those there are RDFManifestConverter.jsm & RDFDataSource.jsm to go.

[117649]

@onemen
utils.zip

[onemen]

since xiaoxiaoflood/firefox-scripts is not active lets do all firefox-scripts discussions and code in https://github.com/onemen/firefox-scripts

can you create a PR?

[117649]

@onemen 135 just drop for DevEd and 136 is on nightly.
As for what I understand JSM should be deprecated by then.
Are we doing good there?

[onemen]

@117649

I am using nightly every day without any issue related to jsm, if you see any issue open new issue

[117649]

@onemen So mozilla not gonna remove JSM API right away?

[onemen]

@117649 ,

I don't know, where did you read about JSM depreciation?

[117649]

@onemen First post:

I want to inform you about the ESMification that was recently completed in Firefox: the code was migrated away from Mozilla's JSM modules to ES6 modules.

In about a year (Firefox 136, planned for March 2025), Firefox will completely remove support for ChromeUtils.import and similar APIs.

[onemen]

Currently Tab Mix have 35 jsm files
I will start to convert them to ESM files.

I'm not sure when Firefox will drop support for ChromeUtils.import

[Timvde]

The timeline was Firefox 136 (to be found here). At least the migration to ESM modules is completed, so I think they're on schedule.

The cleanup bug ticket is here: https://bugzilla.mozilla.org/show_bug.cgi?id=1776174

I don't know enough about the specifics to know for sure which is the one, but these sound relevant:

• Remove the target object argument from ChromeUtils.import
• Remove support for component JSMs
• Remove support for JSMs
• Remove support for loading JSMs in Actors

The three last one have very recent patches (2 days old), but they haven't landed yet.

[117649]

Currently Tab Mix have 35 jsm files I will start to convert them to ESM files.

I'm not sure when Firefox will drop support for ChromeUtils.import

Is that really needed to convert them to ESM at all?
We're not likely to get any benefit from this and also not gonna lose anything over not doing the convert.
Aren't we could just make a function that mimic what ChromeUtils.import do by load .jsm as a plain script with Services.scriptloader.loadSubScript("xxx",t) then insert exposed properties into the context?

function importJSM(url, tgt = this ?? globalThis) {
  let t = {};
  Services.scriptloader.loadSubScript(url, t);
  t.EXPORTED_SYMBOLS.forEach(p => {
    tgt[p] = t[p];
  });
}