Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

module: CJS exports detection for modules with __esModule export #33416

Closed
wants to merge 5 commits into from

Conversation

guybedford
Copy link
Contributor

@guybedford guybedford commented May 15, 2020

This PR provides named exports support for transpiled CommonJS modules (restriction to __esModule sources added based on feedback from discussion at #33416 (comment)) to work out correctly using a very fast Wasm-based analysis process that runs before CJS execution, allowing import { name } from 'transpiled-esm-to-cjs' as most JS users would expect, without incurring any significant performance penalty.

This analysis is cached in a weakmap associated with the module object, which also stores the early-accessed source cache for usage on the actual load.

The approach is very similar to the approach TypeScript uses for detection of CommonJS named exports during the binding phase as we discussed in the last modules meeting with @weswigham.

The major concern for a parsing approach is the performance overhead of running a full parse of all CommonJS modules that are imported from an ES module. Acorn parse time is usually in the 10s of miliseconds for standard size source files to 100s of miliseconds for large sources. Scaling this up to multiple dependencies could introduce some loading overhead.

This PR incorporates a fork of es-module-lexer, cjs-module-lexer (https://github.com/guybedford/cjs-module-lexer) to handle this exports analysis with Web Assembly so that the list of named exports for CJS can be known at link time via a very fast source Wasm lexer process (sub millisecond for most sources).

es-module-lexer is used in many current ES module tooling projects, and has been battle tested against a wide range of real-world JS sources over the past two years. It supports the full JS grammar, including comments, strings, template strings and regex / division operator ambiguity.

cjs-module-lexer forks to handle CommonJS detecting named exports extractions based on the rules outlined at https://github.com/guybedford/cjs-module-lexer#supported. It is lexing matcher for patterns of exports. etc, including a Webpack heuristic as well.

Limitations include that some minified sources that obscure the exports binding cannot be parsed as the analysis does not do value tracking at all as it only detects common token patterns for exports etc.

In testing the approach has shown to work well for standard TypeScript and Babel transpilations. Identifiers are always filtered to be valid JS identifiers.

The major edge case this differs from current semantics with bundlers on is that the default export is retained as the full module.exports instance for backwards compat and consistency. This is important for Node.js semantic consistency I think.

//cc @nodejs/modules-active-members

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

@nodejs-github-bot nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label May 15, 2020
deps/cjs-module-lexer/dist/lexer.js Outdated Show resolved Hide resolved
lib/internal/modules/esm/translators.js Outdated Show resolved Hide resolved
lib/internal/modules/esm/translators.js Outdated Show resolved Hide resolved
lib/internal/modules/esm/translators.js Outdated Show resolved Hide resolved
Copy link
Member

@devsnek devsnek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm against this feature in general, and the specific implementation needs to support running without access to eval (running with --disallow-code-generation-from-strings). The exact behaviour of this will also need to be exhaustively documented and matched to node's semver cycle.

Copy link
Contributor

@jkrems jkrems left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the general direction of this. It offers a pretty reasonable solution to the incremental adoption problem (see comment on @bmeck's repo). It would be great if we could offer an answer and this is the closest thing I'm aware of.

(I didn't mention it here but I agree with the implementation concerns around eval raised by @devsnek, also the requirement for a way to lock down the semantics to match versioning expectations.)

lib/internal/modules/esm/translators.js Outdated Show resolved Hide resolved
lib/internal/modules/esm/translators.js Outdated Show resolved Hide resolved
test/fixtures/es-modules/cjs-exports.mjs Show resolved Hide resolved
@guybedford
Copy link
Contributor Author

@devsnek I've removed the eval usage and also written up a formal grammar of the detection process in https://github.com/guybedford/cjs-module-lexer#grammar.

Does that cover your concerns?

@addaleax
Copy link
Member

@guybedford What exactly is the effect of __esModule here? Is this only applied in that case, or always?

I think it would address @devsnek’s main concern (that this makes otherwise non-breaking changes to CJS code breaking, which I agree is bad) if this only applied to modules that export __esModule: true, because that means that it is originally written as (fake) ESM?

@guybedford
Copy link
Contributor Author

guybedford commented May 16, 2020

@addaleax the only effect of __esModule is that the default export is set to exports.default when there is both an __esModule and exports.default value available. Otherwise, the behaviour is identical.

This feature was specifically requested by @jkrems to support the standard Babel output that transpiles:

export default value;

to:

Object.defineProperty(exports, '__esModule', { value: true });
exports.default = value;

The justification is that by supporting __esModule, replacing the CommonJS "faux module" transpiled source with the original ESM source is a non-breaking gradual migration path to real ES modules.

@devsnek
Copy link
Member

devsnek commented May 16, 2020

@guybedford I think the technical concerns I mentioned above were addressed. That being said, I am still -1 on this pr. I've exhaustively outlined the issues I have with cjs export detection in nodejs/modules#509, and in the case of this pr the issues I have with not using V8's parser.

@guybedford
Copy link
Contributor Author

guybedford commented May 16, 2020

@devsnek, as far as I've been able to condense your feedback from that thread it seems to come down to the following argument:

This parser approach will not detect dynamic exports population, thus CommonJS packages that are currently setting exports.property changing to setting that same property dynamically in a way that can't be analyzed in say a patch release will cause a break for ESM consumers without realising it due to the named exports detection effectively being part of the public API for ESM. This "guesswork" in the parsing approach being unclear to CJS package authors is a source of possible bugs. Node.js core should not implement a feature that has this lack of consistency.

You therefore argue that users who want to support ESM consumers with named exports should write ESM wrappers instead.

Let me know if that seems like an adequate summary of your opinion here?

This still leaves older packages with authors who maybe don't want to publish an ESM wrapper, or simply are no longer maintaining the package.

Are you ok with not supporting named exports for these use cases?

Remind me as well - were you for or against early CJS execution? Between that and this those are the only ways at this point of getting users named exports based on their existing expectations with modules tools.

@devsnek
Copy link
Member

devsnek commented May 16, 2020

@guybedford i'm not going to say i'm content with lack of named exports for old packages, but i think having consistent behaviour is much better for dev experience than named exports.

@guybedford guybedford added the modules-agenda Issues and PRs to discuss during the meetings of the Modules team. label May 19, 2020
@GeoffreyBooth
Copy link
Member

I wrote some code to test this out: https://github.com/GeoffreyBooth/node-test-commonjs-named-exports. It took a lot more code and effort than expected, so please check my work in case my results are tainted by bugs. There's a README in the repo.

Anyway the bottom line is that of 936 packages tested (from within npm's top 1000):

  • 62% of packages (580 of 936) had all CommonJS named exports detected.

  •  5% of packages (51 of 936) had some but not all CommonJS named exports detected.

  • 33% of packages (305 of 936) had no CommonJS named exports detected.

And of all the named exports of all of those packages, 3,390 of 22,579 CommonJS named exports (15%) were detected successfully.

Packages with all CommonJS named exports detected:
  • async
  • minimist
  • ms
  • graceful-readlink
  • through2
  • inherits
  • coffee-script
  • yeoman-generator
  • xtend
  • jquery
  • source-map
  • ansi-regex
  • react
  • extend
  • object-assign
  • wrappy
  • isarray
  • once
  • brace-expansion
  • xml2js
  • redis
  • bindings
  • hoek
  • strip-ansi
  • escape-string-regexp
  • asap
  • esprima
  • path-is-absolute
  • xmlbuilder
  • yosay
  • has-flag
  • jade
  • envify
  • mime-types
  • nopt
  • util-deprecate
  • has-ansi
  • ejs
  • string_decoder
  • inflight
  • body-parser
  • entities
  • number-is-nan
  • concat-map
  • pinkie-promise
  • connect
  • double-ended-queue
  • concat-stream
  • json-stringify-safe
  • classnames
  • mysql
  • redis-commands
  • acorn
  • isstream
  • oauth
  • loader-utils
  • argparse
  • jstransform
  • tough-cookie
  • uuid
  • passport-strategy
  • socket.io-client
  • browserify
  • ramda
  • es6-symbol
  • ember-cli-version-checker
  • babel-core
  • tunnel-agent
  • domelementtype
  • parseurl
  • get-stdin
  • caseless
  • escape-html
  • hawk
  • boom
  • chokidar
  • broccoli-funnel
  • loose-envify
  • oauth-sign
  • utils-merge
  • util
  • passport-oauth2
  • har-validator
  • path-to-regexp
  • aws-sign2
  • stringstream
  • lru-cache
  • lodash.keys
  • cli-color
  • os-tmpdir
  • es6-iterator
  • css
  • options
  • dom-serializer
  • string-width
  • rsvp
  • os-homedir
  • ultron
  • path
  • split
  • js-tokens
  • immutable
  • on-finished
  • sprintf-js
  • xmldom
  • unique-random-array
  • cookie
  • uglify-to-browserify
  • passport-oauth
  • querystring
  • broccoli-babel-transpiler
  • type-detect
  • node-pre-gyp
  • escodegen
  • lodash.rest
  • component-emitter
  • nodemailer
  • temp
  • phantomjs
  • depd
  • ee-first
  • react-dom
  • unique-random
  • cli-table
  • open
  • send
  • ini
  • deep-equal
  • finalhandler
  • jsonify
  • wrench
  • lodash.assign
  • serve-static
  • js-base64
  • crypto
  • socket.io-parser
  • diff
  • estraverse
  • stack-trace
  • xmlhttprequest
  • camelcase
  • type-is
  • decamelize
  • duplexer2
  • invariant
  • sliced
  • node.extend
  • json-stable-stringify
  • duplexer
  • cookie-signature
  • vary
  • tar
  • deep-extend
  • is-finite
  • jshint
  • path-exists
  • replace-ext
  • map-stream
  • is-buffer
  • JSONStream
  • read
  • html-minifier
  • exit
  • accepts
  • walk
  • progress
  • amdefine
  • tmp
  • mute-stream
  • callsite
  • extsprintf
  • repeating
  • findup-sync
  • events
  • swig
  • content-type
  • strip-bom
  • rc
  • source-map-support
  • passport-oauth1
  • is-utf8
  • etag
  • fresh
  • has-binary
  • encoding
  • adm-zip
  • restify
  • deep-eql
  • is-windows
  • foreachasync
  • klaw
  • falafel
  • assertion-error
  • engine.io
  • cookie-parser
  • bytes
  • performance-now
  • thunkify
  • array-uniq
  • typedarray
  • url
  • merge-descriptors
  • convert-source-map
  • is-extendable
  • isobject
  • os-locale
  • text-table
  • uid2
  • sparkles
  • cliui
  • y18n
  • find-up
  • strip-json-comments
  • gulp-rename
  • end-of-stream
  • broccoli-persistent-filter
  • is-relative
  • esprima-fb
  • range-parser
  • defined
  • update-notifier
  • lodash.template
  • raf
  • prr
  • socket.io-adapter
  • broccoli-filter
  • css-parse
  • errno
  • growl
  • is-promise
  • indexof
  • array-flatten
  • function-bind
  • bcrypt
  • image-size
  • ansi
  • markdown
  • requires-port
  • hooker
  • keypress
  • recast
  • lodash.merge
  • brfs
  • content-disposition
  • pretty-bytes
  • gaze
  • lcid
  • component-type
  • stream-combiner
  • node-gyp
  • raw-body
  • kind-of
  • read-pkg-up
  • unc-path-regex
  • extend-shallow
  • nodemailer-shared
  • lodash._reinterpolate
  • machine
  • ndarray
  • cross-spawn
  • babel-polyfill
  • is-unc-path
  • MD5
  • esnextguardian
  • foreach
  • jsprim
  • array-differ
  • watch
  • dtrace-provider
  • lodash.defaults
  • mv
  • negotiator
  • hapi
  • proxy-addr
  • invert-kv
  • punycode
  • node-static
  • copy-dereference
  • is-my-json-valid
  • pify
  • media-typer
  • code-point-at
  • is-fullwidth-code-point
  • node-hid
  • globby
  • xml2json
  • nodemailer-fetch
  • soap
  • wangzhi
  • esutils
  • broccoli-merge-trees
  • arrify
  • pause
  • googleapis
  • multipipe
  • cron
  • lodash._mapcache
  • word-wrap
  • uniq
  • iconv
  • basic-auth
  • del
  • urllib
  • error
  • nedb
  • fancy-log
  • mqtt
  • amqplib
  • express-session
  • is-glob
  • cssom
  • defaults
  • retry
  • on-headers
  • grunt-contrib-jshint
  • symlink-or-copy
  • nano
  • domhandler
  • next-tick
  • setimmediate
  • cryptiles
  • md5
  • htmlparser
  • string-template
  • knox
  • is-extglob
  • lodash.get
  • dargs
  • unpipe
  • koa
  • map-obj
  • emitter-component
  • lower-case
  • urix
  • cookiejar
  • lodash._reescape
  • lodash._reevaluate
  • lodash.isplainobject
  • beeper
  • has-gulplog
  • lodash._stack
  • sockjs
  • blank-object
  • detective
  • base64-js
  • irc
  • grunt-contrib-uglify
  • lazy-cache
  • safe-json-stringify
  • serve-favicon
  • date-now
  • to-array
  • character-parser
  • findit
  • domify
  • babelify
  • batch
  • xml
  • noflo
  • clone-stats
  • constantinople
  • gulp-uglify
  • vinyl-sourcemaps-apply
  • multimatch
  • twit
  • flux
  • dotenv
  • pad-component
  • shell-quote
  • revalidator
  • lodash._arraymap
  • sntp
  • util-extend
  • with
  • jws
  • merge-defaults
  • isomorphic-fetch
  • private
  • glogg
  • parse5
  • topo
  • grunt-contrib-clean
  • sentence-case
  • dnode
  • reduce-component
  • rework
  • memoizee
  • taketalk
  • query-string
  • natural
  • charm
  • byline
  • buffers
  • shortid
  • cors
  • liftoff
  • follow
  • cross-spawn-async
  • strip-indent
  • isemail
  • lodash._baseisequal
  • validate.io-number
  • redux
  • sync-exec
  • abstract-leveldown
  • browserify-transform-tools
  • hypher
  • compression
  • config-chain
  • broccoli-plugin
  • xdg-basedir
  • regexp-clone
  • win-spawn
  • lodash._arrayeach
  • dezalgo
  • read-chunk
  • koa-compose
  • from
  • archy
  • type-component
  • babel-traverse
  • sylvester
  • gulp-concat
  • lodash.assigninwith
  • bit-twiddle
  • strict-uri-encode
  • mongojs
  • matcher-collection
  • ansi-wrap
  • readline-sync
  • http
  • jsonschema
  • better-assert
  • acorn-globals
  • pump
  • gruntfile-editor
  • iniparser
  • signal-exit
  • insert-css
  • json-schema
  • mem-fs-editor
  • github-username
  • any-promise
  • block-stream
  • azure-common
  • has
  • weak-map
  • object.assign
  • for-in
  • event-emitter
  • yeoman-test
  • backoff
  • iota-array
  • spawn-sync
  • each-async
  • imagemagick
  • stream-consume
  • pend
  • detect-conflict
  • validate.io-object
  • warning
  • is-plain-object
  • babylon
  • babel-template
  • typical
  • upper-case
  • lodash.foreach
  • grunt-contrib-watch
  • load-grunt-tasks
  • baconjs
  • nth-check
  • prettyjson
  • dom-walk
  • lodash.keysin
  • observ
  • pretty-hrtime
  • fast-ordered-set
  • precond
  • level-packager
  • promised-io
  • is-plain-obj
  • css-what
  • gulp-template
  • maxmin
  • ast-types
  • array-series
  • three
  • run-sequence
  • broccoli-kitchen-sink-helpers
  • slash
  • passport-local
  • typechecker
  • read-pkg
  • anymatch
  • tunnel
  • regenerator
  • watchify
  • array-parallel
  • @linclark/pkg
  • nodemailer-smtp-transport
  • browser-sync
  • for-own
  • kareem
  • gulp-install
  • gl-matrix
  • static-module
  • url-join
  • tedious
  • binary
  • require-all
  • bufferjs
  • camelcase-keys
  • json3
  • busboy
  • markdown-it
  • loud-rejection
  • redent
  • require-directory
  • trim-newlines
  • onetime
  • pngjs
  • lodash.clonedeep
  • typedarray-to-buffer
  • grunt-contrib-copy
  • component-props
  • html-entities
  • lodash.isstring
  • temporary
  • useragent
  • async-each
  • split2
  • reflect-metadata
  • gulp-conflict
  • to-function
  • lodash.debounce
  • to-no-case
  • cwise-compiler
  • throttleit
  • typedarray-pool
  • lodash.isfunction
  • fd-slicer
  • lodash._basefor
  • collections
  • validate.io-integer
  • forwarded
  • tar-stream
  • is-object
  • ftp
  • lodash._baseslice
  • lodash._baseeach
  • ansi-color
  • fj-curry
  • agentkeepalive
  • ltx
  • coveralls
  • to-space-case
  • opn
  • as-array
  • hook.io
  • lodash.isobject
  • repeat-string
  • buffer-equal
  • lodash._baseflatten
  • tildify
  • is-url
  • grunt-contrib-concat
  • node-promise
  • js-string-escape
  • feedparser
  • quick-temp
  • sync-request
  • trim
  • promise-map-series
  • lodash.clone
  • hoist-non-react-statics
  • recursive-readdir
  • component-bind
  • ap
  • request-progress
  • level-sublevel
Packages with some but not all CommonJS named exports detected:
  • debug
  • readable-stream
  • commander
  • express
  • optimist
  • cheerio
  • uglify-js
  • superagent
  • shelljs
  • mocha
  • core-util-is
  • htmlparser2
  • bl
  • chai
  • tape
  • d3
  • bunyan
  • process
  • requirejs
  • passport
  • fs
  • morgan
  • gm
  • commoner
  • ua-parser-js
  • nodeunit
  • http-errors
  • needle
  • webpack
  • pull-stream
  • autoprefixer
  • sequelize
  • node-fetch
  • follow-redirects
  • babel-types
  • consolidate
  • figures
  • transformers
  • node-notifier
  • assert
  • libmime
  • objnest
  • bn.js
  • xlsx
  • engine.io-client
  • mongoskin
  • jison
  • mssql
  • iced-runtime
  • mathjs
  • LiveScript
Packages with no CommonJS named exports detected:
  • lodash
  • request
  • underscore
  • chalk
  • glob
  • colors
  • bluebird
  • q
  • mkdirp
  • through
  • moment
  • gulp-util
  • ember-cli-babel
  • semver
  • minimatch
  • node-uuid
  • core-js
  • rimraf
  • yargs
  • mime
  • qs
  • sax
  • promise
  • supports-color
  • mongodb
  • wordwrap
  • clone
  • es6-promise
  • fs-extra
  • handlebars
  • socket.io
  • winston
  • ansi-styles
  • iconv-lite
  • js-yaml
  • mongoose
  • ws
  • postcss
  • grunt
  • es5-ext
  • resolve
  • underscore.string
  • mime-db
  • abbrev
  • graceful-fs
  • when
  • process-nextick-args
  • marked
  • balanced-match
  • form-data
  • pkginfo
  • aws-sdk
  • combined-stream
  • gulp
  • event-stream
  • pinkie
  • inquirer
  • delayed-stream
  • co
  • less
  • dateformat
  • is-typedarray
  • methods
  • backbone
  • ember-cli-htmlbars
  • validator
  • forever-agent
  • json5
  • http-signature
  • bson
  • d
  • merge
  • which
  • traverse
  • joi
  • clean-css
  • npm
  • is-absolute
  • mustache
  • should
  • eventemitter2
  • bignumber.js
  • vinyl
  • mongodb-core
  • eventemitter3
  • jsonfile
  • formidable
  • log4js
  • css-select
  • argx
  • assert-plus
  • http-proxy
  • big.js
  • ncp
  • bower
  • is
  • request-promise
  • iftype
  • nomnom
  • cli
  • rx
  • should-type
  • eyes
  • string
  • restler
  • nconf
  • native-or-bluebird
  • highlight.js
  • mout
  • tap
  • js-beautify
  • faye-websocket
  • moment-timezone
  • got
  • ee-class
  • sprintf
  • websocket-driver
  • hogan.js
  • typescript
  • domutils
  • lodash-node
  • websocket
  • pako
  • ip
  • globule
  • pseudomap
  • npmlog
  • yallist
  • inflection
  • user-home
  • prelude-ls
  • jsonwebtoken
  • levelup
  • window-size
  • sinon
  • generic-pool
  • verror
  • ssh2
  • vinyl-fs
  • websocket-extensions
  • csv
  • global
  • eslint
  • react-tools
  • hyperquest
  • hubot
  • vows
  • cycle
  • validate.io-array
  • archiver
  • amqp
  • step
  • statuses
  • pegjs
  • object-keys
  • xregexp
  • ipaddr.js
  • jsonparse
  • boolbase
  • optionator
  • sshpk
  • argv
  • vow
  • nunjucks
  • long
  • jszip
  • utile
  • elasticsearch
  • color
  • babel-preset-es2015
  • virtual-dom
  • change-case
  • yamljs
  • hashish
  • pluralize
  • deepmerge
  • traceur
  • charenc
  • crypt
  • crypto-js
  • es6-shim
  • osenv
  • run-async
  • leveldown
  • log
  • gulplog
  • level
  • kew
  • dot
  • void-elements
  • typpy
  • highland
  • check-types
  • bops
  • hat
  • benchmark
  • xhr
  • base62
  • normalize-package-data
  • readdirp
  • buffer-crc32
  • should-format
  • should-equal
  • keygrip
  • arrayreduce
  • sugar
  • wiredep
  • axios
  • jasmine-node
  • istextorbinary
  • duplexify
  • jstransformer
  • yeoman-environment
  • csv-parse
  • jsbn
  • pause-stream
  • color-convert
  • walk-sync
  • html-wiring
  • log-symbols
  • class-extend
  • through3
  • microtime
  • source-map-resolve
  • ee-log
  • he
  • yaml
  • xmlrpc
  • docopt
  • shimmer
  • big-integer
  • lazy
  • cookies
  • tv4
  • lie
  • dustjs-linkedin
  • phantom
  • ee-types
  • base64url
  • required-keys
  • yeoman-assert
  • mpromise
  • object-path
  • package
  • yeoman-welcome
  • resource
  • samsam
  • asn1
  • colors.css
  • mpath
  • protobufjs
  • mz
  • fis
  • rx-lite
  • crc
  • deferred
  • lodash.isarray
  • knex
  • memcached
  • queue-async
  • lodash-compat
  • ssh2-streams
  • jwt-simple
  • cradle
  • flat
  • blessed
  • signals
  • min-document
  • debounce
  • mquery
  • strftime
  • muri
  • slug
  • plist
  • ul
  • walkdir
  • color-string
  • color-name
  • expect.js
  • node-int64
  • interpret
  • micromatch
  • define-properties
  • hooks-fixed
  • protoclass
  • gulp-sourcemaps
  • vue
  • exec-sh
  • unist-util-visit
  • wd
  • node-forge
  • coffeeify
  • url-parse
  • cylon
  • sift
  • extended
  • harmony-reflect
  • pouchdb
  • knockout
  • eco
  • gh-got
  • platform
  • cuid
  • koa-router
  • node-fs

There aren't 1000 packages tested because the rest had various issues like being Windows-only, or failing to build on my machine, etc. But I think this sample should be representative.

@ljharb
Copy link
Member

ljharb commented May 19, 2020

@GeoffreyBooth it would be interesting to also get results that ignore __esModule, and separately, that take __esModule into account to indicate whether it has named exports or not. I suspect those numbers would look much better with either of those variants, and would love to see my intuition checked.

@GeoffreyBooth
Copy link
Member

@GeoffreyBooth it would be interesting to also get results that ignore __esModule, and separately, that take __esModule into account to indicate whether it has named exports or not. I suspect those numbers would look much better with either of those variants, and would love to see my intuition checked.

You're welcome to check out the repo and modify it however you wish. Please let me know what you find. I already excluded exports named default as those obviously wouldn't be allowed as names of exports in ESM. I think the PR itself includes special handling of __esModule.

@coreyfarrell
Copy link
Member

A concern I have with the handling of default when __esModule is truthy is that this results in a breaking change for node.js ES module consumers of transpiled CJS modules.

import pkg1 from 'pkg1';

pkg1.default('call the default export');
pkg1.named1('call the named1 export');

The above is the current way to import CJS that babel produced by transpiling ESM.

@weswigham
Copy link

But conversely, there's no package author who wanted to author a default export, have it be transpiled, and then reimported with a default import that you then, further, needed to say .default off of. The __esModule marker is unmistakably an opt-in to have the default on the namespace object be the module default (rather than the module as a whole). This is, unmistakably, the intent of the marker. Modules are still experimental in node, and I'd be happy to see them respect the marker (alongside other named exports) before they stabilize. It would be a good compatability olive branch.

@ljharb
Copy link
Member

ljharb commented May 20, 2020

@coreyfarrell that doesn't seem right; if the CJS was produced from babel, importing its default from babel will be module.exports.default (because module.exports.__esModule exists)

@coreyfarrell
Copy link
Member

@wesleytodd that's fair. I know this will break a couple imports I'm using in private code but it will make that code less ugly.

@ljharb I was referring to using node.js ES module loader to import a package produced by babel (not using babel in my own package).

@ljharb
Copy link
Member

ljharb commented May 20, 2020

Ah, the you’re right it would be a breaking change, but modules are experimental, and imo if a CJS module has a “.default” property i consider that a bug :-) since its exposing Babel as an impl detail.

@dfabulich
Copy link
Contributor

  • 33% of packages (305 of 936) had no CommonJS named exports detected.

When I read this bullet, I thought that meant "the automagical parser detector failed in 33% of the cases." But it turns out the situation is much better than this!

I noticed that lodash is at the top of the list of "Packages with no CommonJS named exports detected," as if lodash were a "failed" test case.

lodash doesn't use named exports! The detector worked perfectly when it found that lodash had no named exports. lodash just uses module.exports = _. (With some platform-sensitive indirection thrown in.)

Today in Node 14.3, import _ from 'lodash' works, just as you'd expect, and import {add} from 'lodash' doesn't, but as far as I know, that's not even supposed to work. Even if lodash had been implemented in pure ESM using export default {add}, import {add} from 'lodash' would throw, and is supposed to throw.

So I started manually poking around the top 10 packages with "no named exports detected": lodash, request, underscore, chalk, glob, colors, bluebird, q, mkdirp, and through. I was skimming quickly, but as far as I could see, none of them had named exports. These are all successes for this PR, not failures.

I suggest that this test be rewritten to distinguish four cases:

  • CJS libraries that only use default exports (lodash would appear here)
  • All named exports detected (async would appear here)
  • Some named exports detected (debug would appear here)
  • All named exports undetected (I couldn't find any libraries that fit this bill in a few minutes of testing)

@GeoffreyBooth
Copy link
Member

GeoffreyBooth commented May 28, 2020

I noticed that lodash is at the top of the list of "Packages with no CommonJS named exports detected," as if lodash were a "failed" test case.

It's about user expectations. For example, this works today:

const { shuffle } = require('lodash');
// shuffle is a usable function

So therefore, users would expect to be able to refactor into this:

import { shuffle } from 'lodash';
// throws

Because this import statement doesn't work, either in current Node or under this PR, I consider it a failed test case. Anything that users can destructure from a CommonJS import (the first example above) should be detected as a named export.

@ljharb
Copy link
Member

ljharb commented May 28, 2020

@jdalton is it your intention that lodash have named exports?

Author intention is what I think matters here, not user expectations.

@devsnek
Copy link
Member

devsnek commented May 28, 2020

lodash is meant to have named exports, and I would imagine many of the others are as well.

@ljharb
Copy link
Member

ljharb commented May 28, 2020

@devsnek i don't see anything on its readme that implies that whatsoever, and its use of module.exports = also suggests that's not the case.

@devsnek
Copy link
Member

devsnek commented May 28, 2020

@ljharb lodash-es exports them as named not as a default object. we can get a weigh-in from @jdalton though.

@devsnek
Copy link
Member

devsnek commented May 28, 2020

oh you already pinged him, I missed that. guess the app didn't update the comments properly :(

Copy link
Contributor

@mmarchini mmarchini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach has lower overall accuracy than #35249, so I object on this landing and strongly believe #35249 should land instead.

@guybedford
Copy link
Contributor Author

Closing as #35249 has now landed here. Thanks so much everyone for the feedback on this feature.

@guybedford guybedford closed this Sep 29, 2020
@MylesBorins MylesBorins removed modules-agenda Issues and PRs to discuss during the meetings of the Modules team. tsc-agenda Issues and PRs to discuss during the meetings of the TSC. labels Sep 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
esm Issues and PRs related to the ECMAScript Modules implementation. lib / src Issues and PRs related to general changes in the lib or src directory.
Projects
None yet
Development

Successfully merging this pull request may close these issues.