How to convert package-lock.json to import maps?

[billiegoose]

This looks so cool! Are there any implementations of "package-lock.json -> package-name-map" translators yet? (Is that what I should be thinking of?)

(Edit by @domenic: at the time this comment was written the proposal was known as "package name maps". I've renamed the title to be "import maps", but left the body of subsequent comments alone. Sorry if that's confusing!)

[adrianhelvik]

Something like this should do it. You would use package.json and not package-lock.json.

const fs = require('fs')

const packageMap = fs
  .readdirSync(
    __dirname
    + '/node_modules'
  )
  .reduce((res, name) => {
    try {
      var package =  require(folder + '/package')
    } catch (e) {
      console.log(folder + ' had no package.json. Skipping...')
      return res
    }
    if (! package.module) {
      console.log(`"${name}" had no module filed. Skipping...`)
      return res
    }
    res.packages[name] = {
      main: package.module
    }
    return res
  }, {
    path_prefix: '/node_modules',
    packages: {}
  })

fs.writeFileSync(
  __dirname + '/package-map.json',
  JSON.stringify(packageMap, null, 2)
)

[tilgovi]

I would think you would want to package-lock.json unless you can guarantee that all dependencies are hoisted and totally flat.

[adrianhelvik]

Ah, I didn't consider transitive deps.

[adrianhelvik]

Came to think of it. Do packages support a packages field? This would be crucial for node interop.

Otherwise this will not be supported:

/node_modules
  /some-lib
    /node_modules
      /jquery (v2)
  /jquery (v3)

[billiegoose]

Exactly, package maps cannot be generated directly from the "dependencies" field of package.json because they only specify the first-order dependencies and even those are only by a semver range.

(Edit: and crawling the filesystem is fraught with edge cases like symlinks. You'd have to use module.require.resolve for sure. It will quickly become ugly and complicated to try to generate a package-name-map from the filesystem. Which is why we should piggy-back on tools that take care of all that for us and parse the lockfiles they generate.)

The equivalent to a package-name-map is a "lockfile" format, which is usually created during an actual install.

I suggested package-lock.json because it's the format used by npm and therefore ought to be a defacto standard. (The npm CLI automatically produces a package-lock.json after nearly any operation.) However, for maximum developer happiness and we'd want to be able to convert:

• package-lock.json
• npm-shrinkwrap.json (same format, different use case)
• yarn.lock (generated by yarn)
• shrinkwrap.yaml (generated by pnpm)

So that's at least 3 CLI tools that need to be built, I take it?

[tilgovi]

The scopes feature allows what you're looking for, but see also #5.

[daKmoR]

also Yarn Plug'n'Play: Implementation should be considered - It comes with it's own file/api

More Info here: yarnpkg/yarn#6382

[billiegoose]

Ah! very interesting. This pnp thing ships with a build-pnm (build package-name-map) tool as of last month. yarnpkg/pnp-sample-app@c36baa5

[daKmoR]

oh yeah, that looks promising... unfortunately not yet useful for web components.
I made a test with what is currently available

https://github.com/daKmoR/build-pnm-webcomponent-issue

I raised an issue there as well - hopefully, this will start a discussion about unique elements in a package name map. For anyone who is interested arcanis/build-pnm#1.

[arcanis]

Thanks for the ping! I'm entering a plane but I'll share some points regarding the build-pnm utility later today!

In the meantime, can you details the reasons why webcomponents are expected to be flat, with a single version used accross an application? Since the pnm standard seems to support nested packages I wonder if there is a technical requirement I've missed in the spec 😃

[daKmoR]

comment moved to: arcanis/build-pnm#1

[billiegoose]

WebComponents is a significant change of topic from my original issue, which is asking what tools exist for creating package-name-maps. I would move that discussion to a new issue so as not to get them mixed up.

[domenic]

Looking forward to what people produce here. I'll note that the proposal has just changed radically (and been renamed), in ways that might make this more interesting. As such let me rename the issue.

[CarterLi]

Generates import maps from yarn.lock

import * as lockfile from '@yarnpkg/lockfile';
import { promises as fs } from 'fs';
import * as path from 'path';

export async function getImportMap(targetPath = __dirname) {
  const content = await fs.readFile(path.resolve(targetPath, 'yarn.lock'), 'utf-8');
  const json = lockfile.parse(content);

  return Object.assign({}, ...Object.keys(json.object)
    .map(x => x.slice(0, x.lastIndexOf('@')))
    .map(x => {
      try {
        const result = '/' + path.relative(targetPath, require.resolve(x, { paths: [targetPath] }));
        return { [x]: result, [x + '/']: path.dirname(result) + '/' };
      } catch {
        return { [x]: undefined } ;
      }
    }));
}

A working demo that uses import maps in browser with node_modules: https://github.com/CarterLi/web-esm/blob/master/index.ts#L27

[ftaiolivista]

Generates import maps from yarn.lock

import * as lockfile from '@yarnpkg/lockfile';
import { promises as fs } from 'fs';
import * as path from 'path';

export async function getImportMap(targetPath = __dirname) {
  const content = await fs.readFile(path.resolve(targetPath, 'yarn.lock'), 'utf-8');
  const json = lockfile.parse(content);

  return Object.assign({}, ...Object.keys(json.object)
    .map(x => x.slice(0, x.lastIndexOf('@')))
    .map(x => {
      try {
        const result = '/' + path.relative(targetPath, require.resolve(x, { paths: [targetPath] }));
        return { [x]: result, [x + '/']: path.dirname(result) + '/' };
      } catch {
        return { [x]: undefined } ;
      }
    }));
}

A working demo that uses import maps in browser with node_modules: https://github.com/CarterLi/web-esm/blob/master/index.ts#L27

Would be great to filter out development packages and generate map only for production ones.

[CarterLi]

Would be great to filter out development packages and generate map only for production ones.

It's easy to filter out development packages themselves, but it's hard to filter out packages that dev packages depend on.

I dont know if yarn has an API to do so, or you will have to parse package.json in node_modules recursively

[ljharb]

package-lock already contains this information; anything with dev: true is something that's a dev dep, or a transitive dep of a dev dep that's not a transitive dep of a production dep (iow, safe to prune)

[dgreene1]

Would anyone be interested in turning the two snippets above into a CLI library so that we can have all the benefits of an open source package?

(I.e. modularization, the package can be upgraded centrally, etc)

[adrianhelvik]

@dgreene1 Agree that we need that, but it can't really use require.resolve. Now it works relative to the file and not cwd.

[ljharb]

the resolve package allows a custom basedir; you could use that?

[daKmoR]

hey there 🤗

we released a package that generates a import-map.json and/or injects it into your index.html based on your yarn.lock. It is on npm as @import-maps/generate.

for now, it only supports yarn.lock files (including yarn workspaces for mono repos ✨ ).

There is currently only one mode we call "flat" which flattens all your production dependencies and asks on the command line if no version can be found to can satisfy all needs. If anyone wants to work on a "nested mode" with scopes join the party on github.

if you just wanna try it out... just call npx @import-maps/generate in one of your projects.

This is still very early - so if you can give feedback it would be highly appreciated 🤗

[dmail]

I have done something capable to generate importMap for node_modules.
The npm package is called @jsenv/node-module-import-map.
If you want to test it you can run the following inside a folder with a package.json.

npm i --save-dev @jsenv/node-module-import-map
node -e "require('@jsenv/node-module-import-map').generateImportMapForProjectNodeModules({ projectPath: process.cwd() });"

It recursively reads package.json and tries to find dependencies on your filesystem using a custom node module resolution. The generated import map will scope import per node module.
It means if you depend on lodash importMap will contain the scope below:

{
  "scopes": {
    "/node_modules/lodash/": {
      "/node_modules/lodash/": "/node_modules/lodash/",
      "/": "/node_modules/lodash/"
  }
}

It allows each module to have import starting with /.
So that inside lodash the following import

import '/src/file.js'

would be remapped to /node_modules/lodash/src/file.js inside your project.

I give the link to the github repository in case you want to check source or unit tests for instance but the readme is empty for now: https://github.com/jsenv/jsenv-node-module-import-map.

[chase-moskal]

so cool to see everyone collaborating on new tooling for import maps 🍻

trying a different approach, i've been working on a command line tool

importly generates an import map from a funny-looking importly.config file, like this

📡 unpkg, jsdelivr
📦 mobx
📦 lit-html
📦 lit-element@^2.2.0

the above importly config generates an import map that uses unpkg, but also jsdelivr is set as the fallback -- the importly cli is then used like below

importly < importly.config > dist/importmap.json

i'm currently using importly instead of npm, as a browser package manager

currently, importly doesn't integrate with npm's package.json/package-lock.json files, but that might still be a worthwhile enhancement for a hybrid workflow (importly during development, then bundling for production, perhaps) -- in which case local node_modules as a host would be a nice alternative to unpkg and jsdelivr, and is actually probably required for good local development across multiple packages (eg, npm link)

also, i think importly could be made isomorphic, so perhaps during development it could run in-browser and completely cut out the build step, which might be rather slick 🕶️

i'm open to ideas and discussion, just open an issue on my repo if you're interested :)

cheers friends!
👋 Chase

[domenic]

Just wanting to let people know that I have a pull request up at #147 to consolidate all the tools people have been posting in this thread into the README. Also, I opened a new issue #146 to provide a general discussion space, in lieu of this issue + #108. So I'll close out this issue, but see you in #146! Thanks so much everyone for the cool stuff you've been building!