-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: Utility functions for the steps of resolution and loading #26
Comments
So to paraphrase, basically what you’re asking for is a hook within Node’s resolution algorithm, to inject custom logic to add I understand why you’d want this, because the full resolution algorithm is daunting to reimplement. But I think that’s exactly what popular tools like In the near term, I think that’s the path you’ll need to take; we have enough on our plate at the moment trying to complete the loaders roadmap, and providing utility functions for things that can be (and already are) implemented in the ecosystem should naturally be the lowest priority. That’s not to say that such utilities aren’t worth providing in Node core; we already provide Here are some references to catch you up on related discussions that have happened so far (deep linking to my own comments to spare you from reading the entirety of these long threads, and because I think I summarized some consensuses in those comments):
Also, have you tried simply checking for existence of // This code block is untested
import { access, constants } from 'node:fs/promises';
export async function resolve(specifier, context, defaultResolve) {
// For specifier './file', see if './file.ts' exists
try {
const inferredSpecifier = new URL(`${specifier}.ts`, context.parentURL);
await access(inferredSpecifier.pathname, constants.R_OK);
return { url: inferredSpecifier.href };
} catch {
return defaultResolve(specifier, context, defaultResolve);
}
} |
I want to clarify for anyone else reading this: ts-node is not technically reimplementing node's resolver. We ship a copy of node's source code, with a minimal patch applied to essentially expose the API being described here. Ideally node would provide this API so we don't need to ship a copy of logic that is already within node's binary. This is not to say I disagree about priorities or time constraints, but there's also the risk that this use-case becomes harder than it needs to be because it's not considered in earlier loader design work.
I think it'll miss stuff like when package.json "main" points to |
To support this with another example, Yarn PnP as well doesn't want to replace the full resolver. We only want to inject into a very limited part of the resolution (package name to directory), and it's the way Node currently works that requires us to copy a (much) larger part of the Node logic. This isn't helped with Node's internals sometimes relying on hidden bindings that not even loaders have officially access to (cf nodejs/node#39513). |
@GeoffreyBooth Thanks for the reply! :-). Yes you are correct that is what I'm looking for. I currently can see two main ways it could be implemented. Either as options to the default loader or as a separate API. Your example would work for relative specifiers but not for bare specifiers? For bare specifiers currently you need to re-implement the full ESM algorithm? To give an example of what I mean as options to default loader it could be something like this (which probably is not a good idea): export async function resolve(specifier, context, createDefaultResolve) {
const defaultResolve = createDefaultResolve({ extensions: [".ts", ".js"]});
return defaultResolve(specifier, context, createDefaultResolve);
} EDIT: To clarify, the above is intended as a programmatic and more flexible version of However, I think an API would be more useful so you can compose your own resolve from that without having to fully re-implement all of it. I did not fully follow if the discussions you linked are about such an API or are they about something else? I guess I need to read the full threads after all :-).
I totally understand about the priorities. It is sure possible to re-implement the algorithm so perhaps I will do that but just wanted to check if there was an easier way. I'm not aware of the ESM algorithm being implemented in the ecosystem yet? There is an issue on the resolve repo but that seems to not have gotten far yet: As mentioned above ts-node is maintaining a full copy of the whole ESM resolver from node's code with some tweaks (to solve what I mentioned above). |
For what it's worth, the relevant files are in https://github.com/TypeStrong/ts-node/tree/main/raw For example, you can |
At some point recently, we did discuss exposing Or do you mean you'd want it exposed from some builtin package, like Node's module builtin? |
My first instinct is to create publicly accessible versions of many of the functions in https://github.com/nodejs/node/blob/master/lib/internal/modules/esm/resolve.js, which I guess is more or less what https://github.com/TypeStrong/ts-node/tree/main/raw?rgh-link-date=2021-09-12T22%3A14%3A08Z is doing. This would correspond with the steps in the algorithm, listed at https://nodejs.org/api/esm.html#esm_resolution_algorithm. So then within your |
Yes this is exactly what I meant. Having a fine grained API with functions that does a little part each so you can compose your own loader from that and interject whatever logic you want in between, or skip/replace steps, without having to re-implement the full algorithm. |
Okay, perhaps we should rename the issue (and reword the initial post)? And this can be a feature request. Once you can compose your own |
I think there might be 2 separate issues here, the second being:
You'll need a custom resolve hook to return a (I realise, Andrew, you are quite familiar with TypeScript; including extra detail for others) In both examples you will suffer the same problem forced by the TypeScript team. The former is the correct usage; however, the latter was the lesser-of-two-evils (as well as a popular corner-cut) at the time as a workaround to avoid the TS team's decision to disallow TypeScript file extensions in import specifiers. For the first example, that will work just fine because it's factually accurate (the file being read literally does contain the extension I think relying on
I think this is a promising avenue. We've talked about doing that for at least one of them already; @cspotcode @jonaskello do you need all of them or just a subset? |
Another couple cases that I need to support:
(In this example, the user has followed my recommendation and their code is When we need to Ideally, I imagine we could do something like this pseudocode:
|
I will have to think about this. We need We also need |
See the updated CoffeeScript loader example: https://github.com/nodejs/node/blob/master/doc/api/esm.md#transpiler-loader. We added a new helper function that’s probably a better solution in general than the extension-replacement hack. I think there’s a good argument for this helper being part of core, since if it’s in core it could reference the cached |
Currently our duplicated I forget, will an http loader composed with a TS loader be able to respect |
Exposing that cache would be incredibly useful in general; it'd be great to get something like that shipped and backported to node 12 if possible! |
That’s what I was getting at with the links above in my first comment, which discuss various needs for this data and ideas for APIs for exposing it. Providing easy access to the metadata around a file or a package is useful in general, for a variety of use cases. It’s a little tricky in practice, as there are potentially multiple
This is all hypothetical, since Node itself doesn’t load any modules over HTTP; that would be up to however you design your HTTP loader. The example one in the docs just assumes all JavaScript via HTTPS is ESM. I would think that if you want to get the “type” of JavaScript loaded over HTTP, the standard way to do so (along the lines of how browsers do things) is to look at the MIME type via @cspotcode @jonaskello @arcanis please see (and reply to) #27 (comment), I’m trying to reschedule the meeting where we might discuss this. |
@GeoffreyBooth seems like an API that took a path, and returned the "closest" package.json (perhaps, with an options bag where you could indicate you want a "package" or not), would be super useful? |
I noticed PR 44501 in the typescript repo where @weswigham is working on node12 and nodenext module resolution for the typescript compiler. I have not looked deep into that code but I would suspect it is re-implementing a large part of the the resolve algorithm. Perhaps parts of the API discussed here could be helpful for future versions of the typescript compiler too? |
Specifically I'm thinking about two loaders playing together: how might an http loader integrate with a TS loader. Supposing the HTTP loader is interested in allowing package.json to reside on a remote server, and the TS loader is interested in respecting fields from within the package.json to affect module resolution and classification. The TS loader wants a generic way to get package.json but doesn't care where it comes from. The HTTP loader wants to provide package.json from http sources but doesn't care how it's used. Using the TS API as an example, it has reusable resolver implementations that can do tasks involving filesystem traversal, but the underlying filesystem is pluggable: you provide a I think this'll come up when designing loader composition but maybe it's good to think about now, too.
TS needs to run in non-node environments, so they unfortunately cannot rely on any node APIs. They have an implementation of |
Yeah, @weswigham has clued us in about the TypeScript filesystem resolution algorithm in the past. Feel free to read up more on that at nodejs/help#2642 (comment). We may need to implement pieces of that algorithm in core as well since we probably still have unintelligible error messages for failed resolutions on our supported host OSes. I do like the idea about being able to provide |
Are you referring to one of the |
@guybedford, if I can loop you in, what do you think of #26 (comment)? Since you wrote many of those functions and designed the overall ESM resolution algorithm. |
Regarding which functions I would need in the API. My goal is to be able to resolve typescript files, not just in simple cases for absolute/relative files, but in more complicated cases where you have import { foo } from "@myapp/pkg-a/lib/foo.js"; Now the source for the file being imported above lives in import { foo } from "@myapp/pkg-a"; And in I've done some initial experiments with the code in If there was an API of helper functions for a loader, I think ideally they should be able to be called without hitting the filesystem. One reason is for typescript as I stated above about the files not being compiled yet so if the resolution tries to probe for them there will be an exception. Another reason may be if you don't want to lookup Currently I'm experimenting with some refactorings to the
Probably I don't need all of them in the end. The problem I encountered now is that I'll continue to experiment a bit and see where it takes me. |
I'm wondering if |
Yes, I also think subpath imports could replace For |
Ah, sorry, I meant
I have not used |
I think, but I'm not sure, that I don't think workspaces describe that I didn't mention Caveat that I haven't read the entire thread above, so maybe I'm retreading previous conversation. |
Yes, Additionally |
Jumping in to this conversation pretty late, but I did implement a TS Loader without the need for implementing module resolution. You can see the specific lines that did it here: https://github.com/giltayar/babel-register-esm/blob/eeb9a79c7646c7ee38e0cbb64de85b0548057076/src/babel-register-esm.js#L27. (It's been working pretty well at our company for the past half year or so. No gotchas found.) The idea is that if the In essence, Am I missing something? Is this a bad idea? |
@giltayar Yes that works for simple cases. For example esbuild-node-loader also takes a simple approach. I don't think there is something bad about this if that is all you need. The problem is that it does not work in more advanced cases. The two specific cases that I know of are:
Also you will probably get worse performance if you first scan for |
I had a similar problem, I needed to point directly to the source code instead of the bundle dist when using a local bundle in a monorepo to avoid a lot of initial builds at the beginning, and also to debug the code Update, I solved my problem with plugin codeimport { createRequire } from 'module'
import { Plugin, ResolveIdResult } from 'rollup'
export function tsAlias(options: {
includes: (string | RegExp)[]
excludes?: (string | RegExp)[]
debug?: boolean
}): Plugin & {
enforce: 'pre' | 'post'
} {
const { includes, excludes = [], debug = false } = options
return {
name: 'rollup-plugin-src-alias',
enforce: 'pre',
async resolveId(source: string, importer?: string): Promise<ResolveIdResult> {
excludes.push(/\/.*\//)
const predicate = (item: string | RegExp) =>
typeof item === 'string' ? source.startsWith(item) : item.test(source)
const isRewrite = includes.some(predicate) && !excludes.some(predicate)
debug && console.log('resolveId', source, importer, isRewrite)
if (!isRewrite || !importer) {
return null
}
try {
const res = createRequire(importer)
.resolve(source + '/src')
.replace(/\\/g, '/')
debug && console.log(`rewrite: ${source} => ${res}`)
return res
} catch (e) {
console.warn('rewrite failed: ', source, e)
return null
}
},
}
} |
Today it is possible to alter the resolve algorithm to some extent using
--experimental-specifier-resolution
. However that switch only supports a couple of fixed use-cases and is not available programmatically within a loader implementation. When implementing a custom loader it would be nice to have access to a fine grained API so we can re-use parts of the default resolve algorithm but still be able to interject whatever logic you want in between steps, or skip/replace steps, without having to re-implement the full algorithm. That way a custom loader can be composed from the API rather than have to re-implement the whole resolve algorithm.The old node resolve algorithm was quite easy to re-implement so for that we did not really need a fine grained API. The new algorithm supports more features and is therefore more work to re-implement. So IMHO it would make sense to expose an API for the steps in this more complex algorithm rather than having custom loaders re-implement or copy it in full.
The use-cases for this would be for example implementing a loader for
.ts
files in ts-node where the algorithm is the same but only differ in file extension. Also, Yarn PnP could use this for its resolve. See below for those requests.ORIGINAL POST (before the issue was re-formed as a feature request):
I'm sorry if this is not the right forum to ask this, if not pls let me know where it is appropriate to ask.
I'm curious about how the loader specification and
--experimental-specifier-resolution
interact. Specifically I'm looking into loading ES modules directly from typescript.ts
files using any specifier (bare, relative). With the PR "esm: consolidate ESM Loader methods #37468" merged, this seems to have gotten easier. From what I understand it would now be possible to have something like this:And then just implement the loader hook as the resolve hook will not throw on the unknown
.ts
extension. This seems like a nice improvement as we don't have to re-implement the full resolve anymore. However most typescript projects are setup to import without extensions, like this:I believe having this resolved is related to
--experimental-specifier-resolution
but AFAIK this only support.js
. So in order to resolve the above tofoo.ts
the only way I see today is to re-implement the whole module resolution algorithm, which with the new spec is a rather big task. I think that is what ts-node is doing here, basically copying the impl from the internal nodejs loader.I'm guessing the
--experimental-specifier-resolution=node
flag somehow augments the internal loader to check for.js
andindex.js
(I have not looked deep into the source code though). The point being I think it would be nice if a loader could be allowed to somehow do the same augmentation. Specifically allowing imports without extension to be resolved to.ts
files without having to re-implement the full resolution algorithm. Or is there some other way to achieve this?The text was updated successfully, but these errors were encountered: