import in referenced typescript project fail to load

[steabert]

Bug description

When working with composite projects, our tests failed because it seems that tsx is not taking the reference project's baseUrl or path into consideration.

Reproduction

Test project:

https://github.com/steabert/esbuild-kit-tsx-composite.git

To reproduce the issue:

yarn install
yarn fail

Running yarn bundle and yarn types shows the result with e.g. esbuild or tsc.

It seems this does not work with ts-node either, but I don't know how they handle composite typescript projects.

Environment

npx envinfo --system --npmPackages tsx --binaries
System:
  OS: Linux 5.4 Ubuntu 20.04.4 LTS (Focal Fossa)
  CPU: (8) x64 Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
  Memory: 23.58 GB / 31.21 GB
  Container: Yes
  Shell: 5.0.17 - /bin/bash
Binaries:
  Node: 16.14.2 - ~/.node/bin/node
  Yarn: 3.2.3 - ~/.npm-global/bin/yarn
  npm: 8.5.0 - ~/.node/bin/npm
npmPackages:
  tsx: 3.9.0 => 3.9.0

Can you contribute a fix?

• I’m interested in opening a pull request for this issue.

[steabert]

Also, looks similar to #76 maybe?

[IlyaSemenov]

Yes, this is the same problem as reported in #76. unlike esbuild, tsx ignores tsconfig.json in referenced packages.

[privatenumber]

How does esbuild handle custom tsconfig.json paths via the tsconfig CLI flag for multiple projects?

e.g. If project A uses tsconfig.a.json and its dependency project B uses tsconfig.b.json, how is it configured?

[steabert]

I don't need to specify any configuration, esbuild seems to find the correct tsconfig associated with an imported module. However, I think it only does that when bundling. I forgot where I read that, I think it was in the comments on one of the esbuild issues, I'll try to find it again.

Edit: here was the comment that I found saying what the behaviour is, but nothing in detail how it's resolved.

[steabert]

My own (naive?) way would be to walk up the directory tree of an imported file if there is no associated tsconfig, and if a new tsconfig.json file is found, check its include files and cache it for later use. That way, each included file would have an associated tsconfig file.

[privatenumber]

My question is more about API design than your specific use-case.

Understanding how esbuild's API handles custom tsconfig.json paths across multiple projects would help.

[steabert]

The esbuild author replied in my issue, but that seems to be also about how esbuild itself finds tsconfig files. That doesn't seem to be the info you're after, I'm afraid I haven't dealt with the esbuild API at all.

[privatenumber]

To be fair, you asked him a different question from the one I asked you.

I don't have time for this now, but if you're interested in helping, you can investigate by setting up a project with the structure I described here and trying to make it work with esbuild. Maybe @IlyaSemenov can help.

Based on what Evan said, I'm skeptical if esbuild is also following TS behavior correctly (not that tsx currently is either). For example, does it even allow for custom tsconfig.json paths? And does it check against tsconfig.json#files/includes/excludes?

[mtqperson]

We are also running into this problem. I might be able to help but I'm not sure what you mean by

you can investigate by setting up a project with the structure [...] and trying to make it work with esbuild.

How does esbuild handle custom tsconfig.json paths via the tsconfig CLI flag for multiple projects?
e.g. If project A uses tsconfig.a.json and its dependency project B uses tsconfig.b.json, how is it configured?

Like @steabert described, with esbuild it already works because esbuild detects tsconfig.b.json by itself. Do you want to know whether it still works if we additionally specify tsconfigs manually, even though that's not required?

[privatenumber]

I'm skeptical that esbuild would successfully build. My understanding is that it only detects tsconfig.json, and not tsconfig.b.json.

To prove that this works, please create a repository containing a monorepo with two tsconfigs under custom names and add a build script to demonstrate that esbuild can successfully build it. If you'd like, you can reuse https://github.com/steabert/esbuild-kit-tsx-composite but rename the tsconfig.json of the dependency and test if it builds.

And if that doesn't build, please find out what configuration esbuild offers to specify a custom tsconfig.json name.

[IlyaSemenov]

This issue is a blocker for any monorepo with aliases, which is (arguably) more popular setup than multiple custom named tsconfig's. Also, the current behaviour is not in sync with how esbuild works itself. In my reproduction repo tsx fails to run yet esbuild compiles it just fine:

❯ p esbuild test.ts --bundle > test.js

❯ cat test.js
"use strict";
(() => {
  // ../b/lib.ts
  console.log("imported lib");
})();

❯ node test.js
imported lib

I would like to try to come up with a patch as I'm not happy with my current setup (nodemon + esbuild-runner) and looking for alternatives. @privatenumber do you think you can point me in the right direction, if you foresee any?

[privatenumber]

We should figure out how custom tsconfig paths are handled in nested projects: #96 (comment)

[IlyaSemenov]

I am not following why the custom configs matter so much? It would suffice if nested projects worked at least with default configs. Supporting custom configs in nested projects could be the Phase 2 implemented by whoever cares about that use case.

Chances are esbuild doesn't support custom configs in nested projects at all, then what? That doesn't prevent it from working with default configs; and neither should it prevent tsx from supporting the same case.

[privatenumber]

I am not following why the custom configs matter so much? It would suffice if nested projects worked at least with default configs.

Why it matters: #76 (comment)

tsx has a lot of users and handles many use-cases. Custom tsconfig files are very common so I think the more appropriate question you should be answering is "why should we dismiss those use-cases without even doing an hour of research?".

Supporting custom configs in nested projects could be the Phase 2 implemented by whoever cares about that use case.

FYI, I don't benefit from this use-case but I'm happy to implement it in the future. With regards to your attitude, I'm wondering why you haven't implemented this yourself? As with anything open-source, you should also feel free to implement it yourself and use your own fork.

Chances are esbuild doesn't support custom configs in nested projects at all, then what? That doesn't prevent it from working with default configs; and neither should it prevent tsx from supporting the same case.

That's fine too. But no one has provided a definitive answer with research.

When I decide to take this on, I would like to think of the best solution possible and would investigate different use-cases and solutions. Since ya'll are asking how you can help, I provided one of the questions I would research.

---

I'm not sure how this thread will be productive if my answers to your "how can I help?" is met with "why is that important?".

If no one is going to provide insight into potential problems and how other projects handle it, it might be best to lock this thread until I'm ready to tackle it and do the research myself.

[IlyaSemenov]

Thank you for the response. I apologize if I sounded rude. I know how open-source works, I authored and support a number of npm and python modules myself, and I am happy to provide users the right direction if they ask (or point them where they're wrong).

My point here was that making fully broken system half-broken would still be a good thing. I gathered that the current implementation blocker was that it wasn't clear how to configure esbuild to use different tsconfig's in different sub-projects. In my thinking that should not be a blocker even if that's not possible.

I did make the initial research back then when you asked the first time and it immediately appeared that the public esbuild API only allowed to customize the root/entrypoint tsconfig: https://esbuild.github.io/api/#tsconfig
I didn't share that finding as it appeared too obvious for me.

In light of that, I was merely asking if you could provide your initial high level thoughts of where the source could be extended to load paths from nested tsconfigs (considering the limitation of not being able to use custom-name tsconfig), outline the frames of the possible required architectural changes. I surely understand it's not always possible to answer that kind of questions, but sometimes it is.

[jgoux]

Hello everyone 👋,

I also hit this issue. There is no way to specify custom tsconfig.<whatever>.json config files for external packages bundled with esbuild. It only looks for the regular tsconfig.json.

As far as I understand, just supporting the regular tsconfig.json files would solve my issue (I'm using a pnpm monorepo, I have the regular tsconfig.json for vscode + dev workflow, and a tsconfig.build.json for build). For that, we should not pass an explicit --tsconfig option when calling esbuild, as it will use this single tsconfig file and ignore all the others.

So paths defined in tsconfig.json can work but in a limited way (every tsconfig files need to be named tsconfig.json, including the root one).

Do you think we could add an option to enable this behavior? Maybe --resolve-tsconfigs or something like this?

I'm happy to make a PR if you can point me to the relevant files! 😄

[jgoux]

I'm very sorry for this triple post, but I played some more with my TypeScript monorepo and I think I gained interesting insights for this issue.

Big picture

tldr; TypeScript project references are the way to hint about which tsconfig.json file to apply to parts of the dependency chain

First, the "one true way" of connecting TypeScript projects together is project references. This is the only mechanism that enables TypeScript to split the dependency chain between multiple TypeScript projects and use each project's tsconfig.json.

Custom tsconfig.json files are handled by this mechanism as well:

{
    "compilerOptions": {
        // The usual
    },
    "references": [
        { "path": "../api/tsconfig.build.json" },
        { "path": "../sdk/tsconfig.whatever.json" },
    ]
}

Without references, if you require a TypeScript file from an external package (through either monorepo workspaces or paths alias, I personally recommend avoiding paths alias as you break your package isolation), only the tsconfig.json file of the root package will be used, this is why external packages paths alias break, because the root package is not aware of their internals (and it should not be 😄).

How to have this implemented in tsx

Project references were discussed in esbuild repository, and the conclusion was that it's beyond the scope of esbuild, but could be implemented as a plugin!

A plugin exists: esbuild-plugin-ts-references but it's not working on paths alias, it's only rewriting the import statements to link directly to the references. But it shows that it's possible with esbuild API to walk through each package's tsconfig.json by following references and build some kind of map.

So the solution would be to write an esbuild plugin similar to esbuild-plugin-ts-references that can rewrite imports based on the file's local tsconfig.json gathered through references analysis. The main blocker is that tsx is using esbuild's Transform API and not the Build API, so it can't consume a plugin.

@privatenumber How hard of a requirement is it to use the Transform API vs the Build API? Reading esbuild doc it seems like Transform is just a small subset of Build?

Another solution would be to have tsx do the references analysis work outside of esbuild to build the map and then apply the rewrites somehow with the Transform API, but I'm not familiar with it enough to know if it's possible. 😅

edit: It seems like if we can construct the map <tsconfigPath, filePaths>, we could be able to pass the right tsconfig.json content here: https://github.com/esbuild-kit/cjs-loader/blob/94a69f9e6d41225cb8a132938a3b9b077c15f966/src/index.ts#L74

edit 2: I gave my "projects map" idea a try, I think I'm pretty close to making it works but I have a nasty recursive bug somewhere that I can't figure out. 🤔

But I'm able to :

• Resolve paths alias correctly using the right tsconfig
• Use the right tsconfig file given a file path when calling esbuild transform

Please note that it's not "performance" oriented, right now I'm just trying to understand the code and make it works. 😅

Here are the forks :

• https://github.com/jgoux/cjs-loader/tree/feat-project-references
• https://github.com/jgoux/esm-loader/tree/feat-project-references

And I test against this project:

• https://github.com/jgoux/trpc-internal-types-inference, I use pnpm override linked to both my forks. Then you need to run pnpm --filter cli dev

edit 3: My recursion bug is fixed and the code for https://github.com/jgoux/esm-loader/tree/feat-project-references is fully functional! 🎊

[jgoux]

I opened PRs to esm-loader and cjs-loader to support project references which should solve this issue.

@steabert @IlyaSemenov If you can give it a try it would be great! 👍