Provide a means for mapping back to .ts files in published packages

[svenefftinge]

We have a mono repo with many small npm packages in it.
All of them share the same configuration in which we keep the *.ts files in a src folder and let tsc emit the *.js and *.d.ts to a lib folder.

For relative paths within a package tsc nicely resolves against the local source (*.ts) but for references across packages it resolves against the type definitions, because we use imports like:

import foo from "otherpackage/lib/module"

This makes navigation in code (find references, etc.) useless. Could the paths property in the compiler options, allow something like the following, so that tsc would resolve against the actual source?

   "paths" : {
      "*/lib/*" : ["*/src/*","*/lib/*"]
   }

Or is there another (better) way to deal with this?

[ghost]

If you are publishing a package to NPM it is best to make it so that users don't need to specify path mapping. Maybe you could compile using the --declaration option?
See the handbook on publishing.

[DanielRosenwasser]

There's a proposal for go-to-implementation in .js files at #6209, but I think you want that for the .ts files. One potential idea is using source-maps when available.

[ghost]

@DanielRosenwasser I don't see how this is related to that? This is a compile error (assuming --noImplicitAny is on and you resolve to the .js location instead of the .ts location), not a services issue.

[DanielRosenwasser]

I'm not sure what you mean. My understanding is that the request here is to provide a way to re-map the runtime import to the .ts source. Specifically, this sentence:

This makes navigation in code (find references, etc.) useless.

@svenefftinge can you confirm?

[ghost]

Ah, I was focusing on the path mapping part. So it sounds like the current compile-time behavior of getting the .d.ts file is fine, but in services it would be nicer to go to the .ts file.
Path mapping would change the compile behavior too, of course. That is probably a good thing -- it could be confusing if something was a type error when compiling but not in services, or vice versa.

Just a thought: It might make sense for a package to be published simply as lib with .js files and src with .ts files, and no .d.ts files anywhere. The compiler could use a new kind of value for "types" in package.json to know to resolve imports foo/lib/a.js to foo/src/a.ts.

[DanielRosenwasser]

One of the downsides of that approach is that a full type-check takes much longer and memory consumption grows a lot more for a project of .ts files than its .d.ts counterparts. But I guess you'd still potentially have those problems if you supported any of these scenarios.

[svenefftinge]

I would want tsc to link against the *.ts instead of the ts.d. So that language services work better across node packages.

Regarding the memory consumption, it would be nice if tsc doesn't check bodies if the signatures are complete (have return type) and the file is not part of the include.

[svenefftinge]

I actually managed to do what I want using the current paths semantics. I basically have to put in a path mapping for every of my projects. So a bit inconvenient, but it works :)

[hoegge]

I think the main request is for IDE's to be able to find the implementation done in the .ts files when you have included a library. E.g. Visual Studio 2017 can only show the definition (F12) and jumps to the .d.ts file, but there is no (easy) way to go to the original .ts file, where things were implemented (e.g. if several TS files have been combined into one js. and d.ts file using the --output option. I assume the source maps have all the info to enable you to jump to the right .ts files instead of just the .d.ts file. Right now it seems impossible to arrange a project / solution consisting of multiple projects that each produce an combined .js output and then find your way around easily in the whole solution using the "go to declaration" and "go to implementation" features.

[svenefftinge]

We figured out a way to make it work, by using two different tsconfigs in monorepo (we use lerna). One for VS Code (resolving against *.ts), one for the build (resolving against *.d.ts).
You can have a look here : https://github.com/theia-ide/theia

[JackCai1206]

I was encountering the same problem, where we used path mapping in dev, and lost typing information with the published package

[RyanCavanaugh]

Go-to-def now handles source maps, so as long as your source maps are set up (some assembly may be required), this works as hoped for.

[justingrant]

Go-to-def now handles source maps, so as long as your source maps are set up (some assembly may be required), this works as hoped for.

@RyanCavanaugh by "some assembly may be required" do you mean declaration maps? Or something else? One thing that's pretty painful in the JS/TS ecosystem is a lack of prescriptive guidance for library authors for how they should set up sourcemaps/declaration-maps properly. Maybe this has improved since the last time I went looking for it a year or so ago?

[RyanCavanaugh]

Correct; declaration maps with any necessary post-build steps (depending on your package layout) to ensure they resolve on the client.

For most libraries it's not really desirable to go to the underlying source so this isn't something we've taken time to extensively document.

[justingrant]

For most libraries it's not really desirable to go to the underlying source so this isn't something we've taken time to extensively document.

@RyanCavanaugh could you explain this further? Why is navigating to original source undesirable for most libs? How will the user know where to set a breakpoint or to dig into async/await code that's been transpiled into a confusing state machine? If the library is shipped with a source map and original sources, why would the user not want to go to original source? Seems like libs that don't want "go to original source" behavior can easily opt out by not shipping sourcemaps in the first place.

Or am I misunderstanding the use case?

Finally, if the alternative is "just show the TS declaration instead", honestly this is a bad user experience, because then the user has to go hunting around the library's code in node_modules to find what they're looking for. Doing this is complicated and time consuming for many libraries; often it's easier to just run code in the debugger because it's hard to know which of N different "index.js" files is the right one with the code that will actually execute at runtime. Navigating to transpiled code would be a better alternative, IMHO, although without declaration maps I'm not sure how that could work.

[RyanCavanaugh]

I don't know how to ascertain our disagreement here. If a library is malfunctioning in a way to its publicly-documented contract, it has a defect and I don't need to step into it from my code to investigate further. There are entire features in production debuggers dedicated to the idea that you don't want to step into library code. The same applies to navigation IMO -- anything other than the documented surface area of a library is implementation detail, which is slower to read and not something I need in day-to-day work.

[justingrant]

Ahh, got it. When Just My Code was being designed, I was working closely with the Visual Studio IDE team, e.g. I wrote the original spec for Online F1 Help. So I'm somewhat familiar with the motivations for Just My Code. What I remember was two kinds of developers that the IDE needed to support:

• Developers who were confused by library code and wanted to stay out of it. Just My Code improved things for these folks, esp. VB.NET devs who couldn't read library code in C#.
• Developers who wanted to dig into library code to understand the behavior of their program, to borrow/adapt library code for their own use, and (rarely in .NET) find and report bugs. VS did a lot to help these developers too, like auto-download of symbols and reference source so you could seamlessly step into library code when needed.

This two-track approach seems right for the JS/TS ecosystem too. Developers who want to ignore library code should be able to easily do so, and developers who want to understand, copy, and debug using library code should be able to do so. And those preferences should be respected (and easy to accomplish) whether you're navigating or debugging.

I suspect that the "want to use library code" group is larger in JS/TS than .NET because npm libraries are (on average) easier to read, easier to fork/copypaste, less-well-documented, and buggier than .NET libraries. But the exact ratio probably doesn't matter as long as we can agree that both groups are large enough to deserve first-class support.

What do you think?

[RyanCavanaugh]

It's valid feedback. I believe we already surface original/mapped location diffs to the editor in our responses, so it's a matter of editors consuming that information to present the choice to users.

[justingrant]

It's valid feedback. I believe we already surface original/mapped location diffs to the editor in our responses, so it's a matter of editors consuming that information to present the choice to users.

@RyanCavanaugh your team has done a great job getting this working! Now that the technology works, the challenge has moved downstream to getting library authors to configure things correctly to leverage the new plumbing. This is an evangelism and documentation problem, not a technical one. But it's still a hard problem!

Over the last few years, I've filed more than 15 PRs to help various library authors fix sourcemaps. This has been an uphill climb for a few reasons:

1. Many OSS library authors are JavaScript OGs who came of age before sourcemaps and IDE debuggers were a thing, so they're not intimately familiar with the IDE workflows that sourcemaps dramatically improve. They often have to be convinced why to care. This will be even more true for declaration maps which are even further removed from the day-to-day experience of a stereotypical library maintainer whose main tools are emacs for editing and console.log for debugging.
2. Some library maintainers (thankfully a minority) are actively skeptical about sourcemaps for various reasons including larger download bundles, slower builds, or bad experiences years ago with immature tools. IMO this is really (1) in disguise; these objections can sometimes be overcome when library maintainers learn about the prevalence of IDE usage among their users, even if they personally rely on OG tools.
3. Getting PRs approved for non-critical issues on popular projects is sometimes a heavy lift. I've had to wait a year or more in some cases to get PRs merged that add or fix sourcemaps. It's so much easier to get this stuff fixed in the earliest days of a library before maintainers get burned out.

One thing that would have really helped (and really will help for declaration maps) with the problems above would be canonical documentation on https://www.typescriptlang.org/docs/ that is aimed at library developers and maintainers, beyond "how to write a .d.ts file" docs that are there now. The idea would be to have one place for library maintainers to learn how to config their libs to work well for TS-using and/or VSCode-using developers.

If there's anyone on your team or who's reading this who'd like to collaborate on this content, I'd be happy to help!

Below is a possible outline.

• How and why to add sourcemaps to your library
  • What are sourcemaps, and why are they helpful?
  • How to configure TypeScript and bundlers to correctly emit sourcemaps
  • Remember to publish sourcemaps and original source too!
  • Sourcemap problems and troubleshooting
    • Sourcemaps and/or original source files not published
    • Incorrect paths to source files
    • Incorrect paths to node_modules files
    • Dependencies unexpectedly bundled into ESM libraries
    • TypeScript, node, and/or transpiler builtins
    • How to test sourcemaps
      • source-map-loader
      • (a webpack plugin I wrote to test sourcemaps)
• How and why to add declaration maps to your library
  • What are declaration maps, and why are they helpful?
  • Adding declaration maps to libraries written in TypeScript
  • Adding declaration maps to libraries not written in TypeScript
  • Declaration map problems and troubleshooting
• Writing libraries in TypeScript
  • This section isn't really related to (source|declaration)maps, but would be helpful for a similar audience to avoid pitfalls when writing libs in TS.

BTW, is there a better place to post this suggestion besides a deep-down comment in this issue? ;-)

[domoritz]

I'm running into an issue where I got declaration maps to work in my npm package but I had to put the source files next to the js, d.ts, and js.map files. Now, users have the issue that tsc picks up the ts source files instead of the js + d.ts files. The issue is that our sources only support an older version of ts and so nobody who uses a never ts version can use our 5pm package right now. I would love to guidance for library authors on how to publish a ts package with declaration maps.