Ability to reference code as a declaration (ambient)

[stanvass]

Feature proposal:

A new flag in <reference>:

/// <reference path="..." ambient="true" />

The effect of this flag would be to treat the referenced code as an ambient declaration, even if contains implementation code. The referenced code will be used for type-checking, autocompletion and all other needs by the compiler, but its code won't be included in the compiled output.

This means we don't have to compile separate .d.ts files for every .ts file when we use it in this context (the assumption is the referenced code is compiled and loaded via another channel, like a <script> tag in browsers).

Use case:

There is no lean way to separate code in modules for browser apps right now, without introducing external modules... and then using a third party solution to strip them away and compact them into JS bundles.

The redundancy and complexity of having to use two tools that mostly cancel each other out (adding modules and stripping away modules) can be avoided if we can refer to a code module as an internal module, but not include it in the produced output.

Let's say I have three browser apps on the same site. They all share a module called Lib, which is also written in TypeScript. I compile it separately and load it with a script tag. With the proposed feature, I can compile all my apps by using:

/// <reference path="Lib.ts" ambient="true" />

Without having to compile an explicit declaration for Lib.ts every time I modify it.

I tried to use the --declaration switch, but unfortunately it gives no control over where the .d.ts file is produced and how it's called (it just goes where the JS file goes). Even if it did, it'd be more cumbersome than a reference switch, because the declaration is already contained with a .ts source file, the only thing that's different is the use case. We don't need to have all those redundant files in order to fulfill the use case.

[nycdotnet]

I agree that this is a pain point. It's a decent solution.

[basarat]

The referenced code will be used for type-checking, autocompletion and all other needs by the compiler, but its code won't be included in the compiled output

How is this different from simply compiling without the --out flag. It simply compiles the files in place and seperately.

[stanvass]

@basarat The proposed feature serves to address the need to reference a library/component (made of multiple TS files compiled independently as one JS file), without including it in your output (which output is also compiling multiple TS files from your app in one JS file).

Compiling in place happens around file boundaries, not component/library/app boundaries, so it doesn't address the use case.

If we could somehow instruct the compiler (with a map, or whatever) to split output across custom defined component boundaries, it would also be a solution to the problem at hand. But I think the way I proposed it is simplest to implement, explain and understand.

[NoelAbrahams]

@stanvass,

It's not clear why one can't just generate a .d.ts for the library and include it?

(BTW: if you are using Visual Studio then the discussion #673 is also worth reading)

[stanvass]

@NoelAbrahams In my example, the module isn't a third party one, it's my own module, it's just that I'm sharing it across apps, and updating it at the same time I'm updating the apps.

So this means I have to generate a .d.ts every time I update the module. The app compiler will read that .d.ts and not my real module source, which means extra lag until my application compiler sees my updated module, because the watch now runs in two passes:

1. Module compiler sees new .ts, compiles; produces .d.ts;
2. App compiler sees new .d.ts; compiles app.

Another problem is that the compiler insists on producing the .d.ts where it produces the .js file. In any typical workflow, you want the .d.ts in your /src folder to refer to it, and your .js file in your /bin folder to load it in your browser. It's awkward we can't specify the filename and location of the .d.ts on its own. So I need to choose between:

1. A .d.ts in my /bin folder and refer to it from my source (weird); or..
2. I need to run yet another compiler pass and deal with a .js file ending up in my /src folder (weird); or...
3. Run an entire separate watch script just to put files in the right place and give them the right names (weird).

And think - if I have the module source right there in my sources, why do I need to produce another separate .d.ts file only to avoid compiling it all into the same .js file? Managing all those byproduct definition files can get really annoying in a big project - which is generated, which isn't, which goes in the repository, which doesn't.

Definition generation is very useful on its own, but in this case needing this file constantly regenerated and sitting around is simply a byproduct of a workflow that isn't addressed well, if we have to be honest.

[NoelAbrahams]

@stanvass, one of the benefits of a pre-generated .d.ts is that it's less work for the compiler.

The point being that a library is generally more stable (i.e. locked-down) than the app.

In Visual Studio for instance a solution (i.e. container for projects) can have multiple projects. Each can be compiled individually. So we would compile the library project and normally leave it alone. The app project would then reference the .d.ts.

For large projects this is essential to get the whole thing working together. It's just too much work for the compiler to have to work off raw .ts files.

[stanvass]

@NoelAbrahams I'm not talking about a third party library that's stable, or I wouldn't file this issue.

It's not uncommon for a script-driven site to have separate mini-apps for every page you load (one app per page) and a shared core. It's a very common scenario. And that shared core is part of the app just as much, so it's not frozen and left alone at all.

The reason for the split is simply so you wouldn't preload the JS of all pages at once on the user, but only the parts that are shared, or needed for the particular page.

If it's essential for large projects to precompile a .d.ts so be it, I didn't open this issue to argue against definition files, after all. But I don't know why this means let's not address small and medium projects, where using large project workflow is simply cumbersome.

Regarding performance, a .d.ts file is also a raw .ts file, all types in it still have to be parsed out and analyzed, so you might be overselling the performance benefits of using it. What you're describing is in fact just another workaround: caching compiled units in intermediate form is a common compiler feature that speeds up large project compilation. Then .d.ts wouldn't be needed for that either.

I'm not saying workarounds don't exist for this and that, but we should be open towards better solutions.

[NoelAbrahams]

I'm not talking about a third party library that's stable

I didn't mention third party libraries at all.

a .d.ts file is also a raw .ts file, all types in it still have to be parsed out and analyzed, so you might be overselling the performance benefits of using it

We initially referenced .ts files directly. The compiler just froze up.

But I don't know why this means let's not address small and medium projects

Sure. By all means.

[nycdotnet]

This is a great thread. I'm just chiming in again to say that I've hit all of the problems described by @stanvass and had to fix them with scripts and patience at compile time. TypeScript desperately needs something to help projects of medium complexity. While I love things like grunt-ts, it seems that scripts and hacks come into play too early on the project complexity curve. Imagine if MS said that for a C# project of 3 DLLs (server, client, and shared) and 2 EXEs (client and server) that you'd need custom scripts to get the client EXE code to notice when the shared DLL were updated. Nevermind the "oh, you wanted the code for that DLL split across multiple files?" issue. It'd be crazy and no one would put up with it. But that's what we have with TS today.

To bring it back on topic, @stanvass 's idea is a good one to fix the problem we have now and it would probably be fast to implement, but I really think that there is a gap here that should be addressed more holistically.

PS: I'm not trying to bash TypeScript here. This is constructive criticism of a language I love working with every day and that I whole-heartedly recommend to anyone who will listen.

[basarat]

While I love things like grunt-ts, it seems that scripts and hacks come into play too early on the project complexity curve.

@nycdotnet You are going to love https://github.com/TypeStrong/atom-typescript once I've published it. Its much faster ;) (can't take credit for it though... the language service is pretty awesome).

[basarat]

Revisiting after @nycdotnet brought this up in a discussion. @stanvass your proposal is very clear, but I have a few questions.

The redundancy and complexity of having to use two tools that mostly cancel each other out (adding modules and stripping away modules) can be avoided if we can refer to a code module as an internal module

Agreed.

I compile it separately and load it with a script tag

Agreed.

Without having to compile an explicit declaration for Lib.ts every time I modify it.

Disagree. You cannot load it in a script tag OR verify that it is valid without compiling. So compile it. And then you will get lib.js and lib.d.ts.

This is similar to C#. You cannot use lib in project A or project B without compiling lib first.

So. Is there something here that you cannot do with --out --declaration? I really want to help.

[stanvass]

Hello, @basarat. I'm happy this is under discussion.

When I said "explicitly compiled declaration", I meant the requirement to have a .d.ts file on disk that other files should refer to, explicitly.

The scenario after my proposal is:

1. there will be a compiler instance watching for Lib.ts changes and compiling Lib.js file (no declaration file, just JS), and...
2. there will be a compiler instance watching App.ts (and therefore also Lib.ts which App.ts includes as an ambient reference with my proposal) and compiling App.js without including Lib's code inside (still compiled in-memory and used for type-checking, but not included in the output - as if it was a .d.ts).

Notice, with my proposal:

1. there's no race condition between the two compilers, so there's no need for them to run serially (pass 1 produces .d.ts and pass 2 uses the .d.ts) one after one. Serial execution adds pointless lag and complicates out toolchain (we need ability to run multiple TSC passes on every compile).
2. there's no need for a declaration file to be explicitly put on disk, as both watching compilers from my example read Lib.ts directly, but treat it differently (once for output to Lib.js, second time as an App.js ambient reference).

So the things we can't do with Lib.ts --out --declaration are:

1. We can't compile App.js in one pass. We need to have a special batch script to run two passes in sequence.
2. We can't place the declaration file some place different from the output. So either the declaration file ends up in our "bin" folder, or the Lib.js file ends up in our "src" folder. Or we need a script just to put each file in its place.
3. We need to manage generated .d.ts files. Should they be in the repository? Should we git ignore them? How do we differentiate generated files from other .d.ts files? It represent noise in our project files.

My point with item 3 is that we basically end up with a "temp" file in our "bin" or "src" folder, which is a nuisance. We shouldn't need to have these files on disk only to work around an inability of the compiler to exclude JS output on a reference by reference basis.

[stanvass]

I'd like to say, another even more efficient solution could be the ability to specify how TSC splits output in multiple files from one source tree, in a single compilation pass.

Say "1. output App.ts w/o Lib.ts; 2. output Lib.ts in another file".

But this would be more complex to specify and implement from a user interfacing point, compared to one new flag in <reference>.

[basarat]

I understand.

1.) compilations can run in parallel.
2.) don't need to have a noisy .d.ts file on disk.

[mhegazy]

Looking at this again, I think it is subsumed by the proposal in #3469. the underlying issue, if i am not wrong, is the need to partition code logically into components that depend on each other, and have tools and build systems handle that correctly. @stanvass, @nycdotnet , @basarat , @NoelAbrahams do you agree with this assessment? if so i propose closing this issue in favor of #3469.

[nycdotnet]

I think so. If the issues that are called-out in #3469 are addressed and work with both TSC and the language service (esp in Visual Studio), then this specific proposal would likely not be necessary.

[nycdotnet]

In particular the support for different tsconfig.json files (and the ability to pick one in a language service) will help a lot here too. I forget which issue this is but J think it's already implemented.

[basarat]

I have no 🌲 🔪 (piece of wood == stake) in this issue. The NPM work solved most of my desires 🌹

[basarat]

In particular the support for different tsconfig.json files (and the ability to pick one in a language service) will help a lot here too

@nycdotnet something like this : https://github.com/TypeScriptBuilder/tsb/blob/master/docs/features.md#project-search 🌹

[mhegazy]

thanks!