[Discussion] Best way to provide JavaScript builds with Deno

[Soremwar]

Deno version: 0.35.0

Trying to compile a .js file results in Uncaught Error: Unexpected skip of the emit.

Attempting the following
./app.js

console.log("Compile me");

./compile_app.js

const [compile_diag, compile_res] = await Deno.compile(
  "./app.js",
  undefined,
  {
    allowJs: false,
    checkJs: false,
    lib: ["es2018"],
    sourceMap: false,
    target: "es2018",
  }
);

compile_diag && console.log(Deno.formatDiagnostics(compile_diag));

const truncate_filename = filename => filename.replace('file:///', '');

for (const [filename, text] of Object.entries(compile_res)) {
  console.log(`${truncate_filename(filename)}`);
  console.log(text.split('\n').map(line => `\t${line}`).join('\n'));
}

Results in:

Quite probably because the resulting filename matches the original filename, however since Deno.compile does not write anything into the filesystem this should not be a problem.

Same thing happens when attempting to compile a JS and TS file named the same on the same route. Ex: mod.js & mod.ts,

[bartlomieju]

CC @kitsonk

[kitsonk]

We are ignoring an error in the TypeScript compiler which is protecting possibly overwriting the source, which we do all the time in the internal of Deno. I am not totally sure it is an error, though we need to surface up what is there, because even if Deno.compile() doesn't physically write out files, the logic I think is sound.

What happens when you set out to some other path?

[Soremwar]

out makes the program implode. Setting outDir however solves the problem at certaing extent.

.js files won't fail. But naming two files the same with JS and TS extension keeps being a problem. However I don't see a solution that doesn't involve changing Deno.compile behavior.

[kitsonk]

something.js and something.ts in the same path is always a bad idea. The Deno runtime sort of allows it, but the is no way of disambiguating the output of a compile.

So I don't think this is a bug in functionality, but needs better error messages instead of the obscure emit skipped.

[kitsonk]

Writing it down while it is on my mind. Currently we squash a whole set of errors from the TypeScript compiler that cover over things that aren't relevant in the Deno runtime, but Deno.compile() and Deno.bundle() is really targeted at generating code that would work in other runtimes, so we shouldn't just hide those issues away. So we need two different sets of errors that we squash and allow the invoker to decide if those are useful or not to ignore.

[Soremwar]

Not sure about a different set of errors, but definitely more clearer ones.

It might be the moment to discuss something that's been bugging me for a while. Renaming Deno.compile to fit something more closer to it's real behavior.

[kitsonk]

Like what?

[Soremwar]

Deno.transpile

If compilation through Deno becomes a feature inside the program a rename would be necessary, to not mention the fact that this is not compiling at all

Also, Deno.transpileOnly seems to me like an over simplified version of Deno.compile which could be achieved through TS config. Am I wrong?

[kitsonk]

Transpile, to me, means to convert from one syntax to another. Deno.compile() and Deno.bundle() do type checking and resolution, before it does a transpile. So to me it is less representative to name it .transpile().

Deno.transpileOnly() only erases the types and transform syntax, similar to how Babel handles TypeScript. You can't use compiler options to mimic it. In fact there is a request in TypeScript to add the feature to tsc because it can't be fully replicated via compiler options.

[Soremwar]

In the most traditional meaning of the word, a compile process realizes both transpiling and bundling to deliver one single target(binary or not)

Deno compile doesn't really works by itself currently since it can't erase references to other modules in the spirit of not breaking anything. And Deno bundle doesn't make any real sintax transpilation other than converting module references to embedded code.

Deno compile should do both. Since current behavior doesn't actually deliver a functional output into say ES3.

[nayeemrmn]

Is there a sensible way to merge them both into Deno.transpile()/Deno.compile() and opt in to type checking and resolving as options? (Or are they too fundamentally different?)

[Soremwar]

I see no use case on Bundling without specyfing a target runtime.

Thumbs up for unification.

Edit: Perhaps to bundle or not might be an option inside Deno.compile? Would save some time on module resolution. Not that JS or TS take that much compiling anyways.

[kitsonk]

So is tsc a compiler, or not? In its normal configuration it takes input files, type checks them and outputs them. It does can be configured to output to a single file, but it does not create something that stands alone.

Is there a sensible way to merge them both into Deno.transpile()/Deno.compile() and opt in to type checking and resolving as options? (Or are they too fundamentally different?)

Deno.transpileOnly() uses a totally different TypeScript compiler API than Deno.compile()/Deno.bundle() does. Deno.bundle() is not only a default configuration of Deno.compile() it injects code to make the output a fully stand alone JavaScript file. Deno.transpile() does not exist (because without the Only is was more confusing).

[kitsonk]

My biggest concern about combining Deno.compile() and Deno.bundle() would be that the return types are dramatically different. The output filename of Deno.bundle() makes 0 sense in that context, while as with Deno.compile() the output file name(s) are meaningful. Also, it is more consistent with deno run and deno bundle. Alignment to the concept in the command line is important.

[Soremwar]

While as with Deno.compile() the output file name(s) are meaningful.

I do consider that Deno.compile provides more value than just resources to work on top of Deno.bundle, however Deno.bundle seems to have no place when we are talking about compilation, since it should be achieved in the same step or simply skipped if the target is not a single file.

Also, it is more consistent with deno run and deno bundle. Alignment to the concept in the command line is important.

Consistency is important, but oversimplify the operations behind this commands only diminishes the worth they might have.

Deno bundle | compile in my opinion should mimic the behavior of compilers such as parcel or webpack in Node Land, in fact that was my first impression when I first met this project.

Commands such as those should translate into one or more operations that resemble what we are trying to achieve manually here, and not expect the user to make manual scripts for building their projects unless they require a really specific setting. Without leading into magic territory, to resolve an app structure and produce and efficient build should be no manual task

[Soremwar]

So is tsc a compiler, or not? In its normal configuration it takes input files, type checks them and outputs them. It does can be configured to output to a single file, but it does not create something that stands alone.

You gotta forgive my ignorance in this matter. I don't know if TypeScript compiler is the one producing the final standalone build or how much is Deno playing with it to allow us to do so.

[kitsonk]

So a bit of clarity what happens... First some terminology:

• Deno compiler, a specific "private" worker which contains the TypeScript compiler, implements a "compiler host" which gives the TypeScript compiler access to the "outside world" as well as some other things that we do to make things work that aren't implemented in Rust.
• TypeScript compiler, in particular an implementation of the compiler services to create a TypeScript ts.Program and emitting that program. Commonly these are called the compiler services and are effectively what is implemented in tsc. This is different than the language services, which are what power things like VSCode and other editors.
• There is a whole bunch of Rust logic which supports the compiler, as well as handles the caching of modules and deals with figuring out what needs to be done to a module before it is injected in the runtime. These don't always have clean names.

Now the different ways these work together:

• deno run or deno fetch:

  • If the main module is a TypeScript file (or checkJs is enabled) and the JavaScript emit for that file isn't in the cache, Rust spins up the snapshot of the Deno compiler.
  • The main module is passed to the Deno compiler. The Deno compiler uses a specific API from TypeScript to "preprocess" the files before an AST transform is done to identify all the dependencies.
  • The Deno compiler requests the resolution of the modules names for any dependencies of the main module, and then requests that module content from Rust. If Rust doesn't have the "source" for the module, it goes and fetches it before returning it to the Deno compiler. This then recursively happens for all the code for the program, populating the source content in memory awaiting the rest of the process.
  • Once all dependencies have been loaded, the Deno compiler does a ts.createProgram() passing in the main module. This effectively does an AST parse of all of the code. There are several classes of TypeScript errors that can surface at this point, mostly having to do with the configuration of TypeScript. They aren't very common for us because of the opinionated way we do this. This resolves all the types in memory, but doesn't "enforce" the type checking.
  • We attempt to get any diagnostics from the TypeScript compiler, and if there are any that we aren't ignoring, we send them to Rust and they are treated as terminal. Rust logs them out to stderr.
  • We then do an emit of the program. We don't actually care about the emit result directly, because the TypeScript compiler attempt to "write out" each file in the emit, and we hook that in the Deno compiler and write that to the cache.
  • deno run will continue on, loading the main module as an ESM into the isolate and filling any dependencies of that module from the cache.

• deno bundle:

  • Very similar to deno run except that the Deno compiler is always spun up.
  • We do care about the emit, as the emit is a single file, which we then add a loader to and analyse the exports of the main module, and re-export those out of the bundle module. The output of the emit is passed back to Rust to be either written to file or output to stdout.

• Deno.compile():

  • Works like deno fetch but has a slightly different default configuration of the compiler so it doesn't short circuit when some of the content is JavaScript, in that it always analyses JavaScript dependencies and allows JavaScript as part of the program. It also supports a situation where all the sources are provided, instead of being fetched externally to Deno. If the sources are provided, then it does not use the Deno caching mechanisms in Rust (or the Deno module resolution logic).
  • When we introduced the API we discussed using Deno.fetch() but the problem is that deno fetch is actually really poorly named, and so doing that doesn't make sense, and deno compile isn't right either, because that would be as said be expected to generate a stand alone binary. Deno.transpile() is also imperfect as well, since it does more than just transpile. The justification for Deno.compile() though is that this is what we call it internally (it is a RuntimeCompile versus a Compile) and it aligns to generally what tsc main.ts does (though a lot more opinionated).

• Deno.bundle():

  • Works like deno bundle but again slightly different config and the support for internal sources. This is though properly aligned and effectively deno bundle and Deno.bundle() are pretty much the same.

• Deno.transpileOnly():

  • Not something we use elsewhere in Deno, but a common workflow that is likely needed or useful in user land.
  • Uses the ts.transpileModule() API, which basically just does an AST parse and an emit transform without any syntax/type checking by the TypeScript compiler. Again, this is like the Babel support for TypeScript, it just strips the types and transforms the syntax. This is likely the first API we could move directly to SWC and not use the compiler at all. It would be super super fast if we did that. We can't move other bits to SWC any time soon, because of the overhead of double parsing, so I would be really reticent to merge the APIs at the moment as it really complicates it and likely there would be differences in the output between the two if we did that.

[Soremwar]

If merging the API's is such a hassle then probably we shouldn't do that, but rather provide a simplified version (maybe named differently and treat compile, bundle and transpileOnly as low level APIs) that uses them to accomplish this same goal.

Edit: I'm up for this, but vote for leaving this inside the Deno core and not std.

[Soremwar]

@kitsonk In the meanwhile, some problems regarding the compiler should be addressed. Like not being able to use import-maps to make remote code work. That renders compilation only being able to reach the application and not the modules it depends on.

[kitsonk]

This was resolved in Deno 1.5.1.