"deno compile" into executable

[matthewmueller]

I was wondering if there's any plan to support compiling deno scripts into a single executable?

I saw "Single executable" as a feature, but the code that follows seems to suggest it's deno as the single executable, not the programs executed with it.

It'd be awesome to support something like go build main.go or something like zeit/pkg.

[bartlomieju]

CC @ry

[kitsonk]

I believe this is still on the long term roadmap.

[ry]

I'm into the idea - and it's quite doable with our infrastructure. I'd estimate ~2 weeks of my time to get it working. It seems very cool, but it's unclear to me what the actual use cases are - do you have any? I also think there's some more important features that should come first.

[matthewmueller]

The main use case is to simplify deploys – you don't need to care about what's installed on the server because it's all present in that single executable, including the deno executable for the given architecture and OS.

You also tend to find issues at compile time, rather than runtime (e.g. we've all had something something this node module is not found after deploying to production).

[kitsonk]

You also tend to find issues at compile time, rather than runtime

I don't think a single binary helps or doesn't help in this case. Unless you are going to exercise all the code in tests, you aren't going to find issues. Building it into a binary doesn't help with that.

If you are using TypeScript modules, either local or remote with Deno, and you do a prefetch, all the types will be checked at least. Again, building this into a binary doesn't offer you anything more.

[hayd]

I think the use case is not server deploys but cli/gui apps where you want to lock down the deno version used and it's easier to distribute and call a single file.

The compile before deploy is solved with --prefetch if you include the cached deps.

[matthewmueller]

I don't think a single binary helps or doesn't help in this case. Unless you are going to exercise all the code in tests, you aren't going to find issues. Building it into a binary doesn't help with that.

I think you're right about this – good point @kitsonk!

I think the use case is not server deploys but cli/gui apps where you want to lock down the deno version used and it's easier to distribute and call a single file.

It depends on what you're doing, but a single binary definitely makes it easier in those cases too!

[matthewp]

Want to reiterate what @hayd says here. I like to create small single-use CLI tools that are essentially "done" and not having to worry about Deno version compatibility would be very enticing. Aside from security patches, it would be nice to have a binary that never needs upgrading.

[andyfleming]

A couple of use cases for me:

• CLI — Having a single executable makes distribution easy. Especially if cross-compiling is easy!

• Services built into simple build artifacts — This can be great for small container images, and generally nice for CI/CD workflows.

Support for this would push me over the fence of wanting to use Deno for real projects (once stable). Right now, it's not compelling enough for my use cases.

[phil294]

It seems very cool, but it's unclear to me what the actual use cases are

deno could solve an issue that Python and Node have: Ease of running foreign scripts for casual users.

The use case is simple: When my aunt asks me for a programm to organize her holiday pictures on her Windows machine, she does not want to download and install a 30 MB programming environment - she wants a 20 KB do_stuff.exe she can double click.

[ry]

Bert and I are actively working on this. Hopefully we’ll have a demo in a two weeks or so.

[ry]

FYI development has stalled on this feature while we are fixing other bugs in the system... but still on my back burner.

[andyfleming]

Thanks for the update, @ry. I'd be careful to wait too long on getting some version of this going. It seems like a feature that could quickly become infeasible unless we support it early.

[awkj]

by the way, I really found ry deeply affected by golang, study golang, use rust, write typescript.

[chenxyzl]

Thank you for your work.， Is there any progress?
I look forward to it so much.

[ozanmuyes]

@awkj Good observation, I just thought the same thing. To me taking good parts from other languages/platforms and integrating something we use is great think, so thanks @ry. Haters gonna hate 😎 (that issue gets me everytime).

[mnxn]

@satishbabariya I think that's a vast oversimplification of the amount of work needed to make a compiler. Implementing all the features of js/ts in a compiler is a monumental task and I would assume to be out of the scope of this project. A much more feasible way would be to embed the already existing V8 VM into the executable.

[kitsonk]

I think that's a vast oversimplification

Agreed! Many many people have tried to out perform V8 (and other advanced JavaScript runtimes). JavaScript (and therefore TypeScript) is such a permissive language that it take a lot of work to get anything that performs well. Also "TypeScript AST" also only solves part of a problem of creating what is offered by the runtime, like a highly performant garbage collector and a main event loop, just to start. All of that isn't expressed in a "TypeScript AST".

The snapshot building in Deno has been reworked and we are largely creating the internal Deno bundles from very similar code to deno bundle. We will likely unify that very soon, which leads to a path of creating something that can create a snapshot that includes the Deno runtime and the transpiled target program as one binary. For certain workloads it will greatly improve startup time too, I would suspect.

[andyfleming]

The snapshot building in Deno has been reworked and we are largely creating the internal Deno bundles from very similar code to deno bundle. We will likely unify that very soon, which leads to a path of creating something that can create a snapshot that includes the Deno runtime and the transpiled target program as one binary. For certain workloads it will greatly improve startup time too, I would suspect.

@kitsonk That's what I assumed the path would be for such a feature.

This would still be great to see!

[mehmetb]

For me, the major benefit of having this feature is to be able to make commercial versions of the code. Businesses usually don't want to expose their code to their customers.

In Node.js, there is a package called "pkg" which compiles javascript. I am not sure how they do it but I'm guessing they are forcing V8 to compile each function into bytecode and then put everything into a binary file along with node's executable.

[chenxyzl]

this is a discussion on compiling into single executable on dartlang
I think it inspired the current discussion

[kitsonk]

No. This discussion predates that thread by about a year, and even before it was raised compiling user code into a binary was something i think Ry and I talked about.

[claytongulick]

NW.js does this by sort of "cheating". It's a single exe, but it's actually a very clever zip file that extracts itself and bootstraps node on first execute. I really appreciate the single exe deployment option with NW.js, and I've shipped box software with it, but would love to see deno take a better approach (if possible?).

[MirKml]

Pardon my ignorance, but what would "deal with Rust toolchain" mean for the deno users?

@bartlomieju said about option B
It would require user to set up full Rust toolchain. There's also no option to cross-compile. Users
would have to set up full CI workflow similar to Deno's.

It means is't necessary to setup Rust tooling for Typescript/Javascript developer to produce self executable. And this user have to know something a little about how Rust compilation, building, linking etc. work, what some Rust errors means etc.
It's fine for intermediate step as I understand option B, but it isn't final goal for this issue for me as Typescript node/deno user.

[IllusionPerdu]

I don't like the idea of the "compile" command.

I think it would be better to do a "pack" command which creates a ZIP file with all the dependencies and a boot file included. And that we can do a simple "deno run app.zip". and if it is a library that we can do an "import *" lib.zip ""

I'm afraid that with the "compile" command that we end up with thousands of applications that would result in thousands of 16MB (minimal deno) + ?? Mb useless (deno is just a binary and easy to install). With my idea we would only have a deno + many zip;)

Sorry for my bad english !!!

[somombo]

I think everyone agrees that option-A is ideal but ...

I would like to point out that many/most of us are a devs that are intimately familiar with the node.js ecosystem. However, many of us that are concerned about option-B may have forgotten that node-gyp (which you will probably run into at some point or another even if you don't write native node.js modules yourself) also requires prior installation of a foreign build toolchain (in this case, cpp toolchain possibly sprinkled with some python on top).

So, in my view, we should not look at the option-B installation of a rust toolchain onto our dev/CI environments as something that will be completely foreign to us, especially if we all understand that we are ultimately going to strive to achieve an experience that will, in time, be or approache something close to option-A.

I like the idea above of trying to seek out ways to encapsulate deno compile automatically installing the rust toolchain for you if you don't already have it (or it otherwise "helping" you get the toolchain in some other less invasive manner)

[KaKi87]

I think it would be better to do a "pack" command which creates a ZIP file with all the dependencies and a boot file included. And that we can do a simple "deno run app.zip". and if it is a library that we can do an "import *" lib.zip ""

Not user-friendly.
In that case, I would be doing the same thing I do today using Node, which is creating WinRAR self-extracting packages...

[binaryben]

I think it would be better to do a "pack" command which creates a ZIP file with all the dependencies and a boot file included. And that we can do a simple "deno run app.zip". and if it is a library that we can do an "import *" lib.zip ""

This basically exists already. Use deno bundle if that’s your preference @IllusionPerdu. You’d still benefit from the proposed deno-run binary.

---

I appreciate what you are saying @somombo, but Deno is trying to learn from past experience and lessons to do better. Requiring Rust means we may as well stick with Node for now and use existing, well known options

[somombo]

@binaryben Requiring Rust means we may as well stick with Node for now and use existing, well known options.

Fair enough, I think some people will feel that way.. but others will not. I suspect the latter will instead feel like, "for now, something is better than nothing".

And hopefully as the tool approaches option-A the former will also come back to making full use of Deno.

[IllusionPerdu]

This basically exists already. Use deno bundle if that’s your preference @IllusionPerdu. You’d still benefit from the proposed deno-run binary.

In the zip file you can include somes ressources other than just the js/ts : i think a sort of virtual drive
The benefit it's save 16Mo per app ;)

[bmarkovic]

I personally fail to see what the issue with option B is at all.

I am making an assumption that the deno compile, or what have you, will create a Rust project, grab all the required crates and finally compile Deno libs and my JS/TS app into a single executable with resources (that I specify it to) bundled. And since user's code is (at least for the users that are not interested in getting under the hood, i.e. that seek PKG-like experience) only going to be TS/JS -- it's highly unlikely that, once this whole thing is stable off course, that user will ever need to deal with any rustc errors.

Setting up Rust toolchain is pretty straightforward (see: https://rustup.rs/), and if Deno would handle everything else it's as good as a turnkey solution. The target audience are developers, after all.

[lucacasonato]

The initial implementation of this feature will be released in Deno 1.6. For those interested, this is implemented using option A. Option B is still being worked on and will also be available at some point.

[opensas]

So if I understand correctly (according to #986 (comment)) you are taking the hardest path first, which I think will answer most users needs. Am I correct or have I misread it?

[lucacasonato]

We have taken the path that takes the least effort :-)

If you want to discuss, join the Discord: https://discord.gg/deno

[fabiante]

Wow, this came suprisingly! Thank you very much @lucacasonato

[mathiasrw]

You did it!!! 1.6.0 is now released including this feature.

deno compile --unstable https://deno.land/[email protected]/examples/cat.ts will make you an executable version of the module.

• Startup time is on pair with running via deno!

• The size of the 47 MB output file is an area of improvement but we knew that already from this thread.

I'm very very excited.

[RedactedProfile]

Now this is a celebration! I am really excited to try this :)

[liudonghua123]

Nice feature, I tried deno compile --unstable https://deno.land/[email protected]/examples/cat.ts, and it produced a 31.4 MB sized cat.exe. Look forward to a smaller size in the future.

[awkj]

excited ! I will use deno instead of shell

[samirdjelal]

I've found a way to get Deno output smaller by packing Deno executable before the compile.
if you tried to pack the output after compiling Deno will fail to read the content of the code as the compile subcommand will be embed the JS code in the end of the binary as bytecode, it can be seen using any Hex Editor.

So remember:

1. pack Deno binary.
2. compile your project files using the packed Deno binary.

Deno size reduced from 32MB to 11MB using UPX.

[EDIT]

UPX can trigger the anti-virus as false positive malware. we can try another packer/compressor!
Also extra time can be taken to unpack the binary in-memory (~1s for UPX).

[MarkTiedemann]

Note that packing Deno may make it significantly slower. If packed, executing deno -V takes 1s instead of 20ms on my machine.

@echo off

:: Download upx
curl -LO https://github.com/upx/upx/releases/download/v3.96/upx-3.96-win64.zip
tar xf upx-3.96-win64.zip
move upx-3.96-win64\upx.exe .

:: Download deno
curl -LO https://github.com/denoland/deno/releases/download/v1.6.0/deno-x86_64-pc-windows-msvc.zip
tar xf deno-x86_64-pc-windows-msvc.zip

:: Download timeit
curl -LO https://github.com/MarkTiedemann/rktools2k3/raw/master/timeit.exe

:: Pack deno
upx --ultra-brute -o deno.packed.exe deno.exe

:: Time deno startup (default)
timeit deno.exe -V
:: deno 1.6.0
:: 
:: Version Number:   Windows NT 6.2 (Build 9200)
:: Exit Time:        10:36 pm, Wednesday, December 9 2020
:: Elapsed Time:     0:00:00.020
:: Process Time:     0:00:00.000
:: System Calls:     4496
:: Context Switches: 1444
:: Page Faults:      2303
:: Bytes Read:       2568
:: Bytes Written:    23152
:: Bytes Other:      65540

:: Time deno startup (packed)
timeit deno.packed.exe -V
:: deno 1.6.0
:: 
:: Version Number:   Windows NT 6.2 (Build 9200)
:: Exit Time:        10:36 pm, Wednesday, December 9 2020
:: Elapsed Time:     0:00:00.975
:: Process Time:     0:00:00.968
:: System Calls:     32825
:: Context Switches: 18158
:: Page Faults:      12024
:: Bytes Read:       191782
:: Bytes Written:    554334
:: Bytes Other:      70106

Also, be careful about shipping packed binaries to users: Anti-virus software, including Windows Defender, is more likely to flag your binary and prevent it from running in the first place. Since packing obfuscates the binary to some degree, it is sometimes considered "defense evasion".

[samirdjelal]

I just mentioned that we can pack it before compiling,
However you are right, UPX will be flagged as false positive by few AVs and it will take extra time to unpack the app in-memory!
and maybe obfuscating the JS before embedding to the end of the binary will be great.
It is a starting point.

[coreybutler]

@MarkTiedemann has some valid points, but it's also worth mentioning a few additional points:

Antivirus

To better handle antivirus, code sign your executables using a trusted certificate. Most of the antivirus apps will account for that and allow the app to run.

Boot Time

While there may be a delay, remember to consider the usability of the executable. There are some cases where a 1s delay would be kind of annoying, but there are also many use cases where a 1s delay would go completely unnoticed by end users.

--

Other Considerations

Remember there are other ways to reduce the size of the output, primarily by understanding everything being included in your release. It's easy to "import everything" and not consider where there may be overlap. Perhaps you don't need that hefty logging library when console.log will do. Point: be mindful of what goes in your apps.

If you're interested in how others are using this...

I've been working with a CLI-based dev approach for awhile now, where this feature will come in handy. I reflected on what this means to me in a small piece called JavaScript Executables. So, if anyone is browsing these issues looking for examples or just trying to understand how others are using this feature, here's another resource.

[KaKi87]

sign your executables using a trusted certificate.

Are there free certificate providers for executables as there are for HTTPS ?

[coreybutler]

@KaKi87 - nothing free that I'm aware of. It requires a greater degree of trust. An HTTPS cert is hosted, whereas a local executable can (conceivably) do just about anything once installed. That's where things like background checks (which are common for paid certs) are useful. You have to have at least some degree of verifiable legitimacy to get a cert. Unfortunately, that usually means paying. There's a whole thread about why LetsEncrypt does not provide free code signing certs at https://community.letsencrypt.org/t/do-you-support-code-signing/370

[KaKi87]

I think Let's Encrypt could achieve viable code signing from domain names owners using WHOIS.
WHOIS information is directly provided by hosters, which contains either an organization name, an individual name, or both.
That was the problem pointed in the linked topic, right ?

[samirdjelal]

I don't think you can find a free code-signing, as it is all about trust factor, if you can get free cert it means malware authors can get it too and use it to bypass AV, however there is open source code signing code for €25!
https://shop.certum.eu/open-source-code-signing-code.html

[daotoad]

Years ago, I used PAR::Packer and ActiveState's now defunct PerlApp to create distributable perl code. Both tools used a strategy of packing the files, then caching the unpacked files after the first run. The end result was a significant speed boost on subsequent executions. This strategy works very nicely.

I don't have any good ideas on the antivirus flagging of packed files. It was an occasional issue with PerlApp and is part of the reason ActiveState gave up on it.

[kitsonk]

Enhancements to the feature should probably be discussed as a seperate thread in Discussions instead of comment on a closed issue where the thoughts and feedback will likely be lost or ignored.