feat: deno compile

[lucacasonato]

Closes #986

This implements the deno compile subcommand. It works be generating a new standalone binary, by appending a bundle of the source file and magic trailer from the current binary. If we find a magic trailer at the end of the the binary, we try to load the bundle that is embedded in the binary, and run it.

~/p/g/d/deno ❯❯❯ ./target/debug/deno compile https://deno.land/std/http/file_server.ts file_server
Bundle https://deno.land/std/http/file_server.ts
Check https://deno.land/std/http/file_server.ts
Compile https://deno.land/std/http/file_server.ts
~/p/g/d/deno ❯❯❯ ./file_server 
HTTP server listening on http://0.0.0.0:4507/

[bartlomieju]

I think this is a great first pass, but suggest to get more eyes on it before landing

[kevinkassimo]

Probably not for this PR, but I think it would also be nice to allow optional permission flags in the future, where we add an extra trailer for the bit vector of permissions.

[lucacasonato]

Yes. I have some ideas on how to do this in the future.

[ry]

LGTM

Milestone achievement.

[bartlomieju]

LGTM 🚀

[jveres]

I tried this new feature and found an issue where deno::program_state::ProgramState::get_emit tries to load file://$deno$/bundle.js which is fake for compiled bundles and deno::disk_cache::DiskCache::get_cache_filename panics. It was trivial to fix by patching cli/program_state/get_emit(...) with this:

fn get_emit(&self, url: &Url) -> Option<(Vec<u8>, Option<Vec<u8>>)> {
    match url.scheme() {
      // we should only be looking for emits for schemes that denote external
      // modules, which the disk_cache supports
      "wasm" | "file" | "http" | "https" => {
        // ignore when called from a self-contained binary
        if url.to_string() == crate::standalone::SPECIFIER {
          return None;
        }
      },
      _ => {
        return None;
      }
    }
...

Not sure if this is the most appropriate way, though.

[bartlomieju]

I tried this new feature and found an issue where deno::program_state::ProgramState::get_emit tries to load file://$deno$/bundle.js which is fake for compiled bundles and deno::disk_cache::DiskCache::get_cache_filename panics. It was trivial to fix by patching cli/program_state/get_emit(...) with this:

fn get_emit(&self, url: &Url) -> Option<(Vec<u8>, Option<Vec<u8>>)> {
    match url.scheme() {
      // we should only be looking for emits for schemes that denote external
      // modules, which the disk_cache supports
      "wasm" | "file" | "http" | "https" => {
        // ignore when called from a self-contained binary
        if url.to_string() == crate::standalone::SPECIFIER {
          return None;
        }
      },
      _ => {
        return None;
      }
    }
...

Not sure if this is the most appropriate way, though.

@jveres could you open a new issue and provide a reproduction?

[nayeemrmn]

found an issue where deno::program_state::ProgramState::get_emit tries to load file://$deno$/bundle.js

Yep, the fix is this:

diff --git a/cli/standalone.rs b/cli/standalone.rs
index 06c20bc2..7c9e9fcb 100644
--- a/cli/standalone.rs
+++ b/cli/standalone.rs
@@ -58,7 +58,7 @@ pub fn try_run_standalone_binary(args: Vec<String>) -> Result<(), AnyError> {
   }
 }
 
-const SPECIFIER: &str = "file://$deno$/bundle.js";
+const SPECIFIER: &str = "file:///$deno$bundle.js";
 
 struct EmbeddedModuleLoader(String);

I'm trying to come up with tests.

@bartlomieju
I was able to reproduce with:

import * as path from "./std/path/mod.ts";

Seems pretty random though.

[jveres]

@nayeemrmn quickly checked with file:///$deno$bundle.js but it gives error: Self-contained binaries don't support module loading so maybe it's a different issue.

[bartlomieju]

CC @lucacasonato

[bartlomieju]

@nayeemrmn quickly checked with file:///$deno$bundle.js but it gives error: Self-contained binaries don't support module loading so maybe it's a different issue.

@jveres do you have dynamic imports or workers in your code? If so they are not supported

[jveres]

@nayeemrmn quickly checked with file:///$deno$bundle.js but it gives error: Self-contained binaries don't support module loading so maybe it's a different issue.

@jveres do you have dynamic imports or workers in your code? If so they are not supported

No, I don't. Using my fix above it works.
I will create a test case for this later.

[jveres]

as simple as

const e = new Error();
console.log(e);

deno compile --unstable test.ts test
RUST_BACKTRACE=full ./test
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ()', cli/disk_cache.rs:72:39
stack backtrace:
   0:        0x10b8c56d3 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd4b962ed89f71a03
   1:        0x10b543b50 - core::fmt::write::h94ae1e793baa7a00
   2:        0x10b8c4dbb - std::io::Write::write_fmt::h5c716758fdc3057f
   3:        0x10b8c4745 - std::panicking::default_hook::{{closure}}::hc6119c7d16548caf
   4:        0x10b8c3e85 - std::panicking::rust_panic_with_hook::hc36596b4257bea99
   5:        0x10b8e133c - std::panicking::begin_panic_handler::{{closure}}::h49e5ddc3f21ca2fb
   6:        0x10b8e1307 - std::sys_common::backtrace::__rust_end_short_backtrace::h9bd32c9ad3fad18f
   7:        0x10b8e12c0 - _rust_begin_unwind
   8:        0x10c60cc4f - core::panicking::panic_fmt::hcdc9362d34d55302
   9:        0x10c60d225 - core::option::expect_none_failed::hef0458611b13747b
  10:        0x10b39962e - core::result::Result<T,E>::unwrap::h5376c858e103cf24
  11:        0x10b48c14d - deno::disk_cache::DiskCache::get_cache_filename_with_extension::h733f5eb5af07bc3e
  12:        0x10b4c086c - deno::program_state::ProgramState::get_emit::h13846ddd08588463
  13:        0x10b4c1514 - deno::source_maps::get_orig_position::hd223e3d5eecacb10
  14:        0x10b44515b - deno_core::ops::json_op_sync::{{closure}}::h1c84ae06d203eb0e
  15:        0x10b3d76c5 - <alloc::boxed::Box<F> as core::ops::function::Fn<A>>::call::h1744985c3c7bbe57
  16:        0x10b4a4917 - deno::metrics::metrics_op::{{closure}}::hfbfe0e6b2fb6b282
  17:        0x10b3d76c5 - <alloc::boxed::Box<F> as core::ops::function::Fn<A>>::call::h1744985c3c7bbe57
  18:        0x10b557b7e - <extern "C" fn(A0) .> R as rusty_v8::support::CFnFrom<F>>::mapping::c_fn::hf0d234ae6b0410d0
  19:        0x10bb6c059 - __ZN2v88internal25FunctionCallbackArguments4CallENS0_15CallHandlerInfoE
  20:        0x10bb6b5c8 - __ZN2v88internal12_GLOBAL__N_119HandleApiCallHelperILb0EEENS0_11MaybeHandleINS0_6ObjectEEEPNS0_7IsolateENS0_6HandleINS0_10HeapObjectEEESA_NS8_INS0_20FunctionTemplateInfoEEENS8_IS4_EENS0_16BuiltinArgumentsE
  21:        0x10bb6ab80 - __ZN2v88internalL26Builtin_Impl_HandleApiCallENS0_16BuiltinArgumentsEPNS0_7IsolateE
fatal runtime error: failed to initiate panic, error 5
[1]    62205 abort      RUST_BACKTRACE=full ./test

[lucacasonato]

@jveres See #8578

[zlianon]

output size of file_server is 48MB 😬

[lucacasonato]

@zlianon As discussed in the release post, we are confident we can reduce size to around 20 MB on Linux / macOS. Probably even less on Windows. See https://deno.land/posts/v1.6#codedeno-compilecode-self-contained-standalone-binaries