Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove nodejs dependency #3

Closed
3 of 4 tasks
jimafisk opened this issue Apr 29, 2020 · 23 comments · Fixed by #40
Closed
3 of 4 tasks

Remove nodejs dependency #3

jimafisk opened this issue Apr 29, 2020 · 23 comments · Fixed by #40
Labels
epic high high priority

Comments

@jimafisk
Copy link
Member

jimafisk commented Apr 29, 2020

https://github.com/zeit/pkg

Should:

  • Not require devs to have nodejs installed on their computer
  • Not require internet access to start new project (currently runs npm install)
  • Still be extensible via NPM
  • Still allow custom Gopack package to create web_modules for ESM
@jimafisk
Copy link
Member Author

jimafisk commented May 8, 2020

denoland/deno#986

@jimafisk jimafisk added the epic label May 14, 2020
@jimafisk
Copy link
Member Author

@jimafisk
Copy link
Member Author

@jimafisk
Copy link
Member Author

jimafisk commented Jun 4, 2020

@jimafisk
Copy link
Member Author

jimafisk commented Jun 5, 2020

Based on the reddit conversation, I'm thinking through how to run a V8 sandbox to compile svelte. The most current + supported project seems to be https://github.com/rogchap/v8go (also doesn't require us to build v8 ourselves).

Even though the actual script might not be faster than running node directly, I think it would speed up the build because we could cut out the exec.Command and the bundling (in Go) and unpacking (in JS) of templates we're currently doing. It would also make installation easier which would resolve issues like #32 and we would no longer need to execute the system nodejs which should fix the snapcraft issue #31.

V8go can't require/import other scripts (see rogchap/v8go#22) so we'll have to rethink a couple of things:

  1. Writing to the filesystem will have to be handed back over to Go, which is desired because it will likely be faster.
  2. We'll need to bundle build script dependencies (like svelte) using something like esbuild (https://github.com/evanw/esbuild)

@jimafisk
Copy link
Member Author

jimafisk commented Jun 5, 2020

Watching for a Go API on esbuild: evanw/esbuild#152

@jimafisk
Copy link
Member Author

@jimafisk
Copy link
Member Author

jimafisk commented Jul 1, 2020

Proposal:

  1. Bundle up the 4 NPM dependencies required by Plenti core (navaid, regexparam, require-relative, svelte) and put them as defaults in the binary using go:generate. Possibly can remove require-relative since it's part of the node build process that will likely get replaced by Go code.
  2. When creating a new project (plenti new site my-site) plenti will write these dependencies to /node_modules/ folder inside the project.
  3. If a collaborator pulls down an existing project, they will not be running the plenti new site command. So plenti build and plenti serve must check for the existence of a /node_modules/ folder and if it's not there, write the defaults included with the binary to the filesystem. This check is necessary since the /node_modules/ folder is not typically tracked by git.
  4. The build will use the esbuild bundler to pull all the dependencies together (mainly the svelte compiler) so that we can compile the site using v8go.

Things to keep in mind:

  1. If you have modified the /node_modules/, either to extend your site with additional functionality or to simply change the version of a core dependency, this should be reflected in your package-lock.json file. It is now your responsibility to communicate with your team that using NPM like normal is required (e.g. when pulling an existing project, if you simply plenti serve you are going to get the defaults, you should instead npm install first).
  2. The check for the existence of the /node_modules/ folder will not inspect versions or individual packages inside the /node_modules/ folder, it will simply check that the /node_modules/ folder exists - so if you've added this folder and removed a core package, it will break the build.
  3. If you've customized the /node_modules/ in any way (extended or updated), you will need to account for this in the CI that builds and deploys your site. Basically you will need to add an npm install step. Currently the official plenti image (https://hub.docker.com/r/plentico/plenti) does not have nodejs or npm installed on it, so you will need to take care of setting up that. In the future we'd like to support a second container with NPM on it already to make this easier, but we'll keep it separate to avoid bloat on the official image for folks who want a faster experience when using plenti core without modification.

@jimafisk
Copy link
Member Author

jimafisk commented Jul 3, 2020

We can remove require-relative by just pulling the part we need into build.js, however it still requires referencing the module nodejs package:

import Module from 'module';

let root = new Module();
let htmlWrapper = '/home/jimafisk/Desktop/my-site/layout/global/html.svelte';
let component = root.require(htmlWrapper).default;

Where component is equal to { render: [Function: render], '$$render': [Function: $$render] } which allows us to create static HTML and CSS with

let { html, css } = component.render(props);

@jimafisk
Copy link
Member Author

jimafisk commented Jul 9, 2020

The current implementation using v8go is working for the client build, but it might be worth looking into QuickJS for speed improvements and module support:

Another conversation recommending QuickJS: https://www.reddit.com/r/golang/comments/cd5gja/does_anyone_have_experience_with_parsing/

List of embeddable javascript interpreters: https://gist.github.com/maxogden/c61a58498c1933ece598

Examples:

Tried otto but it took almost 4 minutes to compile the client SPA, vs about 300ms with v8go:

otto

@jimafisk
Copy link
Member Author

jimafisk commented Jul 10, 2020

Tried some rough benchmarking of QuickJS against v8go:

Attempt # QuickJS v8go
1 649.606009ms 53.061618ms
2 686.947327ms 61.719715ms
3 673.800305ms 60.363899ms
4 670.392486ms 57.559065ms
5 673.398664ms 57.768148ms
6 683.323302ms 60.482449ms
Code Used
package main

import (
	"fmt"
	"time"

	"github.com/lithdew/quickjs"
	"rogchap.com/v8go"
)

func main() {
	start := time.Now()
	rt := quickjs.NewRuntime()
	ctx := rt.NewContext()
	result, err := ctx.Eval(`
	function factorial(n) {
		return n === 1 ? n : n * factorial(--n);
	}
	
	var i = 0;
	
	while (i++ < 1e6) {
		factorial(10);
	}
	`)
	if err != nil {
		fmt.Printf("Eval error: %v", err)
	}
	fmt.Println(result.String())
	elapsed := time.Since(start)
	fmt.Println(elapsed)

	start = time.Now()
	vm, _ := v8go.NewIsolate()
	ctx1, _ := v8go.NewContext(vm)
	result1, err1 := ctx1.RunScript(`
	function factorial(n) {
		return n === 1 ? n : n * factorial(--n);
	}
	
	var i = 0;
	
	while (i++ < 1e6) {
		factorial(10);
	}
	`, "math.js")
	if err1 != nil {
		fmt.Printf("Eval error: %v", err1)
	}
	fmt.Println(result1)
	elapsed = time.Since(start)
	fmt.Println(elapsed)
}

@jimafisk
Copy link
Member Author

jimafisk commented Jul 10, 2020

Wrapping my head around process for creating static html in Svelte.

A simplified version of how we're getting static HTML using NodeJS looks something like this (click to reveal)
Example staticBuildStr format
[
  {
    "node": { 
      "path": "/blog/adding_pletiform",
      "type": "blog",
      "filename": "adding_pletiform.json",
      "fields": {
        "title": "Build sites with good form",
        "body": [ "Need an easy webform solution?", "Try adding a <a href='https://plentiform.com' target='blank' rel='noopener noreferrer'>plentiform</a>! (Coming soon)" ],
        "author": "Jim Fisk",
        "date": "1/26/2020"
      }
    }, 
    "componentPath": "layout/content/blog.svelte",
    "destPath": "public//blog/adding_pletiform/index.html"
  },
  ...
]
import 'svelte/register.js';

let htmlWrapper = path.join(path.resolve(), 'layout/global/html.svelte')
let root = new Module();
let component = root.require(htmlWrapper).default;

let staticBuildStr = JSON.parse(args[0]); // See example format above
staticBuildStr.forEach(arg => {
  const route = root.require(arg.componentPath).default;
  let props = {
    route: route,
    node: arg.node
  }
  let { html, css } = component.render(props);
}

What's actually happening behind the scenes is it looks like node_modules/svelte/register.js actually imports node_modules/svelte/compiler.js then runs a svelte.compile() passing the generate: 'ssr' option. It then wraps the js.code outputted in a CJS module using module._compile().

Manually running the svelte compiler with SSR as a test in plenti's cmd/build/client.go:

ctx.RunScript("var { js, css } = svelte.compile(`"+componentStr+"`, {generate: 'ssr'});", "compile_svelte")
jsCode, _ := ctx.RunScript("js.code;", "compile_svelte")
fmt.Println(jsCode)
Returns SSR components that look like this (click to reveal)
layout/global/html.svelte
/* generated by Svelte v3.23.2 */
import {
	create_ssr_component,
	missing_component,
	validate_component
} from "svelte/internal";

import Head from "./head.svelte";
import Nav from "./nav.svelte";
import Footer from "./footer.svelte";
import { makeTitle } from "../scripts/make_title.svelte";

const css = {
	code: "body.svelte-sk4nou{font-family:'Rubik', sans-serif;display:flex;flex-direction:column;margin:0}main.svelte-sk4nou{flex-grow:1}.container{max-width:1024px;margin:0 auto;flex-grow:1;padding:0 20px}:root{--primary:rgb(34, 166, 237);--primary-dark:rgb(16, 92, 133);--accent:rgb(254, 211, 48);--base:rgb(245, 245, 245);--base-dark:rgb(17, 17, 17)}main a{position:relative;text-decoration:none;color:var(--base-dark);padding-bottom:5px}main a:before{content:\"\";width:100%;height:100%;background-image:linear-gradient(to top, var(--accent) 25%, rgba(0, 0, 0, 0) 40%);position:absolute;left:0;bottom:2px;z-index:-1;will-change:width;transform:rotate(-2deg);transform-origin:left bottom;transition:width .1s ease-out}main a:hover:before{width:0;transition-duration:.15s}",
	map: "{\"version\":3,\"file\":null,\"sources\":[null],\"sourcesContent\":[\"<script>\\n  import Head from './head.svelte';\\n  import Nav from './nav.svelte';\\n  import Footer from './footer.svelte';\\n  import { makeTitle } from '../scripts/make_title.svelte';\\n\\n  export let route, node, allNodes;\\n</script>\\n\\n<html lang=\\\"en\\\">\\n<Head title={makeTitle(node.filename)} />\\n<body>\\n  <Nav />\\n  <main>\\n    <div class=\\\"container\\\">\\n      <svelte:component this={route} {...node.fields} {allNodes} />\\n      <br />\\n    </div>\\n  </main>\\n  <Footer {allNodes} />\\n</body>\\n</html>\\n\\n<style>\\n  body {\\n    font-family: 'Rubik', sans-serif;\\n    display: flex;\\n    flex-direction: column;\\n    margin: 0;\\n  }\\n  main {\\n    flex-grow: 1;\\n  }\\n  :global(.container) {\\n    max-width: 1024px;\\n    margin: 0 auto;\\n    flex-grow: 1;\\n    padding: 0 20px;\\n  }\\n  :global(:root) {\\n    --primary: rgb(34, 166, 237);\\n    --primary-dark: rgb(16, 92, 133);\\n    --accent: rgb(254, 211, 48);\\n    --base: rgb(245, 245, 245);\\n    --base-dark: rgb(17, 17, 17);\\n  }\\n  :global(main a) {\\n    position: relative;\\n    text-decoration: none;\\n    color: var(--base-dark);\\n    padding-bottom: 5px;\\n  }\\n  :global(main a:before) {\\n    content: \\\"\\\";\\n    width: 100%;\\n    height: 100%;\\n    background-image: linear-gradient(to top, var(--accent) 25%, rgba(0, 0, 0, 0) 40%);  \\n    position: absolute;\\n    left: 0;\\n    bottom: 2px;\\n    z-index: -1;   \\n    will-change: width;\\n    transform: rotate(-2deg);\\n    transform-origin: left bottom;\\n    transition: width .1s ease-out;\\n  }\\n  :global(main a:hover:before) {\\n    width: 0;\\n    transition-duration: .15s;\\n  }\\n</style>\\n\"],\"names\":[],\"mappings\":\"AAwBE,IAAI,cAAC,CAAC,AACJ,WAAW,CAAE,OAAO,CAAC,CAAC,UAAU,CAChC,OAAO,CAAE,IAAI,CACb,cAAc,CAAE,MAAM,CACtB,MAAM,CAAE,CAAC,AACX,CAAC,AACD,IAAI,cAAC,CAAC,AACJ,SAAS,CAAE,CAAC,AACd,CAAC,AACO,UAAU,AAAE,CAAC,AACnB,SAAS,CAAE,MAAM,CACjB,MAAM,CAAE,CAAC,CAAC,IAAI,CACd,SAAS,CAAE,CAAC,CACZ,OAAO,CAAE,CAAC,CAAC,IAAI,AACjB,CAAC,AACO,KAAK,AAAE,CAAC,AACd,SAAS,CAAE,iBAAiB,CAC5B,cAAc,CAAE,gBAAgB,CAChC,QAAQ,CAAE,iBAAiB,CAC3B,MAAM,CAAE,kBAAkB,CAC1B,WAAW,CAAE,eAAe,AAC9B,CAAC,AACO,MAAM,AAAE,CAAC,AACf,QAAQ,CAAE,QAAQ,CAClB,eAAe,CAAE,IAAI,CACrB,KAAK,CAAE,IAAI,WAAW,CAAC,CACvB,cAAc,CAAE,GAAG,AACrB,CAAC,AACO,aAAa,AAAE,CAAC,AACtB,OAAO,CAAE,EAAE,CACX,KAAK,CAAE,IAAI,CACX,MAAM,CAAE,IAAI,CACZ,gBAAgB,CAAE,gBAAgB,EAAE,CAAC,GAAG,CAAC,CAAC,IAAI,QAAQ,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAClF,QAAQ,CAAE,QAAQ,CAClB,IAAI,CAAE,CAAC,CACP,MAAM,CAAE,GAAG,CACX,OAAO,CAAE,EAAE,CACX,WAAW,CAAE,KAAK,CAClB,SAAS,CAAE,OAAO,KAAK,CAAC,CACxB,gBAAgB,CAAE,IAAI,CAAC,MAAM,CAC7B,UAAU,CAAE,KAAK,CAAC,GAAG,CAAC,QAAQ,AAChC,CAAC,AACO,mBAAmB,AAAE,CAAC,AAC5B,KAAK,CAAE,CAAC,CACR,mBAAmB,CAAE,IAAI,AAC3B,CAAC\"}"
};

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { route } = $$props, { node } = $$props, { allNodes } = $$props;
	if ($$props.route === void 0 && $$bindings.route && route !== void 0) $$bindings.route(route);
	if ($$props.node === void 0 && $$bindings.node && node !== void 0) $$bindings.node(node);
	if ($$props.allNodes === void 0 && $$bindings.allNodes && allNodes !== void 0) $$bindings.allNodes(allNodes);
	$$result.css.add(css);

	return `<html lang="${"en"}">${validate_component(Head, "Head").$$render($$result, { title: makeTitle(node.filename) }, {}, {})}
<body class="${"svelte-sk4nou"}">${validate_component(Nav, "Nav").$$render($$result, {}, {}, {})}
  <main class="${"svelte-sk4nou"}"><div class="${"container"}">${validate_component(route || missing_component, "svelte:component").$$render($$result, Object.assign(node.fields, { allNodes }), {}, {})}
      <br></div></main>
  ${validate_component(Footer, "Footer").$$render($$result, { allNodes }, {}, {})}</body>
</html>`;
});

export default Component;
layout/content/pages.svelte
/* generated by Svelte v3.23.2 */
import {
	create_ssr_component,
	each,
	escape,
	validate_component
} from "svelte/internal";

import Uses from "../components/template.svelte";

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { title } = $$props, { description } = $$props;
	if ($$props.title === void 0 && $$bindings.title && title !== void 0) $$bindings.title(title);
	if ($$props.description === void 0 && $$bindings.description && description !== void 0) $$bindings.description(description);

	return `<h1>${escape(title)}</h1>

<div>${each(description, paragraph => `<p>${paragraph}</p>`)}</div>

${validate_component(Uses, "Uses").$$render($$result, { type: "pages" }, {}, {})}

<p><a href="${"/"}">Back home</a></p>`;
});

export default Component;
layout/content/index.svelte
/* generated by Svelte v3.23.2 */
import {
	create_ssr_component,
	each,
	escape,
	is_promise,
	missing_component,
	validate_component
} from "svelte/internal";

import Grid from "../components/grid.svelte";
import { loadComponent } from "../scripts/load_component.svelte";

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { title } = $$props,
		{ intro } = $$props,
		{ components } = $$props,
		{ allNodes } = $$props;

	if ($$props.title === void 0 && $$bindings.title && title !== void 0) $$bindings.title(title);
	if ($$props.intro === void 0 && $$bindings.intro && intro !== void 0) $$bindings.intro(intro);
	if ($$props.components === void 0 && $$bindings.components && components !== void 0) $$bindings.components(components);
	if ($$props.allNodes === void 0 && $$bindings.allNodes && allNodes !== void 0) $$bindings.allNodes(allNodes);

	return `<h1>${escape(title)}</h1>

<section id="${"intro"}"><p>${intro.slogan}</p></section>

<section id="${"intro"}">${each(intro.help, paragraph => `<p>${paragraph}</p>`)}</section>

<div><h3>Recent blog posts:</h3>
	${validate_component(Grid, "Grid").$$render($$result, { items: allNodes, filter: "blog" }, {}, {})}
	<br></div>

${components
	? `${each(components, ({ component, fields }) => `${(function (__value) {
			if (is_promise(__value)) return `
			loading component...
		`;

			return (function (compClass) {
				return `
			${validate_component(compClass || missing_component, "svelte:component").$$render($$result, Object.assign(fields), {}, {})}
		`;
			})(__value);
		})(loadComponent(component))}`)}`
	: ``}`;
});

export default Component;
layout/components/template.svelte
/* generated by Svelte v3.23.2 */
import {
	add_attribute,
	create_ssr_component,
	escape,
	null_to_empty
} from "svelte/internal";

const css = {
	code: ".template.svelte-kyi9jr{display:flex;align-items:center}pre.svelte-kyi9jr{display:flex;padding-left:5px}code.svelte-kyi9jr{background-color:var(--base);padding:5px 10px}code.copied.svelte-kyi9jr{background-color:var(--accent)}button.svelte-kyi9jr{border:1px solid rgba(0,0,0,.1);background:white;padding:4px;border-top-right-radius:5px;border-bottom-right-radius:5px;cursor:pointer}",
	map: "{\"version\":3,\"file\":null,\"sources\":[null],\"sourcesContent\":[\"<script>\\n  export let type;\\n\\n  let path;\\n  let copyText = \\\"Copy\\\";\\n  const copy = async () => {\\n    if (!navigator.clipboard) {\\n      return\\n    }\\n    try {\\n      copyText = \\\"Copied\\\";\\n      await navigator.clipboard.writeText(path.innerHTML);\\n      setTimeout(() => copyText = \\\"Copy\\\", 500);\\n    } catch (err) {\\n      console.error('Failed to copy!', err)\\n    }\\n  }\\n</script>\\n\\n<div class=\\\"template\\\">\\n  <span>Template:</span>\\n  <pre>\\n    <code bind:this={path} class=\\\"{copyText}\\\">layout/content/{type}.svelte</code>\\n    <button on:click={copy}>{copyText}</button>\\n  </pre>\\n</div>\\n\\n<style>\\n  .template {\\n    display: flex;\\n    align-items: center;\\n  }\\n  pre {\\n    display: flex;\\n    padding-left: 5px;\\n  }\\n  code {\\n      background-color: var(--base);\\n      padding: 5px 10px;\\n  }\\n  code.copied {\\n      background-color: var(--accent);\\n  }\\n  button {\\n    border: 1px solid rgba(0,0,0,.1);\\n    background: white;\\n    padding: 4px;\\n    border-top-right-radius: 5px;\\n    border-bottom-right-radius: 5px;\\n    cursor: pointer;\\n  }\\n</style>\"],\"names\":[],\"mappings\":\"AA4BE,SAAS,cAAC,CAAC,AACT,OAAO,CAAE,IAAI,CACb,WAAW,CAAE,MAAM,AACrB,CAAC,AACD,GAAG,cAAC,CAAC,AACH,OAAO,CAAE,IAAI,CACb,YAAY,CAAE,GAAG,AACnB,CAAC,AACD,IAAI,cAAC,CAAC,AACF,gBAAgB,CAAE,IAAI,MAAM,CAAC,CAC7B,OAAO,CAAE,GAAG,CAAC,IAAI,AACrB,CAAC,AACD,IAAI,OAAO,cAAC,CAAC,AACT,gBAAgB,CAAE,IAAI,QAAQ,CAAC,AACnC,CAAC,AACD,MAAM,cAAC,CAAC,AACN,MAAM,CAAE,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAChC,UAAU,CAAE,KAAK,CACjB,OAAO,CAAE,GAAG,CACZ,uBAAuB,CAAE,GAAG,CAC5B,0BAA0B,CAAE,GAAG,CAC/B,MAAM,CAAE,OAAO,AACjB,CAAC\"}"
};

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { type } = $$props;
	let path;
	let copyText = "Copy";

	const copy = async () => {
		if (!navigator.clipboard) {
			return;
		}

		try {
			copyText = "Copied";
			await navigator.clipboard.writeText(path.innerHTML);
			setTimeout(() => copyText = "Copy", 500);
		} catch(err) {
			console.error("Failed to copy!", err);
		}
	};

	if ($$props.type === void 0 && $$bindings.type && type !== void 0) $$bindings.type(type);
	$$result.css.add(css);

	return `<div class="${"template svelte-kyi9jr"}"><span>Template:</span>
  <pre class="${"svelte-kyi9jr"}"><code class="${escape(null_to_empty(copyText)) + " svelte-kyi9jr"}"${add_attribute("this", path, 1)}>layout/content/${escape(type)}.svelte</code>
    <button class="${"svelte-kyi9jr"}">${escape(copyText)}</button></pre>
</div>`;
});

export default Component;
layout/ejected/router.svelte
/* generated by Svelte v3.23.2 */
import { create_ssr_component, validate_component } from "svelte/internal";

import Navaid from "navaid";
import nodes from "./nodes.js";
import Html from "../global/html.svelte";

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let route, node, allNodes;

	const getNode = (uri, trailingSlash = "") => {
		return nodes.find(node => node.path + trailingSlash == uri);
	};

	let uri = location.pathname;
	node = getNode(uri);

	if (node === undefined) {
		node = getNode(uri, "/");
	}

	allNodes = nodes;

	function draw(m) {
		node = getNode(uri);

		if (node === undefined) {
			// Check if there is a 404 data source.
			node = getNode("/404");

			if (node === undefined) {
				// If no 404.json data source exists, pass placeholder values.
				node = {
					"path": "/404",
					"type": "404",
					"filename": "404.json",
					"fields": {}
				};
			}
		}

		route = m.default;
		window.scrollTo(0, 0);
	}

	function track(obj) {
		uri = obj.state || obj.uri;
	}

	addEventListener("replacestate", track);
	addEventListener("pushstate", track);
	addEventListener("popstate", track);

	const handle404 = () => {
		import("../content/404.js").then(draw).catch(err => {
			console.log("Add a '/layout/content/404.svelte' file to handle Page Not Found errors.");
			console.log("If you want to pass data to your 404 component, you can also add a '/content/404.json' file.");
			console.log(err);
		});
	};

	const router = Navaid("/", handle404);

	allNodes.forEach(node => {
		router.on(node.path, () => {
			// Check if the url visited ends in a trailing slash (besides the homepage).
			if (uri.length > 1 && uri.slice(-1) == "/") {
				// Redirect to the same path without the trailing slash.
				router.route(node.path, false);
			} else {
				import("../content/" + node.type + ".js").then(draw).catch(handle404);
			}
		});
	});

	router.listen();
	return `${validate_component(Html, "Html").$$render($$result, { route, node, allNodes }, {}, {})}`;
});

export default Component;
layout/global/head.svelte
/* generated by Svelte v3.23.2 */
import { create_ssr_component, escape } from "svelte/internal";

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { title } = $$props;
	if ($$props.title === void 0 && $$bindings.title && title !== void 0) $$bindings.title(title);

	return `<head><meta charset="${"utf-8"}">
  <meta name="${"viewport"}" content="${"width=device-width,initial-scale=1"}">

  <title>${escape(title)}</title>

  <link href="${"https://fonts.googleapis.com/css2?family=Rubik:ital,wght@0,300;0,700;1,300&display=swap"}" rel="${"stylesheet"}">
  <link rel="${"icon"}" type="${"image/svg+xml"}" href="${"/assets/favicon.svg"}">
  <link rel="${"stylesheet"}" href="${"/spa/bundle.css"}"></head>`;
});

export default Component;
layout/scripts/sort_by_date.svelte
/* generated by Svelte v3.23.2 */
import { create_ssr_component } from "svelte/internal";

const sortByDate = (items, order) => {
	items.sort((a, b) => {
		// Must have a field specifically named "date" to work.
		// Feel free to extend to other custom named date fields.
		if (a.fields.hasOwnProperty("date") && b.fields.hasOwnProperty("date")) {
			let aDate = new Date(a.fields.date);
			let bDate = new Date(b.fields.date);

			if (order == "oldest") {
				return aDate - bDate;
			}

			return bDate - aDate;
		}
	});

	return items;
};

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	return ``;
});

export default Component;
export { sortByDate };

Notably each starts with an import { create_ssr_component } from "svelte/internal"; ESM.

If you look at node_modules/svelte/internal/index.js it defines the create_ssr_component() function on line 1317, which is where you get the $$render functions. This ties back to the first snippet in this comment where:

let component = root.require(htmlWrapper).default;
console.log(component);

Yields the following value:

{ render: [Function: render], '$$render': [Function: $$render] }

@jimafisk
Copy link
Member Author

Using v8go instead of Node is getting a lot closer, but there are persistent issues. One of those is importing components with custom names. Right now we add SSR compiled JS to the ctx for each component. We don't have a smart way of doing this so we're simply naming based on the filename (e.g. nav.svelte becomes Nav, pages.svelte becomes Pages, etc). This does not account for importing a component with a custom name, for instance the default starter does something like this: import Uses from "../components/template.svelte";.

One solution is when reading the layout file, check for imports and when found assign the aliased name to the base component in the ctx. So using the above example:

SSRctx.RunScript("var Uses = Template", "create_ssr")
Whiteboard visualization

IMG_20200716_130347203

The downside is you would have a naming collision if different pages used the same alias for different components.

A more robust solution might be isolating the ctx to the current component and its specific imports using a custom gopack-like lookup. Would need to consider the performance implications of setting something like this up. Currently the v8go implementation is roughly as fast as the node implementation. I'll do some benchmarking once this is a little more stable.

@jimafisk
Copy link
Member Author

New thought: give each component a signature based on its path and use that as the variable name where it's declared and everywhere it's referenced, e.g. layout/components/grid.svelte would become var layout_components_grid;. That will avoid any trouble with two components with the same name in different folders, which is actually a different naming collision issue than what was mentioned in the previous comment.

When reading the file we will have to analyze relative paths (e.g. ../components/grid.svelte) to get the full path from root. Then change whatever it is imported as to the signature. Taking a different example:

Before converting
/* generated by Svelte v3.23.2 */
import {
	create_ssr_component,
	each,
	escape,
	validate_component
} from "svelte/internal";

import Uses from "../components/template.svelte";

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { title } = $$props, { description } = $$props;
	if ($$props.title === void 0 && $$bindings.title && title !== void 0) $$bindings.title(title);
	if ($$props.description === void 0 && $$bindings.description && description !== void 0) $$bindings.description(description);

	return `<h1>${escape(title)}</h1>

<div>${each(description, paragraph => `<p>${paragraph}</p>`)}</div>

${validate_component(Uses, "Uses").$$render($$result, { type: "pages" }, {}, {})}

<p><a href="${"/"}">Back home</a></p>`;
});

export default Component;
After converting
/* generated by Svelte v3.23.2 */
/*import {
	create_ssr_component,
	each,
	escape,
	validate_component
} from "svelte/internal";*/

/*import Uses from "../components/template.svelte";*/

var layout_content_pages = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { title } = $$props, { description } = $$props;
	if ($$props.title === void 0 && $$bindings.title && title !== void 0) $$bindings.title(title);
	if ($$props.description === void 0 && $$bindings.description && description !== void 0) $$bindings.description(description);

	return `<h1>${escape(title)}</h1>

<div>${each(description, paragraph => `<p>${paragraph}</p>`)}</div>

${validate_component(layout_components_template, "layout_component_template").$$render($$result, { type: "pages" }, {}, {})}

<p><a href="${"/"}">Back home</a></p>`;
});

/*export default Component;*/

Note we comment out the import/export statements in the SSR'd JS because v8go can't handle it. We essentially bundle the dependencies together manually in the vm ctx.

For the same example above, here is the Template component that is being imported:

Before being converted
/* generated by Svelte v3.23.2 */
import {
	add_attribute,
	create_ssr_component,
	escape,
	null_to_empty
} from "svelte/internal";

const css = {
	code: ".template.svelte-kyi9jr{display:flex;align-items:center}pre.svelte-kyi9jr{display:flex;padding-left:5px}code.svelte-kyi9jr{background-color:var(--base);padding:5px 10px}code.copied.svelte-kyi9jr{background-color:var(--accent)}button.svelte-kyi9jr{border:1px solid rgba(0,0,0,.1);background:white;padding:4px;border-top-right-radius:5px;border-bottom-right-radius:5px;cursor:pointer}",
	map: "{\"version\":3,\"file\":null,\"sources\":[null],\"sourcesContent\":[\"<script>\\n  export let type;\\n\\n  let path;\\n  let copyText = \\\"Copy\\\";\\n  var copy = async () => {\\n    if (!navigator.clipboard) {\\n      return\\n    }\\n    try {\\n      copyText = \\\"Copied\\\";\\n      await navigator.clipboard.writeText(path.innerHTML);\\n      setTimeout(() => copyText = \\\"Copy\\\", 500);\\n    } catch (err) {\\n      console.error('Failed to copy!', err)\\n    }\\n  }\\n</script>\\n\\n<div class=\\\"template\\\">\\n  <span>Template:</span>\\n  <pre>\\n    <code bind:this={path} class=\\\"{copyText}\\\">layout/content/{type}.svelte</code>\\n    <button on:click={copy}>{copyText}</button>\\n  </pre>\\n</div>\\n\\n<style>\\n  .template {\\n    display: flex;\\n    align-items: center;\\n  }\\n  pre {\\n    display: flex;\\n    padding-left: 5px;\\n  }\\n  code {\\n      background-color: var(--base);\\n      padding: 5px 10px;\\n  }\\n  code.copied {\\n      background-color: var(--accent);\\n  }\\n  button {\\n    border: 1px solid rgba(0,0,0,.1);\\n    background: white;\\n    padding: 4px;\\n    border-top-right-radius: 5px;\\n    border-bottom-right-radius: 5px;\\n    cursor: pointer;\\n  }\\n</style>\"],\"names\":[],\"mappings\":\"AA4BE,SAAS,cAAC,CAAC,AACT,OAAO,CAAE,IAAI,CACb,WAAW,CAAE,MAAM,AACrB,CAAC,AACD,GAAG,cAAC,CAAC,AACH,OAAO,CAAE,IAAI,CACb,YAAY,CAAE,GAAG,AACnB,CAAC,AACD,IAAI,cAAC,CAAC,AACF,gBAAgB,CAAE,IAAI,MAAM,CAAC,CAC7B,OAAO,CAAE,GAAG,CAAC,IAAI,AACrB,CAAC,AACD,IAAI,OAAO,cAAC,CAAC,AACT,gBAAgB,CAAE,IAAI,QAAQ,CAAC,AACnC,CAAC,AACD,MAAM,cAAC,CAAC,AACN,MAAM,CAAE,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAChC,UAAU,CAAE,KAAK,CACjB,OAAO,CAAE,GAAG,CACZ,uBAAuB,CAAE,GAAG,CAC5B,0BAA0B,CAAE,GAAG,CAC/B,MAAM,CAAE,OAAO,AACjB,CAAC\"}"
};

const Component = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { type } = $$props;
	let path;
	let copyText = "Copy";

	var copy = async () => {
		if (!navigator.clipboard) {
			return;
		}

		try {
			copyText = "Copied";
			await navigator.clipboard.writeText(path.innerHTML);
			setTimeout(() => copyText = "Copy", 500);
		} catch(err) {
			console.error("Failed to copy!", err);
		}
	};

	if ($$props.type === void 0 && $$bindings.type && type !== void 0) $$bindings.type(type);
	$$result.css.add(css);

	return `<div class="${"template svelte-kyi9jr"}"><span>Template:</span>
  <pre class="${"svelte-kyi9jr"}"><code class="${escape(null_to_empty(copyText)) + " svelte-kyi9jr"}"${add_attribute("this", path, 1)}>layout/content/${escape(type)}.svelte</code>
    <button class="${"svelte-kyi9jr"}">${escape(copyText)}</button></pre>
</div>`;
});

export default Component;
After being converted
/* generated by Svelte v3.23.2 */
/*import {
	add_attribute,
	create_ssr_component,
	escape,
	null_to_empty
} from "svelte/internal";*/

var css = {
	code: ".template.svelte-kyi9jr{display:flex;align-items:center}pre.svelte-kyi9jr{display:flex;padding-left:5px}code.svelte-kyi9jr{background-color:var(--base);padding:5px 10px}code.copied.svelte-kyi9jr{background-color:var(--accent)}button.svelte-kyi9jr{border:1px solid rgba(0,0,0,.1);background:white;padding:4px;border-top-right-radius:5px;border-bottom-right-radius:5px;cursor:pointer}",
	map: "{\"version\":3,\"file\":null,\"sources\":[null],\"sourcesContent\":[\"<script>\\n  export let type;\\n\\n  let path;\\n  let copyText = \\\"Copy\\\";\\n  var copy = async () => {\\n    if (!navigator.clipboard) {\\n      return\\n    }\\n    try {\\n      copyText = \\\"Copied\\\";\\n      await navigator.clipboard.writeText(path.innerHTML);\\n      setTimeout(() => copyText = \\\"Copy\\\", 500);\\n    } catch (err) {\\n      console.error('Failed to copy!', err)\\n    }\\n  }\\n</script>\\n\\n<div class=\\\"template\\\">\\n  <span>Template:</span>\\n  <pre>\\n    <code bind:this={path} class=\\\"{copyText}\\\">layout/content/{type}.svelte</code>\\n    <button on:click={copy}>{copyText}</button>\\n  </pre>\\n</div>\\n\\n<style>\\n  .template {\\n    display: flex;\\n    align-items: center;\\n  }\\n  pre {\\n    display: flex;\\n    padding-left: 5px;\\n  }\\n  code {\\n      background-color: var(--base);\\n      padding: 5px 10px;\\n  }\\n  code.copied {\\n      background-color: var(--accent);\\n  }\\n  button {\\n    border: 1px solid rgba(0,0,0,.1);\\n    background: white;\\n    padding: 4px;\\n    border-top-right-radius: 5px;\\n    border-bottom-right-radius: 5px;\\n    cursor: pointer;\\n  }\\n</style>\"],\"names\":[],\"mappings\":\"AA4BE,SAAS,cAAC,CAAC,AACT,OAAO,CAAE,IAAI,CACb,WAAW,CAAE,MAAM,AACrB,CAAC,AACD,GAAG,cAAC,CAAC,AACH,OAAO,CAAE,IAAI,CACb,YAAY,CAAE,GAAG,AACnB,CAAC,AACD,IAAI,cAAC,CAAC,AACF,gBAAgB,CAAE,IAAI,MAAM,CAAC,CAC7B,OAAO,CAAE,GAAG,CAAC,IAAI,AACrB,CAAC,AACD,IAAI,OAAO,cAAC,CAAC,AACT,gBAAgB,CAAE,IAAI,QAAQ,CAAC,AACnC,CAAC,AACD,MAAM,cAAC,CAAC,AACN,MAAM,CAAE,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAChC,UAAU,CAAE,KAAK,CACjB,OAAO,CAAE,GAAG,CACZ,uBAAuB,CAAE,GAAG,CAC5B,0BAA0B,CAAE,GAAG,CAC/B,MAAM,CAAE,OAAO,AACjB,CAAC\"}"
};

var layout_components_template = create_ssr_component(($$result, $$props, $$bindings, $$slots) => {
	let { type } = $$props;
	let path;
	let copyText = "Copy";

	var copy = async () => {
		if (!navigator.clipboard) {
			return;
		}

		try {
			copyText = "Copied";
			await navigator.clipboard.writeText(path.innerHTML);
			setTimeout(() => copyText = "Copy", 500);
		} catch(err) {
			console.error("Failed to copy!", err);
		}
	};

	if ($$props.type === void 0 && $$bindings.type && type !== void 0) $$bindings.type(type);
	$$result.css.add(css);

	return `<div class="${"template svelte-kyi9jr"}"><span>Template:</span>
  <pre class="${"svelte-kyi9jr"}"><code class="${escape(null_to_empty(copyText)) + " svelte-kyi9jr"}"${add_attribute("this", path, 1)}>layout/content/${escape(type)}.svelte</code>
    <button class="${"svelte-kyi9jr"}">${escape(copyText)}</button></pre>
</div>`;
});

/*export default Component;*/

@jimafisk
Copy link
Member Author

jimafisk commented Jul 30, 2020

Benchmarking:

Attempt # plenti build --nodejs=true (With Node) plenti build (No Node)
1 509.542643ms 607.597406ms
2 493.929817ms 449.384268ms
3 490.312936ms 446.878569ms
4 477.554289ms 441.949252ms
5 484.30959ms 440.094771ms
6 492.960634ms 454.619161ms
7 488.414887ms 447.365403ms
8 488.466514ms 440.468207ms
9 481.036876ms 452.058979ms
10 480.749496ms 440.164956ms
average 488.7277682ms 462.0580972ms

@jimafisk
Copy link
Member Author

In this hacker news thread leaveyou wrote:

Please someone make a Svelte compiler in Go.. I can't stand Node.js and I can't install it on my machine without feeling dirty.

batisteo responded with:

There’s a decent JS to JS compiler (replacement of Babel) in Rust which understand TypeScript: https://swc-project.github.io

I remember looking into SWC early on, but doesn't seem like it made it onto this thread for some reason. I have some concerns about the amount of processing we're having to do to run everything in v8, but probably not going to make any switches in the short term. If there was a Svelte compiler written in Go with an API it would make things faster and more consistent for this project. It would also be nice not to require CGO so we could start cross compiling for Windows again.

@zaydek
Copy link

zaydek commented Feb 5, 2021

@jimafisk I just caught your talk on creating a Go SSG for Svelte: https://www.youtube.com/watch?v=4tD3Jz7JUfk, I’m about halfway through. I thought I’d comment here because you and I seem to be interested in the same problem space, I even tried making a SSG for React and just gave up and decided Svelte is the future. This is as far as I got: https://github.com/zaydek/retro.

I figured out how to solve for these problems:

  • Using a Makefile to simplify cross-platform compilation
  • Using NPM for distribution (I think you’re shipping a CLI binary, not sure)
  • Using esbuild as a bundler
  • Page-based routing
  • Resolving / caching props on the server (e.g. getStaticProps or whatever in Next.js / Remix Run)
  • Etc.

All that being said, I realized React is actually very problematic as the frontend target because it’s so reliant on Babel / ESLint, etc. that even if you solve the backend problem, you still have the whole DX problem which is significant and I don’t care enough about to solve simultaneously.

So I thought I’d leave a comment here to get your attention and see if you want to connect. 🙂 Feel free to email me at zaydekdotcom [at] gmail or DM me on Twitter @username_ZAYDEK. I couldn’t easily find your email so I thought I’d just comment here. 😬

Solid work you got going on here.

@jimafisk
Copy link
Member Author

jimafisk commented Feb 5, 2021

Retro looks cool @zaydek! I'd love to connect, just followed you on twitter and sent you an email :)

@jimafisk
Copy link
Member Author

jimafisk commented Feb 8, 2021

How the svelte compiler works: https://dev.to/joshnuss/svelte-compiler-under-the-hood-4j20 + simplified example: https://github.com/joshnuss/micro-svelte-compiler

@jimafisk
Copy link
Member Author

I took another look at a pure Golang javascript interpreter to see if it was feasible to drop the complexities that come with a cgo dependency. Currently it does seem like that would come at a significant performance cost.

Rough benchmarking of Goja against v8go:

Attempt # Goja v8go
1 3.222790268s 56.900828ms
2 3.16795169s 56.840583ms
3 3.228237879s 57.287548ms
4 3.262925706s 57.178266ms
5 3.188262189s 56.869266ms
Benchmarking code
package main

import (
  "fmt"
  "time"

  "github.com/dop251/goja"
  "rogchap.com/v8go"
)

func main() {
  start := time.Now()
  gojavm := goja.New()
  result, err := gojavm.RunString(`
  function factorial(n) {
    return n === 1 ? n : n * factorial(--n);
  }
  
  var i = 0;
  
  while (i++ < 1e6) {
    factorial(10);
  }
  `)
  if err != nil {
    fmt.Printf("Goja error: %v", err)
  }
  if result.Export().(int64) == 3628800 {
    elapsed := time.Since(start)
    fmt.Println(elapsed)
  }

  start = time.Now()
  vm, _ := v8go.NewIsolate()
  ctx1, _ := v8go.NewContext(vm)
  result1, err1 := ctx1.RunScript(`
  function factorial(n) {
    return n === 1 ? n : n * factorial(--n);
  }
  
  var i = 0;
  
  while (i++ < 1e6) {
    factorial(10);
  }
  `, "math.js")
  if err1 != nil {
    fmt.Printf("Eval error: %v", err1)
  }
  if result1.String() == "3628800" {
    elapsed := time.Since(start)
    fmt.Println(elapsed)
  }
}

From Goja's README:

Although it's faster than many scripting language implementations in Go I have seen (for example it's 6-7 times faster than otto on average) it is not a replacement for V8 or SpiderMonkey or any other general-purpose JavaScript engine.

Running a factorial like the example used for the benchmark is a heavy computation:

If most of the work is done in javascript (for example crypto or any other heavy calculations) you are definitely better off with V8.

The typical Plenti site might be lighter on computational requirements. If down the road we start moving Svelte compiling features into Go, there's a chance the performance between these projects would even out a bit:

If you need a scripting language that drives an engine written in Go so that you need to make frequent calls between Go and javascript passing complex data structures then the cgo overhead may outweigh the benefits of having a faster javascript engine.

Another consideration would be full ES6 compatibility: https://github.com/dop251/goja/milestone/1?closed=1

@jimafisk
Copy link
Member Author

Also took a quick look at https://github.com/gojisvm/gojis, but the docs are still WIP (https://gojisvm.github.io/api.html) so it would need to mature before evaluating more seriously.

Tried running the Svelte compiler.js inside Goja with es6 support: go get github.com/dop251/goja@es6. Unfortunately it does not seem to have support for block scoped variables like const and let: dop251/goja#167 (comment)

Test Goja code
package main

import (
	"fmt"
	"io/ioutil"

	"github.com/dop251/goja"
)

func main() {
	compiler, err := ioutil.ReadFile("compiler.js")
	if err != nil {
		fmt.Println("Can't read compiler.js")

	}
	vm := goja.New()
	result, err := vm.RunScript("test", string(compiler))
	if err != nil {
		fmt.Printf("Goja error: %v", err)
	}
	println(result.Export())
}
Error message
Goja error: SyntaxError: test: Line 7:2 Unexpected reserved word (and 14443 more errors)panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x702239]

goroutine 1 [running]:
main.main()
        /home/jimafisk/Desktop/benchmarks/test.go:24 +0x219

In case it's useful, here's what you currently get out of the box with Plenti for build times:

Default "learner" starter with v8go build benchmarks (ran 15 times individually)
  • 993.95377ms
  • 888.726728ms
  • 983.571047ms
  • 953.345185ms
  • 1.063483094s
  • 918.495382ms
  • 961.942967ms
  • 944.106465ms
  • 953.192401ms
  • 968.931752ms
  • 968.555285ms
  • 1.069975253s
  • 1.088950308s
  • 974.294567ms
  • 993.814659ms

@jimafisk
Copy link
Member Author

It seems like Goja has come a long way since I last tested it. The ES6 milestone has been completed and I ran some performance tests against v8go again and the results were very different from last time:

Attempt # Script Size Goja V8go
1 Big 163ns 🤏 161ns ✔️
1 Small 167ns ✔️ 180ns
2 Big 174ns 146ns ✔️
2 Small 250ns 175ns ✔️
3 Big 245ns 182ns ✔️
3 Small 179ns 🤏 175ns ✔️
4 Big 179ns 157ns ✔️
4 Small 234ns 175ns ✔️
5 Big 172ns 148ns ✔️
5 Small 252ns 170ns ✔️
6 Big 235ns 164ns ✔️
6 Small 171ns ✔️ 182ns
7 Big 168ns ✔️ 173ns 🤏
7 Small 149ns 131ns ✔️
8 Big 152ns ✔️ 163ns
8 Small 168ns 138ns ✔️
9 Big 141ns 🤏 139ns ✔️
9 Small 169ns ✔️ 239ns
10 Big 158ns 151ns ✔️
10 Small 154ns ✔️ 159ns 🤏

🤏 = within 5ns

So the numbers are much closer, in some cases goja even comes out ahead. My next step is to try using it to actually compile some svelte components and see how that goes.

Benchmarking code
package main

import (
	"fmt"
	"time"

	"github.com/dop251/goja"

	"rogchap.com/v8go"
)

func main() {
	js1 := `
function factorial(n) {
	return n === 1 ? n : n * factorial(--n);
}	
var i = 0;	
while (i++ < 1e6) {
	factorial(10);
}`
	js2 := `
function factorial(n) {
	return n === 1 ? n : n * factorial(--n);
}
var i = 0;
while (i++ < 10) {
	factorial(10);
}`
	runV8go(js1)
	runGoja(js1)

	runV8go(js2)
	runGoja(js2)
}

func runGoja(js string) {
	elapsed := time.Since(time.Now())
	vm := goja.New()
	v, err := vm.RunString(js)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Print(v)
	fmt.Printf(" goja " + elapsed.String() + "\n")
}

func runV8go(js string) {
	elapsed := time.Since(time.Now())
	vm, _ := v8go.NewContext(nil)
	v, err := vm.RunScript(js, "")
	if err != nil {
		fmt.Println(err)
	}
	fmt.Print(v)
	fmt.Printf(" v8 " + elapsed.String() + "\n")
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic high high priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants