Automatically recompile and reload Diet templates in the background in development mode

[s-ludwig]

During development, a file system watcher could be used to monitor file changes in the Diet templates that make up the application. Each diet template, instead of being statically compiled into the application, would then be compiled as a separate shared library/DLL and dynamically loaded and unloaded as needed.

[etcimon]

It could be possible to have a separate library for compiling diet templates into a .d format and generating the dub information, so that a vibe.d project can refer to them as a dependency?

[s-ludwig]

That would definitely be possible (just basically writeToFile("somefile.d", dietStringParser!(...));), but it wouldn't solve this particular issue, since the application would still have to be relinked and restarted in that case.

[s-ludwig]

Unless you mean to use DUB just as a tool for building the dynamic library. In that case, the discussed support for single-file packages would come in handy.

[etcimon]

Oh I see where you're getting at. My approach was to make the server restart less damaging, but reloading the diet templates as a shared object during runtime could do well. In my current design of a config manager I'm also wondering if the server could be restarted without closing the processes (to re-build routes and listeners).

[MartinNowak]

I'll try to work on that, as it's a good showcase for shared libraries.
What does during development mean and which process would monitor the filesystem, dub?
The other problem is that render uses templates, so it's a bit tricky to forward the arguments through a function pointer.

[s-ludwig]

That would definitely be great, I've actually heard that argument quite a number of times, that D was bad because of this increase of turnaround time and that people would rather stay with their dynamic interpreted language.

"During development", I'd say, should just mean some kind of opt-in switch, probably using -version=. Using -debug= would have the advantage that it could be directly specified on the DUB command line, but that would of course not be very clean semantically.

For file system monitoring, watchDirectory* could be used directly from within the vibe.d process itself. Building of the individual templates could be done using DUB by generating a dummy package with the vibe.d project as a dependency to get all the build settings.

Regarding the template issue, even if it slightly changes the semantics, I think using a Variant[] parameter to forward the arguments, similar to what used to be done for renderCompat, should usually be fine.

* This is currently only implemented for the win32 driver, but I could invest some time to get that running in the libevent driver, too.

[zhaopuming]

So this mechanism is very similar to Martin's repl mechanism :-) Hopefully one day we can also have dub repl just like lein repl in Clojure.

And for a more general case, can we make this mechanism work for other parts of the application so that when each part is modifed, it will be hot compiled and reloaded.

Of couse, before that we need to define a part, or a function unit. As a reference, vert.x has a 'vertex' concept that encapsulates a function unit and each they communicate with each other via an Eventbus, using messages. That behaves very much like Erlang/Scala's actor model. We already have all the tools for actor model support in vibe, so once we make it a more official pattern, we can do this hot recompile/reload on these actor/units. Then diet temlate would be just a special case of an actor (which is called directly).

[s-ludwig]

What I just fear is that generalizing this to ordinary D modules makes it too easy for people to step on their own feet by for example changing the data layout of certain types. Java has a lot more possibilities to detect such situations due to the detailed run time reflection, but a D application would either just crash, or produce wrong output.

[zhaopuming]

You mean data layout for messages? maybe we need another unit instead of D modules, which is a source code unit. vert.x's module is like a plugin, and each module has its own project, modules communicate only by messages (JSON). Play framework has a similar approach. These are runtime plugin systems for hot reload.

[s-ludwig]

The problem I was thinking about is when you allow a component to reuse its old memory after it has been reloaded, which would be the most efficient approach, but would be prone to silent data corruption errors when for example a struct layout has changed after a reload. If, on the other hand, the component would have to start with a clean state every time, that issue would be gone and message passing should mostly be fine*. However, that would mean that components may not depend on the state of each other, which opens a lot of potential for high level issues.

Maybe a hybrid approach, where each component gets the chance to explicitly serialize it's state before going down and then gets that serialized blob of data when starting up next time to restore its state. Still leaves potential for silently lost state, but at least enables the possibility to make a reload seamless.

BTW, there is Cmsed by @rikkimax, which supports component reloading. I'm not sure if it uses any form of message passing or another runtime machanism, or simply allows sharing memory between the components.

In general I think architecturally it would be the best solution to keep this functionality to a separate framework, either a higher-level one like Cmsed, or a totally generic one (one that is vibe.d compatible, of course) - because it should be possible to separate it and to avoid feature creep in vibe.d itself (there is already a lot of functionality that would be better off in Phobos for example). Template reloading on the other hand would just be a development aid and would not not really be part of the overall architecture.

* If component A imports a module from component B that defines a struct and then this struct gets sent from component A to component B, it's still possible to get incompatible data layouts if A decides to change the struct layout after a reload. So to mitigate this, it would be necessary to track dependencies between components and reload all affected components at once.

[rikkimax]

Cmsed does not yet support reloading of templates ext. at runtime just yet. This is mostly from the state of my Actor framework Dakka. It is so far not able to call methods on other nodes. But that is not my next target, but the other one. Current is singleton actors, yes awful but hey great for controller classes.

The policies I was going to use was quite simple:

• Template updates:
  • Code regenerated
• After code regenerated for templates or routes change
  • Recompile appropriate templates
  • Stop previous nodes
  • Start new nodes
• On changes to data models
  • Recompile all routes

Basically it shouldn't automatically include in the compilation any code that isn't required. Use the route as being the source file to be compiled and the rest just follows so to speak. But have the appropriate import paths added.

Now this doesn't solve the dependency problem. My method for this is to have a directory by this which essentially allows for single dependency binary file and a directory with all import files.
Templates should be generated and then committed to version control so that if you had a package that only provides templates then they will be available to the routes if they are in that dependency package.

I am worried about how this will hold up on the routers end, and pushing it out to so many different processes to a request. But atleast this is only for testing and not production level usage.

But right now this is purely theory. I'm personally open to making this less Cmsed specific if thats what you guys want.

[zhaopuming]

@s-ludwig I agree your thoughts about separating vibe.d and the component framework (akka like). Cmsed looks interesing :)

About memory layout, is there an aproach to diff the data layouts of each modification? But that would become too complicated. In JVM which I'm more familiar, hotswap is a very complicated problem that leads to a commercial product, JRebel. I would not sugget we go that far :-)

[zhaopuming]

@rikkimax Does Dakka use the same technique to Akka to send messages between actors? like:

remoteActor.send(action, &data, &handler);

or do some compile time magic to make actor RPC just like function calls?

remoteActor.action(data, &handler); // this translate into remoteActor.send(action, &data, &handler);

or even better, use technich similar to vibe.d, making async calls look like sync?

auto result = remoteActior.action(data); // suspends current Task waiting for remote actor's reply

[rikkimax]

@zhaopuming Lots and lots of CTFE magic. I certainly love that stuff.

Example:

auto aref = new ActorRef!MyActorA;
aref.onChildError(null, "okay hi there!");
aref.test("Hiii from the client");
aref.die();

Ignoring that so far remote method call aren't implemented (but onChildError is as its a special case) here is a sample code I'm using to test with. Keep in mind it can create an instance of MyActorA locally. The reason its not, is because of the capabilities feature.
Essentially every class has a list of capabilities that it requires to be usable. Every node at start up can register its capabilities. So for testing its rather easy to break it up.
https://github.com/rikkimax/dakka/blob/b4b9611e165c33ebdcb8afffbb3268775d9b3f2f/source/app.d

It will support async and sync calls, based upon if it has a return value. I.e. ref, out or a return type.

[etcimon]

but would be prone to silent data corruption errors when for example a struct layout has changed after a reload.

There's also the option of the front-end pulling data directly from some (cache?) storage to force a stable API on it. I'm leaning more towards that, but sending serialized data could also be a stable API type of option.

[s-ludwig]

About memory layout, is there an aproach to diff the data layouts of each modification?

One idea would be to create a unique "fingerprint" of each type that uniquely identifies the data layout and always send that together with the actual data. The receiver could then at least detect a mismatch and perform an appropriate action.

[etcimon]

Or the diet compiler could find the modules/files in which the types are defined and detect changes/import those automatically.

[etcimon]

On another hand, for frequent iterations I'm thinking it would be a good idea to enhance the serveStaticFiles into a serveDynamicFiles, to be able to parse-replace an html document with contents containing commands like <!-- include:file/addr.html -->, and <!-- var:$1 --> where $1 can be extracted from the address in the vibe.d router or, <!-- get:$1~"data" -> from a key-value store, or even <!-- acl:admin,registered --> for ACL. This restriction of the API can serve to develop a fast, dynamic interface with vibe.d using html files, javascript models and json serialization/deserialization instead.

The filesystem can prove to be a great document database vs. a dll or binary, Javascript is great for user-interface interactivity, and JSON is a great back-end connector.

For debugging, using a <!-- debug { --> html <!-- } --> could be used when designing an interface with vibe.d as a backend (the stuff in debug appears in a browser when double-clicking the html file, but doesn't get delivered by the vibe.d server unless the url is queried with a specific flag)

e.g. For redaction of a blog post, a <!-- debug { --> <link rel="stylesheet" type="text/css" href="mystyle.css"> <!-- } --> would allow the use of pretty much any HTML WYSIWYG application.

Also, it's easier for a CMS to interface with HTML files with a restricted API, than to interface with a dynamic templating format with (hard to parse) D code inside it.

Finally, it's easier this way for ANY front-end developer around the world to implement third-party templates like those from http://themeforest.net .. copy-paste?

[MartinNowak]

The other problem is that render uses templates, so it's a bit tricky to forward the arguments through a function pointer.

The main problem is that the alias arguments are only accessible in nested functions,
but it's not possible to compile a nested function without it's parent (i.e. the call-site of render).
So we can either revert the logic, so that the render template calls back to get fed with the arguments. Not sure how to exactly implement this though.
Or we try to pass all arguments as copies or via ref. It doesn't seems possible to pass Types or aliases to other symbols this way.

[s-ludwig]

I'd just do it like renderCompat does it and pass all of the values as a Variant[], that would avoid issues where a private type would otherwise be needed to define an internal function signature (when passing by ref, for example). If types or other kinds of aliases are passed, the code could simply output a pragmanotice and fall back to ordinary compilation.

[MartinNowak]

I'd just do it like renderCompat does it and pass all of the values as a Variant[]

Variant would at least avoid the necessity to import most of the project and vibe.d.

If types or other kinds of aliases are passed, the code could simply output a pragma notice and fall back to ordinary compilation.

Yeah, I also arrived there, still juggling some other ideas though.

There are two remaining questions.

• How do we compile the shared library?
  I guess nobody want this sort of build script code in vibe.d. Maybe we can move it into dub which
  already knows all import paths.
• How to improve compile times?
  When the view template has to import much of vibe.d or phobos we won't gain
  much improvement in compile times (mostly due to parsing). A file watcher is nice,
  but a 5 second refresh still sucks.
  Using Variant to pass the arguments might help to reduce import dependencies.

[rikkimax]

I've built wrappers around HTTPServerRequest and HTTPResponse (HTTPServerResponse was too specific) upon Dakka https://github.com/rikkimax/dakka/blob/master/source/vibe-d_wrappers/dakka/vibe/client.d
So theoretically if you were able to set up Dakka correctly and the routers on both ends it would be possible to use it for live reloading of routes ext.

Also for reference: https://github.com/rikkimax/skeleton https://github.com/rikkimax/livereload (livereload handles the actual recompilation and rerunning of Dakka nodes).

[MartinNowak]

I fiddled around with this a bit and here are my findings.

I still haven't found a way to pass data from the app to a shared
library without severely restricting what is possible with a diet
template.

• Variant doesn't even allow to access fields of UDTs and thus fails to support ranges. We could try to come up with a better Variant and use inputRangeObject.
• Passing the real types whenever possible (must be public and non-Voldemort) would resolve many issues. It'd require to import the app when recompiling the template. This would be almost as slow as recompiling the whole app and requires linker trickery (--export-dynamic, linking the template against the app).
• Another option would be to recompile the whole app, load it as a shared library and taking out the render function.
  This would get rid of all the type passing issues.

The latter two solutions have the severe drawback, that they require to import the app.
Let's recap our goals, we want to speed up development by automatically recompiling diet templates whenever they change.
As I said earlier a filewatcher is nice, but fairly trivial to achieve with a script like this.

#!/bin/sh

if [ $# -lt 1 ]; then
    echo "Usage: $0 <cmd>"
    exit 1
fi
APP=$*

dub build
$* &
pid=$!
while inotifywait -r -e modify ${PWD}; do
    kill -TERM $pid
    dub build
    $* &
    pid=$!
done

While that works pretty nice, it's the the 5s turnaround time that we need to get down to about 1s.

Let's break it down for a simple vibe.d project (vibe.d/examples/diet).

From the 4.3s it takes to run dub build with a pre-built libvibe-d.a after changing the diet.dt template, the linker takes a surprisingly big 2.3s. Well the final binary is 15M and libvibe-d.a is 94M, so the linker has quite something to do, but ld.gold can do the same thing 5x faster.
It becomes even better when linking against a shared libvibe-d.so, because the linker doesn't have to copy code into the binary.
Now the final binary is only 940K while libvibe-d.so is 13M, not sure where the big 94M -> 13M shrink comes from.

Shrinking the time needed to compile the app will be a bit harder. For debug builds the compiler spends a big part of it's time lexing and parsing all imported source files.
So what would help is to reduce the number of modules that must be opened and processed.
Here is a list of all dependencies for that little app. The 220 files (236626 LOC, 8MB) shed a different light on the 0.87s it takes to compile the app.
It would be nice to get lazy static imports in D at some point, but until then
using local imports is the best option (next to not importing anything) to avoid
useless imports in client code. For example when recompiling a vibe.d app the compiler
should not parse any openssl or libevent2 headers, because those are
dependencies of vibe.d, not the app. Likewise the app doesn't use any std.datetime code, but the module is still parsed on every recompile.

The last big part is spend by dub itself which is loading metadata and checks
git versions, dependencies, and upgrades. While there might be some potential
to speed this up, it might be worth to integrate the above inotify script into
dub itself, so that calling dub run --auto-rebuild would allow to rebuild the
app on-the-fly with much less overhead.

Overall I think we can reach the 1s even for real-sized apps.
This approach is much more worthwhile IMO than to use amputated diet templates.

[etcimon]

I still haven't found a way to pass data from the app to a shared
library without severely restricting what is possible with a diet
template.

I've had a solution in mind that involved sharing the types between multiple libraries, by specifically putting them in a "common" D file or folder of files.

For the speed issue, I also think a better solution would be to build a diet template compiler as a separate application, rather than using purely CTFE. This way, a simple set of D files that consist of functions and string appenders would be generated for each diet template. Maybe the type definitions could be isolated by the diet compiler itself.

Thanks for the benchmarks =)

[etcimon]

I'm thinking of DScanner, and it would be of great use to search for the diet template render calls and resolve the types used to instantiate it, if you find those types you can generate the corresponding diet D files with proper function signatures directly from a GC collected diet compiler and make a very fast to compile/link shared library.

[MartinNowak]

I've had a solution in mind that involved sharing the types between multiple libraries, by specifically putting them in a "common" D file or folder of files.

Yes, that's what you would need to do, although you can cheat, use --export-dynamic and link the template against your app.

For the speed issue, I also think a better solution would be to build a diet template compiler as a separate application, rather than using purely CTFE.

Yeah with many templates CTFE will become the bottleneck.

This way, a simple set of D files that consist of functions and string appenders would be generated for each diet template.

They'd also contain arbitrary code otherwise it would be trivial.

Maybe the type definitions could be isolated by the diet compiler itself.

Still wouldn't allow to access private or voldemort types.

I'm thinking of DScanner, and it would be of great use to search for the diet template render calls and resolve the types used to instantiate it, if you find those types you can generate the corresponding diet D files with proper function signatures directly from a GC collected diet compiler and make a very fast to compile/link shared library.

Interesting idea, DScanner isn't ready yet for code rewriting though.
Also I can generate those D files from within the render template itself, where I have direct access to the types.

[MartinNowak]

BTW, automatic reloading is always an option for further improvements, but it also benefits from any compile time reductions.

[Beyarz]

Any updates on this? @s-ludwig

[schveiguy]

FYI, anyone who wants to just make HTML changes to their diet templates can use the new live mode to avoid having to recompile. And it's much faster than a recompile.

See the latest diet-ng description here: https://github.com/rejectedsoftware/diet-ng#experimental-html-live-mode

I'm wondering if this issue shouldn't be moved to diet-ng (if not already) and closed here. An automatic recompile would be very nice intermediate step in addition to the live mode.

[Geod24]

Going to close this on account that:

• @schveiguy implemented the live-mode in diet-ng;
• This is a diet-ng issue (and in fact, I believe we should reduce the dependency);