[Discussion] How to handle multiple languages with the CLI

[johnnyman727]

I recently started investing more effort towards using Rust on Tessel 2. After opening a preliminary PR (#511), @rwaldron brought up some concerns about the code getting too complicated. I wanted to open up a general dialogue about how we might handle multiple languages being deployed to T2.

My initial thought was to use this existing t2-cli to deploy code for any type of language. The rationale is simple: because deployment is only one of a handful of actions one might take upon a T2 (among wifi configuration, listing available Tessels, setting up access points, erasing code, etc.). It doesn't seem like much of a stretch to me to have one similar code base and then as soon as code starts being deployed, we detect the language and route the rest of the deployment to a language-specific process. For example, in pseudo-code:

// Figure out which language this project is written in
return detectLanguage(opts)
.then((language) => {
  // Load the language deployment code and pass it the project info
  return loadLanguageDeployment(language)(opts);
}); 

To be clear, I don't mean that the deployment of any other language is a separate program written in that language - I just mean that we have a separate piece of JS that deals with it (because some languages have compilation steps and different ways to run the code).

The downside is that we have more deployment code to maintain and it continues to grow as we support more languages. Another downside is that folks coming from a specific background (like python) might expect to download the tool using that language's package manager (like pip).

@rwaldron suggests we have a CLI written in every language that is supported on the Tessel 2. So, if we want to support Rust, we would need to write a CLI in Rust that supports all the same actions this t2-cli supports. That way, Rustaceans (or Pythonitas or whatever) can interact with the CLI in a way that is more idiomatic to the language at hand and we keep each CLI deployment code more simple.

I'd love to hear other thoughts and/or suggestions on the topic!

[Frijol]

The central conflict seems to me to be between doing it in the most correct way vs doing it in a way we have the ability to maintain. By which I mean: creating the CLI in every language seems like the canonically correct approach, but I don't think we have the resources to actually create and maintain that many CLIs.

[reconbot]

What languages are we supporting and how are we keeping the binaries up to date?

The basic run a command and copy everything action could easily be duplicated for all languages. A binary cross compile of c extension or whatever is a taller oder. Even then I don't see why many cli's are needed. That seems like a lot of duplicate code. I think a standard abstraction and breaking it out into plugins might be a good approach.

I also don't think that we need to fully serve every language like we serve JS.

[johnnyman727]

What languages are we supporting and how are we keeping the binaries up to date?

We stated that we would support Rust and Python. I think plugins that we can submodule make it more feasible to have the community that is excited about any particular language have ownership over it rather than relying on our small team to cover more bases. Certainly, a separate CLI would have the same effect.

[MrNice]

I'm completely unfamiliar with any of the technological progress which has enabled the t2-cli to work. With that in mind, the following is entirely based off of intuition, please disregard as necessary.

THE HOLY GRAIL GOAL

Let every user plug a t2 into their computer, install the CLI using their favorite package manager (mine is npm, some people like apt, others brew or yum, still others like gem, pip, crate, and cargo), and deploy hello_world.* with tessel run

A SOLUTION

Why not create the CLI's internals in a portable language, and deliver that, both as a binary and as a library which can then be integrated into every other language we support through Foreign Function Interfaces?

Adding a new language then becomes straightforward:

1. Add the runtime to the t2's base image
2. Tell the CLI how to give code to that language's runtime
3. Wrap the CLI, following each language's conventions

Details

Creating a portable CLI library

We can write most of the CLI in Rustlang, which also offers itself up nicely for bare metal optimizations from embedded systems geeks. Rust loves FFI: https://doc.rust-lang.org/book/rust-inside-other-languages.html

Furthermore, the CLI could then be wrapped in a daemon process in order to power tools such as an automatic firmware / device flasher or a Tessel2 cluster status reader.

A similar approach can (and maybe should?) be used to give tessel and USB module support to every language we want to add. In fact, I think you're already doing something similar with the connection between OpenWRT and the IO Coprocessor.

Deploying the command line

Every attempted installation should try to put the tessel rustlang CLI in the same place.

Every language I've briefly mentioned has some ability to install related dependencies. A worst case example: when you install an electron-shell application with npm install -g vmd, npm ends up grabbing a prebuilt zipping electron-shell binary, extracting it, and connecting the vmd script to it.

I also found a python package which allows you to distribute rust extensions for python: https://github.com/novocaine/rust-python-ext

So long as the main CLI plumbing is in the same place, creating language specific CLI's would be a simple act of creating glue code via FFI, IPC, or simply by shelling out, the way the CLI mostly just runs shell commands on the tessel2.

THE TESSEL COMMAND

Scenario

Ideally, I should be able to deploy a project which is a mix of JavaScript and Rust to a tessel, so that I can have good network interfaces (socket.io...?) combined with good bare metal performance without waiting for Crankshaft's optimizing compiler to kick in. The codebases communicate with standard linux inter-process communication (ipc), and the codebases manage the named pipe (or whatever)'s location.

rustlang: http://alexcrichton.com/ipc-rs/ipc/index.html
nodejs: https://github.com/RIAEvangelist/node-ipc

Example Execution

The new tessel command would do the following:

1. Traverse up the filesystem tree to find a language agnostic package.json which describes the project, including which languages the project utilizes and where each language's codebase is.
2. Discover this is a rust and nodejs 6.0.0 project
3. Ensure that nodejs 6.0.0 is available on the tessel2
4. If necessary because of changes, crosscompile nodejs binary deps and rustlang deps
5. Idempotently deploy nodejs javascript, binary deps, and rust binary
6. Tell the t2 it's done

The tessel then starts the processes in parallel

Final Analysis

With this approach, the tessel CLI becomes a consistent platform which is scriptable from other languages. Sensible conventions should allow the tessel command to work with almost any language, given a suitable package.json. Furthermore, language specific idiomatic interfaces can be easily created and then installed using the language's favorite dependency manager.

There are drawbacks, of course.

We've already written the command line in JavaScript. Throwing that work away and starting from scratch only makes sense if we can make the new CLI 5-10x "better" at achieving our goals. Many lessons have been learned through writing the CLI twice, though, and we should have a pretty good estimate of what we're doing before we do it, which speeds along porting quite nicely.

I don't want to write this in C, but Rust is an evolving language. Since the t2 is a JS microcontroller, I don't think this matters.

There's increased complexity in management, and in communicating with end users if something goes wrong:

$ pyssel run
Warning: There is an updated tessel command line.
Warning: New goodies: cloud deployment
Warning: Please run `pyssel update` to update to the new CLI
Warning: If you use multiple tessel command lines (russel, pyssel)
Warning: You will need to update each in order to use new features
$ pyssel update
Updated tessel CLI to version 1.3.4
$ cd ../rustproj
$ rustel run
Warning: There is a difference between your rustel version and your tessel CLI version
Warning: All your commands will still work, but you're missing out on goodies
Warning: New goodies: cloud deployment
Warning: Please run `rustel update` to update to the new CLI
$ cd ../rubyproj
$ russel run
Error: There is a breaking change between your russel version and your tessel CLI version
Error: This happens when you update that tessel CLI without updating russel as well
Error: Please run `russel update` to update to the new CLI

My only contributions to tessel have been to the CLI, which was simple enough that I could get started immediately and bang out in a few hours. Making the CLI more complicated and writing it in Rust makes this much harder for our audience to do.

The Big Problem I Forgot About

The largest problem I can see would be defining a language agnostic way of deploying codebases to the tessel, such that one underlying tessel command can interact with rust, ruby, or nodejs.

It makes sense to use language specific "plugins" to handle steps such as compilation. Ideally, that "plugin" would be built into the language specific CLI, such that tessel run in a nodejs codebase calls jessel to crosscompile binary packages during step 4 above.

In this way, tessel run would always work in any tessel compatible codebase, at the expense of adding a line to the package.json:

"languages" { "rust": {"root": "math", "cmd": "rustel", "standard": true }},

Wherein root is the source root, cmd is the name of the command that tessel-cli should communicate with, and standard is a boolean flag which says that the rustel API is a superset of the tessel API.

[johnnyman727]

@MrNice Creating a cross platform tessel executable would be a really elegant solution (especially if we were able to write it in Rust). I think the biggest hurdle right now (besides having the time to get such a big undertaking done) is the lack of USB support in Rust. It looks like @kevinmehall started such a project but it hasn't been updated in a while. We could potentially call out to a C/C++ lib for USB support until a Rust USB lib matures. Rust already has an ssh lib for LAN operations.

The largest problem I can see would be defining a language agnostic way of deploying codebases to the tessel, such that one underlying tessel command can interact with rust, ruby, or nodejs.

Just to make sure I understand, you're trying to tackle the problem of deploying a Tessel application that has dependencies in multiple languages?

[kevinmehall]

I deprecated my usb crate because another project surpassed it: https://github.com/dcuddeback/libusb-rs. It should have everything you need.

[johnnyman727]

Ah, thanks @kevinmehall my meager Googling couldn't find a Rust USB lib.

[MrNice]

@kevinmehall Awesome!

@johnnyman727
I was not clear!

I want to not write rust code when I add BASIC support to tessel.

I want to, at most, write BASIC code that interacts with the rust code. I'm willing to at least write out JSON which describes how to deploy and run BASIC code. A JSON DSL, like Grunt or npm or any number of other tools use.

I don't know enough about the current deployment strategy to say, "Oh, it's already language agnostic, we just need to expose four knobs; one for the interpreter, command arguments, source location, source destination"

[johnnyman727]

@MrNice I think we could to that by giving the standard library a callback to execute once we're ready to deploy code (ie a language plugin). Each language will be responsible for how it compiles, deploys, and runs, etc. The standard library can expose the same sort of tessel.exec function for running arbitrary shell commands on the Tessel.

[adkron]

Could we have a more modular interface, like that of the git command? Then the communities can write their own deployers, and other interaction pieces. t2 js deploy hellow_world.js t2 python deploy hello_world.py. This means that each command/module could be written in any language. This allows us some control, but anyone could easily have their own subcommands. Maintaining our own command line for each language is not a good use of the core team time.

[johnnyman727]

Could we have a more modular interface, like that of the git command?

@adkron I think you're suggestion is more or less in line with @MrNice's: make a cross-platform library that any other language can call into. Does that sound about right?

[adkron]

@johnnyman727, I wasn't thinking a library. With git there is a single command and you can create any executable and name it git-foo.<extension>. As long as it is in the path the git command will find it when you run git foo and execute it. We could do the same thing and anyone could write it. If we had a script in the path called t2-rust.rs and the command like could call it with t2 rust deploy.

[adkron]

Here is someone talking about building git extensions. =
http://blog.santosvelasco.com/2012/06/14/extending-git-add-a-custom-command/

[MrNice]

I'm specifically saying: Build a cross platform library, then wrap it in a
CLI. That provides FFI for languages which support it, and shell commands
for those which don't (and also human users).

The "modular interface" refers to how the Git command line api is
structured, as in git remote add and git add are completely different.

I'm against 'tessel python init' 'tessel python run', and for 'tessel init
--runtime python2.7' 'tessel run'.

I spent a lot of time talking about hypothetical commands like "pyssel"
which could be installed via pip. I imagine pyssel as being the
crossplatform library with a CLI interface written in python, that is
specifically tailored to managing deployment of python projects to the
tessel2. (def pyssel (partial tessel python2.7))

I currently hate the idea of a pyssel command.

On Wed, Jan 20, 2016 at 5:01 PM, Jon [email protected] wrote:

Could we have a more modular interface, like that of the git command?

@adkron https://github.com/adkron I think you're suggestion is more or
less in line with @MrNice https://github.com/MrNice's: make a
cross-platform library that any other language can call into. Does that
sound about right?

—
Reply to this email directly or view it on GitHub
#513 (comment).

[rwaldron]

I don't understand why npm would be used to distribute a Rust binary, when clearly Node.js is installed on said machine—otherwise npm wouldn't be installed.

[johnnyman727]

I'd like to re-open this conversation and accelerate development in other languages (driven primarily by my interest in Rust).

I agree with @MrNice that a portable CLI library that can be used by any other wrapper language would be ideal and it's certainly a great goal. For now, I think we should use the same CLI (installed via npm) for other languages since it's much more work to build all the functionality of the CLI in the desired language than it is to create another code path for deploys. I think the goals wrt language deployments should be:

1. Build deployment solution with existing JS CLI
2. Rewrite most of the underlying, language agnostic features into an executable (hopefully, Rust) and use that executable with the existing JS CLI
3. Create a new CLI for each additional language that is distributed with the common package manager for that language.

This process would allow us to start building out more language support immediately but provide us with a roadmap to the best use experience for each language.

[rwaldron]

1. Build deployment solution with existing JS CLI

Sure, a switch statement the switches on "something", and delegates to language specific handling. This is not the worst solution, but it's still additional complexity cost for an unquantified benefit.

1. Rewrite most of the underlying, language agnostic features into an executable (hopefully, Rust) and use that executable with the existing JS CLI
2. Create a new CLI for each additional language that is distributed with the common package manager for that language.

I'm going to be blunt here: this reeks of second system syndrome and will certainly serve to hurt the progress of the existing code bases (by disproportionately demanding time and effort). If I had known that a year's worth of work was going to be tossed out like this, I would've found something else to do with my time. This also means that a certain section of current maintainers are excluded, to some degree, from participating in future development.

[johnnyman727]

Sure, a switch statement the switches on "something", and delegates to language specific handling. This is not the worst solution, but it's still additional complexity cost for an unquantified benefit.

The benefit is certainly difficult to quantify but I've heard a lot of excitement from the Rust community about T2. If the community rallied around Tessel as the embedded platform for Rust, we could potentially grow the community much larger. This could bring in contributors who not only work on Rust-specific code but the CLI as a whole and that benefits everyone. Plus, I want to learn Rust and this is a good way for me to do it while continuing to engage with the project :)

I'm going to be blunt here: this reeks of second system syndrome and will certainly serve to hurt the progress of the existing code bases (by disproportionately demanding time and effort).

Yes, I agree about second system syndrome. But the hope is that it would be resolved by eventually splitting it back up into language-specific CLIs. It's only temporary to bootstrap the effort. In the meantime, I think we can pretty well sandbox the deploy stuff so that JS users don't have a worse experience because of it.

If I had known that a year's worth of work was going to be tossed out like this, I would've found something else to do with my time.

Rewriting a bunch of the CLI into an executable is a pretty significant undertaking that is probably more work than the original CLI was. I imagine it would take a long time to get this done such that even if we did, the CLI written as it is, is still a critical stepping stone. By building on top of what we have now, we can let the community experiment with Rust immediately which is super valuable, in my opinion. Rust hadn't even hit 1.0 when we started work on the CLI- Getting the chance to build a cross-platform executable simply would not have been possible if we didn't invest in a Node-based CLI first.

This also means that a certain section of current maintainers are excluded, to some degree, from participating in future development.

Yes, I agree. But I think if we have the core working really solidly, there will be lost of other things to work on in JS. But yes, I agree we could be excluding some people if we swapped in an executable.

[MrNice]

I'm going to recap my assumptions and work from there

1. Non-JS users don't want to use npm
2. Tessel should strive to be encompassing to many communities
3. Tessel as a project and community has very limited resources of money
   and time
4. A cross platform executable built in C or Rust would be straightforward
   to consume as an API from many languages and therefore would increase the
   t2's consumer base

My driving goal:
Minimize the amount of effort needed to widen the t2's audience.

I suggest this, without committing my own hours:

1. Create a rust library for t2 + rust community
2. Use this library within the CLI using node-ffi
3. If this hybrid approach works in reverse, consider moving common and
   core functionality from javascript to rust

This satisfies Jon's wish to play with rust more and lets us do a real
experiment with rust before committing anything. It also tests out the
strategy of mixing languages based on each one's strengths and community.

In response to rwaldron:

The second-system effect (also known as second-system syndrome) is the
tendency of small, elegant https://en.wikipedia.org/wiki/Elegant, and
successful systems to have elephantine, feature-laden monstrosities as
their successors due to inflated expectations.

The goal with having an executable is to create a small, elegant, and cross
platform (in both directions!) core which would then be useful by itself as
well as in conjunction with foreign programming languages. I hope this
would enable the kind of meteoric adoption that tessel deserves.

Every bit of work put into the CLI to date has been immensely valuable and
lays the groundwork for the future success of this project. The
evolutionary / migrational approach outlined above attempts to build upon
this previous success while opening new roads forward.

Thanks for listening!
-Nicholas

On Tue, Apr 26, 2016 at 1:27 PM, Jon [email protected] wrote:

Sure, a switch statement the switches on "something", and delegates to
language specific handling. This is not the worst solution, but it's still
additional complexity cost for an unquantified benefit.

The benefit is certainly difficult to quantify but I've heard a lot of
excitement from the Rust community about T2. If the community rallied
around Tessel as the embedded platform for Rust, we could potentially
grow the community much larger. This could bring in contributors who not
only work on Rust-specific code but the CLI as a whole and that benefits
everyone. Plus, I want to learn Rust and this is a good way for me to do it
while continuing to engage with the project :)

I'm going to be blunt here: this reeks of second system syndrome and will
certainly serve to hurt the progress of the existing code bases (by
disproportionately demanding time and effort).

Yes, I agree about second system syndrome. But the hope is that it would
be resolved by eventually splitting it back up into language-specific CLIs.
It's only temporary to bootstrap the effort. In the meantime, I think we
can pretty well sandbox the deploy stuff so that JS users don't have a
worse experience because of it.

If I had known that a year's worth of work was going to be tossed out like
this, I would've found something else to do with my time.

Rewriting a bunch of the CLI into an executable is a pretty significant
undertaking that is probably more work than the original CLI was. I imagine
it would take a long time to get this done such that even if we did, the
CLI written as it is, is still a critical stepping stone. By building on
top of what we have now, we can let the community experiment with Rust
immediately which is super valuable, in my opinion. Rust hadn't even hit
1.0 when we started work on the CLI- Getting the chance to build a
cross-platform executable simply would not have been possible if we didn't
invest in a Node-based CLI first.

This also means that a certain section of current maintainers are
excluded, to some degree, from participating in future development.

Yes, I agree. But I think if we have the core working really solidly,
there will be lost of other things to work on in JS. But yes, I agree we
could be excluding some people if we swapped in an executable.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#513 (comment)

[rwaldron]

Rewriting a bunch of the CLI into an executable is a pretty significant undertaking that is probably more work than the original CLI was.

I'm very aware of this fact. Have you considered that you will need to rewrite every cli dependency?

I imagine it would take a long time to get this done such that even if we did, the CLI written as it is, is still a critical stepping stone.

Considering my last point, I don't know if you realize how long this will actually take.

By building on top of what we have now,

I don't understand what you mean by this? Are you going to make another project that just delegates to t2-cli?

@MrNice

1. Non-JS users don't want to use npm
2. Tessel should strive to be encompassing to many communities
3. Tessel as a project and community has very limited resources of money
   and time
4. A cross platform executable built in C or Rust would be straightforward to consume as an API from many languages and therefore would increase the t2's consumer base

Given 3, who will spend the months, possibly years, building 4?

I suggest this, without committing my own hours

My kindest advice: don't ever lead with this when asking others to take on some massive amount of work.

In response to rwaldron:
The second-system effect (also known as second-system syndrome) is the tendency of small, elegant https://en.wikipedia.org/wiki/Elegant, and successful systems to have elephantine, feature-laden monstrosities as their successors due to inflated expectations.

Thanks for the passive aggressive attempt at a correction, accompanied by a link to the wikipedia definition of "elegant"—you've succeeded in alienating me from your cause.

[rwaldron]

Create a rust library for t2 + rust community

For the CLI or for programming on the T2? Are you conflating these two, completely different contexts?

Use this library within the CLI using node-ffi

Given my previous question, it's unclear to me what this library does.

[Frijol]

Interjecting in a moderator role - please remember to be kind to one another in this and every thread. It's easy to misinterpret intent on the internet, and even easier to make a place unfriendly out of frustration. Keep it constructive. Seek mutual understanding. Make this a good place to be :)

[Frijol]

My thoughts on the subject:

We're anticipating a solution to a problem that hasn't actually surfaced. I'd like to let user needs drive our design, particularly given resource constraints. Let's go for the quick fix (since I think you have a mostly-working solution there, right @johnnyman727?) and then build out more "proper" tooling if there is demand for it - and particularly if there is some champion within the community who wants to lead that.

[johnnyman727]

I'm very aware of this fact. Have you considered that you will need to rewrite every cli dependency?
Considering my last point, I don't know if you realize how long this will actually take.

I probably don't appreciate how long it would take. The good news is that the Rust ecosystem is building out very fast so there are a lot of libraries we could make use of. We would not have to rewrite all dependencies but likely a few of them. Also, the only way I can imagine we would bootstrap a community of Rust devs together willing to invest in building an executable is if we release something immediately useful rather than an abstract goal.

I don't understand what you mean by this? Are you going to make another project that just delegates to t2-cli?

I mean building out Rust-specific deployment logic (and project initialization) that is part of the Node project. So Rust devs would need to have Node/npm installed but could use t2 init --lang rust and t2 run RUST_PROJECT from this same code base.

There are folks asking me if they can use Rust on Tessel right now so it would be great if we could start merging related PRs like #292 and #511 (after cleaning it up). @kevinmehall mentioned that he also wrote the equivalent of tessel-export.js in Rust so we could start building out module support in Rust as well.

[rwaldron]

#511 is a non-starter in it's current form

[johnnyman727]

@rwaldron 100% agree. I was just making a proof of concept. I have a lot to clean up.

[tikurahul]

This is based on a conversation i had with @johnnyman727.
Here is an alternative proposal, open for comments and feedback.

#716

[johnnyman727]

Thanks for the additional proposal @tikurahul, I'll follow up there.

I'd like to reach a consensus on whether or not we should start building additional language support into the JS API as a first step. I would start by adding Rust and I propose we use it as a trial of sorts. If it doesn't go well and/or puts the JS experience in jeopardy, we can revisit the issue.

@rwaldron I don't think we have explicit agreement from you. Would you be okay if I got started on my proposal?

[johnnyman727]

We've laid down the framework for adding more languages since the recent addition of Rust. I'm going to archive this conversation for now but thanks to everyone who contributed!