Researching a cargo-script for rustup

[epage]

Background: this originated in the Learnability of Rust blog post.

Vision

Rust has harmonized several apparent dichotomies, including

• Memory safety without garbage collection
• Abstraction without overhead
• Concurrency without data races
• Stability without stagnation

Following in those footsteps, this project seeks to offer

• Exploratory programming while allowing fine-grained control

Roles of exploratory programming

• Learning
• Verify assumptions on an API's behavior
• Prototypes
• One-off utilities
• Collaborating, like bug reports

Impediments

Thought exercises:

• For the last script you wrote in bash, python, etc, why didn't you use Rust?

Friction within the existing process that gets in the way includes:

1. Cargo requires a dedicated directory with a Cargo.toml and src/main.rs with common dependencies (arg parser, json support, etc) added to Cargo.toml, commonly with a version-requirement
2. Rust requires precision with types, lifetimes, scopes, etc
3. Using cargo run --manifest-path <abs-path> -- <args> to try the idea (need for --manifest-path depends on the experiment)
4. Compile times from every one-off project re-compiling the same dependencies
5. Taking all of these files and sharing them, like in a bug report

We do have https://play.rust-lang.org/ which can help. Compared to cargo new,

• Pros
  • No cargo setup; has default set of dependencies defined
  • One-click running
  • Couple clicks to get a shareable link
• Cons
  • Limited in available dependencies
  • Reproducing outside of playground is unclear with implicit dependencies
  • Limited in checking assumptions due to sandboxing (e.g. verifying behavior of git2)
  • Can't be used as a local utility

Note: we are going to narrow our scope to the project-management side of this. See Learnability of Rust blog post and xshell for ideas on the language side.

Prior Art

Bash allows (#4)

• bash to get an interactive way of entering code
• bash file will run the code in file, searching in PATH if it isn't available locally
• ./file with #!/usr/bin/env bash to make standalone executables
• bash -c <expr> to try out an idea right now
• Common configuration with rc files, --rcfile <path>

Python allows (#12)

• python to get an interactive way of entering code
• python -i ... to make other ways or running interactive
• python <file> will run the file
• ./file with #!/usr/bin/env python to make standalone executables
• python -c <expr> to try out an idea right now
• Can run any file in a project (they can have their own "main") to do whitebox exploratory programming and not just blackblox

gorun attempts to bring that experience to a compiled language, go in this case (#13)

• gorun <file> to build and run a file
• Implicit garbage collection for build cache
• Project metadata is specified in HEREDOCs in comments

scriptisto (#10)

• Supports any compiled language
• Comment-directives give build commands

nix-script

• Nix version of scriptisto, letting you use any Nix dependency

Existing Solutions

• cargo-script (Investigate cargo-script #1)
  • Single-file (.crs extension) rust code
    • Partial manifests in a cargo doc comment code fence or dependencies in a comment directive
    • run-cargo-script for she-bangs and setting up file associations on Windows
  • Performance: Shares a CARGO_TARGET_DIR, reusing dependency builds
  • --expr <expr> for expressions as args (wraps in a block and prints blocks value as {:?} )
    • --dep flags since directives don't work as easily
  • --loop <expr> for a closure to run on each line
  • --test, etc flags to make up for cargo not understanding thesefiles
  • --force to rebuildand--clear-cache`
  • Communicates through scrpts through some env variables
• cargo-scripter (Investigate cargo-scripter #2)
  • See above with 8 more commits
• cargo-eval (Investigate cargo-eval #7)
  • See above with a couple more commits
• rust-script (Investigate rust-script #4)
  • See above
  • Changed extension to .ers
  • Single binary without subcommands in primary case for ease of running
  • Inferred-main support, including async main (different implementation than rustdoc)
  • --toolchain-version flag
• cargo-play (Investigate cargo-play #9)
  • Allows multiple-file scripts, first specified is the main
  • Dependency syntax //# serde_json = "*"
  • Otherwise, seems like it has a subset of cargo-scripts functionality
• cargo-wop (Investigate cargo-wop #3)
  • cargo wop is to single-file rust scripts as cargo is to multi-file rust projects
  • Dependency syntax is a doc comment code fence
• runner (Investigate runner #14)
  • Global Cargo.toml with dependencies added via runner --add <dep> and various commands / args to interact with the shared crate
  • Global, editable prelude / template
  • -e <expr> support
  • -i <expr> support for consuming and printing iterator values
  • -n <expr> runs per line
• evcxr
  • Umbrella project which includes a REPL and Jupyter kernel
  • Requires opting in to not ending on panics
  • Expressions starting with : are repl commands
  • Limitations on using references
• irust
  • Rust repl
  • Expressions starting with : are repl commands
  • Global, user-editable prelude crate
• papyrust
  • Not single file; just gives fast caching for a cargo package

Proposal

Add to rustup a new command that learns the lessons from the past to reduce the barrier to exploratory programming

Phase 1

• Easy to use as she-bang or file associations
  • She-bang lines are legal within these files
• Full control over manifest
• Shared binary cache
  • --force and --clear-cache flags
• Shared version selections (Cargo.lock) for reducing cache misses
  • --update flag to update shared versions
• Build profile defaults to favoring total execution time after change (build time + runtime) for fast experimentation
• Run a script file or a script expression
  • Implicit main for both (no specialized behavior for script expressions)
  • No -loop, --iterator, etc support (switch to a script file)
• Pass scripts some common environment variables
• Either cargo new or the script runner have a flag to explode a script into an idiomatic rust project for scaling up

(rust-script seems like the closest base)

Phase 2

• Support passing these scripts to cargo commands as --manifest-path (test, build, etc)
  • Consider a short flag for --manifest-path like -m or -M to make this less cumbersome

Future Exploration

• Ephemeral compilation
  • Don't create a crate out of the file / -c <expr> behind the scenes
  • Will still need a Cargo.lock, intermediate, and final build artifacts
• Allowing these to be workspace members (e.g. for use as xtasks)
• REPL

Open Questions

• How do we reduce boilerplate in scripts while allowing reproducibility across systems (How do we balance reducing boilerplate in scripts while allowing reproducibility across systems #46)
• Command name (Settle on a command name #41)
• What extension to use? (Settle on an extension #42)
• Is there a way we can support whitebox exploratory programming like Python where you can run files within a project?
• Can we help scale down a crate to a single-file reproduction case?
• What, if any, GC approach should we take?
  • Are there parts of cargo sweep that'd be useful?

Next Steps

1. Reach out to cargo-script-like tool authors for feedback
2. Create a minimal-viable solution for collecting wider feedback
3. Post a Pre-RFC to internals

FAQ

Why a new command instead of one of the existing implementations?

• We are just wanting to emphasize that the decision making should re-evaluate decisions for today to see if they still make sense. If an existing tool wants to morph into this, thats great!

Why ship this in rustup?

• The ecosystem is fragmented with different features, fixes, and syntaxes. Making one official will make it easier for collaboration.
• Third-party commands require "being in the know" to use
• There can be user-fatigue for the 10 or so commands that are "best practices" to install
• For example for the above, when I got a bug report with a cargo-play snippet, I just manually created a cargo project from it because I didn't want to install a command just for this one report
• This is a step towards native integration into users workflows
  • e.g. each existing implementation provides a way for running tests in scripts that isn't cargo test
  • e.g. more likely to get IDE integration

[epage]

rdmd is an interesting prior art for this. It has --eval=<code> and --loop=<code> arguments, like rust-script