FR: make jj commands more composable

[arxanas]

Is your feature request related to a problem? Please describe.
By "composable", I mean that it should be possible to efficiently use the output of one command as the input to another. Oftentimes, jj features like

• operating uniformly on commits and the working copy
• commits having stable change IDs across rebases

mean that many workflows can be accomplished without explicitly using the output of previous commands, instead just relying on stable change IDs or similar, but it's not enough for all use-cases.

For example, if I would like to run jj split and then immediately rebase the newly-created commits (or maybe just those which touch a certain file path, etc.), there's not an easy way to address the newly-created commits, so I have to manually construct and issue two different commands to do so.

Describe the solution you'd like
🤷

Describe alternatives you've considered
Two main approaches come to mind:

• jj commands should produce machine-readable output, detailing the effects/results of the command.
  • Relies on the user's shell or similar for composition.
  • Can interoperate with other tools more easily.
  • Have to pick serialization formats and design schemas.
• The existing revset DSL could be expanded to include side-effecting operations (creating, rebasing, splitting commits, etc.).
  • Complex workflows can be implemented without manually managing many processes, parsing output, etc.
  • Rich internal code and data types can be accessed. For example, you can call methods on revsets without having to expose every such method via an external API.
  • Declarative rather than imperative in principle.
    • This probably isn't inherently advantageous, but there could be efficiency gains e.g. with lazy evaluation that might not be feasible when leaving the language.
    • Unclear how interactive operations would be executed.
  • The revset language is perhaps clunky to write long scripts in.
  • It's probably weird for users to think about at first, in comparison with commands just dumping their output.

There's also some alternative abstraction boundaries (via a stable compile-time or runtime API):

• FR: Add jj api command to establish structured interprocess communication for seamless IDE integration #3219
• FR: JJ extension suppport #3575

Additional context

• git-rebase --interactive: tool built into Git to conduct certain operations (rebases, cherry-picks, merges), although such rebase plans are usually written by hand: https://git-scm.com/docs/git-rebase#_interactive_mode
• git-assembler: a tool for defining a kind of rebase plan once and executing it multiple times in the future: https://www.thregr.org/wavexx/software/git-assembler/

[ChrisPenner]

I'll second the desire for machine-readable output; I often like to build small tools in fzf for things like "select one of the files that have conflicts" or "select a branch to new onto"; but jj status's current output isn't very easy to chop into the right format for things like finding conflicts or w/e.

I'd love either:

• template functionality for most commands
• something like the git --porcelain flag which would tell jj to only output the info I asked for, and in a structured format so I can hack-and-slash it with other unixy tools.

😄

[emilazy]

Some relevant inspiration here is libxo, which is used by the core command‐line utilities that ship as part of FreeBSD. It provides an API that allows output formatting to be annotated such that all those utilities can output the same information in a structured JSON or XML format.

Similarly, it would be nice if jj commands could return structured data and pretty output in one go, rather than handling them as separate code paths; this would ensure that all the same information is available in both, and cut down on code duplication. Of course, they’re dealing with the case of decades‐old Unix utilities that really need to be able to output in whatever bespoke format people have become used to; It’s possible that Jujutsu’s output is uniform enough, or could be made uniform enough, that we could do something simpler and less flexible and have a more limited standard set of structured command output structures that can be mapped to user‐friendly unstructured output.

In general, I would be happy to see jj grow a --json argument on all commands, even if some kind of embedded scripting DSL is also added; you can’t anticipate all the possible contexts someone might want to do scripting in. However, I do think that we might want to consider how high‐value this is compared to factoring things out so that there’s a high‐level Rust API you can use to accomplish the same things the commands do? We could bind it to Python if we want a more lightweight and scriptable interface. Complex shell scripts are painful and JSON is a pretty lowest‐common‐denominator type to stuff everything into; it’s always nicer to use a high‐level API than string commands together once you pass a pretty low complexity threshold.

[PhilipMetzger]

Similarly, it would be nice if jj commands could return structured data and pretty output in one go, rather than handling them as separate code paths; this would ensure that all the same information is available in both, and cut down on code duplication. Of course, they’re dealing with the case of decades‐old Unix utilities that really need to be able to output in whatever bespoke format people have become used to; It’s possible that Jujutsu’s output is uniform enough, or could be made uniform enough, that we could do something simpler and less flexible and have a more limited standard set of structured command output structures that can be mapped to user‐friendly unstructured output.

In general, I would be happy to see jj grow a --json argument on all commands, even if some kind of embedded scripting DSL is also added; you can’t anticipate all the possible contexts someone might want to do scripting in. However, I do think that we might want to consider how high‐value this is compared to factoring things out so that there’s a high‐level Rust API you can use to accomplish the same things the commands do? We could bind it to Python if we want a more lightweight and scriptable interface. Complex shell scripts are painful and JSON is a pretty lowest‐common‐denominator type to stuff everything into; it’s always nicer to use a high‐level API than string commands together once you pass a pretty low complexity threshold.

I think this part of your comment belongs in #3262. As @khionu has requested something similar.

[emilazy]

I guess it would make sense to integrate the structured output into the templating language :) But I didn’t necessarily mean to imply that with my comment. Thanks for the pointer to the discussion in #3262.

[jennings]

@ilyagr mentioned "scripting language" in Discord recently. It's the first time I've heard it mentioned so I don't know if there's been much discussion about that, but I think that sounds like a wonderful way to enable both interactive and non-interactive use of jj without needing to go the jj api route and without needing to make every jj command work for both use cases.

Using Lua as the hypothetical language, I'm imagining something vaguely like this:

$ echo duplicate-and-rebase.lua
tx = jj.start_transaction()
dupes = tx.duplicate({ revset = ARGS["r"] })
rebased = tx.rebase({ revisions = dupes, destination = ARGS["d"] })
tx.finish("Duplicated and rebased revisions")
print(json(rebased))

$ jj script <duplicate-and-rebase.lua -- -d main -r main..my-branch

In other words, some way to do useful things like:

• Executing jj commands
• Optionally grouping those commands into a single transaction
• Pass arguments to the script
• Output results in whatever structured form someone might want: JSON, CSV, text, etc.

This idea appeals to me because it means normal jj commands don't need to worry so much about non-interactive use. It might also partially solve #3673.