Bikeshedding how we use/offer templates

[khionu]

Templating (like scripting) is quite a deep topic, and how it's relevant to this project has had a fair amount of conversation to date, and will likely continue to need more for a while to go. This issue is meant to serve as a place to aggregate and hash out desires, concerns, and anything else that might inform the total story of templating in Jujutsu.

A gentle reminder that push back does not mean that desires/needs are invalid, we're all here to try and make Jujutsu as good of a tool as it can be.

It would be appreciated if collaborators could help keep the below sections updated

Stance summaries

• @khionu
  • would like strong templates, but looking to figure out a minimal set of functionality for 80% of possible cases

Comments of note

[khionu]

My thoughts, in no particular order, and assuming no particular prior knowledge of the reader:

• The flexibility that templates can provide is often very high. Sometimes that flexibility is needed, sometimes it isn't. Too much can be painful, to both maintainers of jj and the templates, as can too little can be stifling.
• More popular Helm charts can be virtually untouchable by anyone who doesn't work on the charts themselves regularly
• It might be possible to create loose measures of template potential to estimate how much flexibility we're comfortable with. May be worth a working group of some kind?
• I'm thinking that some good bounds may be:
  • No user functions
  • No variables
  • No arithmetic
  • No ability to access or measure time beyond what's already in eg commits
  • Can overwrite built-in templates
  • Values for a given template can be T | List<T>, but builtin functions can return nested maps/objects as well. T being one concrete type for the context (eg string or rev)
  • User expressions can return new values, but can't mutate existing values
  • Focus on user expressions being filters, joins, queries. Additive or selective operations.
  • Filesystem access restricted to repo directory, -x -w, no following links
    • If there are some files we want them to be able to save to, maybe a persistent kv that they access with functions, then any modules can work form the same store.

[PhilipMetzger]

Summarizing my thougths on the templating system:

• It has been clear for a while that our users want to use it, wherever possible
• A minimal subset of the language should be available for configuration
• We definitely need to avoid the language in a language problem
• It shouldn't be a bigger burden than the current log templating language

[dairyisscary]

From #3219:

I'd love to see some common encoding functions built into the templating language to facilitate integrations with editors, shell prompts, and really anything that would prefer to parse structured data, rather than human-readable plain text.

Something like:

jj log --template 'json({ id: commit_id, author: author, desc: description })'
jj log --template 'csv([ commit_id, author, description ])'
jj log --template 'toml(commit_id)'

This also means that Jujutsu doesn't need to commit to a JSON output interface beforehand and each integration gets to pick its preferred output with exactly the data it wants. I'm not sure if Jujustu currently has support for list literals or "object/map" literals, so this proposal might mean an expansion in expressiveness to the language.

[khionu]

Composing object literals shouldn't be a problem. In the case of structuring output, what I suggested would mean string | List<string>, and a json function could take a nested map of strings to format.

[yuja]

Implementation-wise, jj's templater is a typed language right now (and has method name resolution based on type), so introducing first-class user object literal or heterogeneous list will add complexity. I'm not against adding some sort of dynamic types, but I generally like the strictness of jj's template error handling.

JSON specially has another problem that template outputs can't be simply streamed. Mercurial has a nasty hack to omit the last "," iirc.

[khionu]

I didn't mean to suggest heterogeneous lists, I'm actually not a fan of those in general.

I mispoke in my last, "s/return/take/ a nested map".

Mercurial has a nasty hack to omit the last "," iirc.

We shouldn't need to worry about that, we can just invoke serde_json and let it handle everything.

[yuja]

Mercurial has a nasty hack to omit the last "," iirc.

We shouldn't need to worry about that, we can just invoke serde_json and let it handle everything.

It works only for one serializable object, right? jj log -T.. emits streams of template outputs. Each output can be serialized by serde_json, but they can't be simply concatenated (as [u8]) in order to form one big JSON array or object.

If we don't need to support arbitrary template queries, using static serde derive would be the easiest option to generate machine-readable output.

[necauqua]

but they can't be simply concatenated

Why?

Json streams are a format

λ echo '{}{}{}' | jq
{}
{}
{}

λ echo '{}{}{}' | jq --slurp
[
  {},
  {},
  {}
]

Yes I was thinking it would be super-easy to just dump template objects through serde_json and call it a day 🤷
I think any savings from selectively querying things would probably be negligible

[yuja]

Yeah, it's easy if the caller expects json streams. I just mean a pure json array can't be constructed that way (and Mercurial has a workaround for that.)

[matts1]

FWIW, I'm not a huge fan of adding additional functionality to the templating language for this kind of thing. IMO, templates should be used by humans, and not by computers, so I don't like the idea of attempting to support json in them.

If you're using a template for use by a computer, IMO jj api would be a better choice, and I don't think we should be encouraging users to use templates for it.

[PhilipMetzger]

FWIW, I'm not a huge fan of adding additional functionality to the templating language for this kind of thing. IMO, templates should be used by humans, and not by computers, so I don't like the idea of attempting to support json in them.

If you're using a template for use by a computer, IMO jj api would be a better choice, and I don't think we should be encouraging users to use templates for it.

Then how do you solve the bootstrapping problem, while nothing of sorts exists yet? The template language is available and we've had repeated requests to support some kind of dynamic configuration with it. And 3rd parties want to integrate w/o using the library directly.

I agree that in the future jj api will solve it correctly, providing this functionality seems to be a lesser pain until it exists.

[matts1]

I agree that in the future jj api will solve it correctly, providing this functionality seems to be a lesser pain until it exists.

I agree that this will make the pain lesser until it exists, but I have a few concerns about doing so

• It seems like additional technical debt to maintain
• Once we release this feature, since it's used by tooling rather than by humans, we can't just un-release it once jj api is ready, since it'll break things.

And 3rd parties want to integrate w/o using the library directly.

This is a slightly awkward currently, but it is possible, and not particularly difficult. I've already done so myself (I wrote a python script that allows you to run jj gerrit upload and runs repo hooks by interfacing with the jj CLI). I use the following template:

TRASH = "GOBBLEDIGOOSH anotis\n"
TEMPLATE = f"""concat(
    conflict,
    "\n",
    change_id,
    "\n",
    commit_id,
    "\n",
    parents.map(|c| c.change_id()).join(" "),
    "\n",
    description,
    "{TRASH}"
)"""

I then run:

for change in run(["jj", "log", "-T", TEMPLATE, "-r", revset]).split(TRASH):
    conflict, change_id, commit_id, parents, desc = entry.split("\n", 4)

IMO, the actual implementation of jj api isn't hard, it's just a bunch of design questions that need to be answered. I think that if this needs to be hurried, we should just accelerate the implementation of jj api. I think that a ListChanges RPC would be a great candidate for the first thing to migrate over to jj api.