use a script instead of submodules for grammars

[the-mikedavis]

closes #1549

there's still a bunch to do before this is ready:

• grammars.cmd for windows
• replace any existing documentation for fetching and adding new grammars
• fall back to a regular clone when the remote doesn't support cloning at exact revisions (I believe most/all popular hosted git remotes support fetch-at-revision these days)
• a check/status command that checks if you're currently synced
• a fetch command that downloads a new grammar repository (maybe this isn't necessary, you can always just git clone)
• a save command that updates revisions.txt with your current local grammar set (useful for adding new grammars) (eh, I think it makes sense to download new grammars using the revisions.txt file declaratively)

But the results are already pretty nice. By taking advantage of shallow fetches at exact revisions, we get down to ~30s to fetch all grammars and ~1.5s to re-run sync (idempotently). For perspective:

• a full recursive clone of helix currently takes ~660s (11 minutes)
• git submodules update --init takes ~250s (a little over 4 minutes)
• git submodule update --init --depth 1 takes ~80s

Ideally I'd like this script to allow one to hack on a grammar as they're integrating it, so it's meant to not care about other remotes and to fail when checking out a revision that would overwrite local modifications.

[Omnikar]

You'll also need to change the documentation that mentions the submodule system, such as README.md and the adding languages docs.

[archseer]

For perspective, a regular recursive clone of helix currently takes me ~660s (11 minutes) on a very good connection and the first run of git submodules update --init takes ~250s (a little over 4 minutes).

Note: You can use git submodule update --init --depth 1 to do a shallow clone there too (or git clone --shallow-submodules -j8 https://github.com/helix-editor/helix as per README). I do agree that the new scripts will be more flexible though

[archseer]

Bonus points:

• convert the helix-syntax/build.rs into a cargo xtask action
• move the helix-syntax/src/lib.rs into helix-core/src/syntax.rs and delete the helix-syntax crate

This would make it easier to build specific grammars and avoid issues with helix-syntax cache where one needs to cargo clean -p helix-syntax right now to trigger a grammar build.

We could also turn the build.rs into a shell script, that would allow users to compile grammars without needing a Rust installation (all we're really doing is invoking the C compiler)

[dead10ck]

There seem to be a few people that are in favor of this PR, so could someone help me understand? It seems like this approach is also:

• Requiring all grammars be built
• Pinning down all the grammars to a specific commit with a bespoke config format
• Implementing a bespoke CLI for updating grammars, except with no built in notification when your local repos are out of sync with the config. The user has to explicitly opt into even checking if their local subrepos are in sync.
• Needs at least two implementations of the same said CLI

Is this actually faster than git submodule update --init --depth 1?

In general, I like the idea of moving away from submodules because they can be a huge pain, but if the purpose is to improve flexibility, then why pin versions? It seems like we're just reimplementing git submodules here in bash. I understand the reasons in general for pinning versions, as it provides a reproducible set of deps that were tested against, but they naturally come with maintenance overhead. And will this really improve that?

It seems to me like between flexibility and reproducibility, this is like the worst of both worlds.

[archseer]

Rather than improve performance, the goal is to have the set of grammars easier to configure. Using submodules isn't scaling for us anymore: CI tests now frequently fail on fetching these submodules due to network issues because we have so many.

With a script the user can:

• Build grammars more selectively
• Add new grammars locally without having to upstream them
• Build the grammars without requiring a Rust compiler / having to rebuild Helix

• The user has to explicitly opt into even checking if their local subrepos are in sync.

I think most users already ignore the git submodule warnings. We frequently get questions on why X doesn't work because they forgot to pull the grammars, and PRs often include accidental submodule reverts.

but if the purpose is to improve flexibility, then why pin versions?

We're pinning versions because grammars often have breaking changes that break existing highlights. If they add new grammar items we also need to add them to our queries.

[archseer]

• Needs at least two implementations of the same said CLI

We could make this a part of cargo xtask but that would bring back the compiler dependency. We can keep the scripts small though, all we really need is fetching the dependencies then running a C compiler. (https://github.com/tree-sitter/tree-sitter/tree/master/script also has Windows versions of various scripts)

[dead10ck]

Rather than improve performance, the goal is to have the set of grammars easier to configure. Using submodules isn't scaling for us anymore: CI tests now frequently fail on fetching these submodules due to network issues because we have so many.

Does this approach help? Don't we have to clone just as many repos? Or is it that right now, we must clone everything, but this way we don't?

With a script the user can:

• Build grammars more selectively

How will this work? Will the user edit the revisions.txt that's checked in and make dirty builds? Or are we going to make some kind of mechanism to override this file out of tree?

• Add new grammars locally without having to upstream them

Can't they do this now? If I'm not mistaken, users are currently able to do this by just building the .so, and putting that and the queries in the runtime folder, right? Would this change that?

If you mean that the user workflow for adding a new language becomes just adding a line to revisions.txt, then that's not really true either. I don't disagree with this decision, but helix implements its queries differently than other popular tree-sitter editors/plugins, and so a consequence of this is that the user can't simply add another tree-sitter grammar repo and have it work; they also need to learn tree-sitter from the ground up and write all the queries in a way that's compatible just with helix. The user could do this, of course, but I don't see how the way helix manages its dependencies has any bearing on how easy it is to add other languages to their own runtime.

This might be going off topic a bit, but this is also why I think having languages be plugins would be the best long term solution. The tree-sitter dependency management, build process, and helix tree-sitter queries can live completely out of tree. This is how vim, VSCode, and probably others do it.

• Build the grammars without requiring a Rust compiler / having to rebuild Helix

Does how we fetch and manage the grammars have a bearing on how we build them? To be honest, I'm not really sure why the Rust toolchain is involved in them at all. Is that a vestige of when the grammars were statically linked into the hx binary?

I think most users already ignore the git submodule warnings. We frequently get questions on why X doesn't work because they forgot to pull the grammars, and PRs often include accidental submodule reverts.

Yeah I understand users will be users, but this means that even for devs who do pay attention, it now becomes much more easy to not realize that your grammars are out of date. There is nothing to tell you so after doing a git pull of the latest master, for example. Just all of a sudden the highlighting and tree-sitter navigation is broken and you have no idea it's because the grammar and queries were updated. Now a dev has to always think about updating their grammars, whereas now, git tells you when you have to.

but if the purpose is to improve flexibility, then why pin versions?

We're pinning versions because grammars often have breaking changes that break existing highlights. If they add new grammar items we also need to add them to our queries.

That certainly makes sense, but then what kind of flexibility are we after?

[dead10ck]

Don't get me wrong here, I hate git submodules with a burning passion, and I would love to get rid of them. I would not be loathe to work with bash instead of submodules. But as much as I hate them, they do provide some measure of benefit, and just based on the albeit limited experience I have with the repo so far, this particular way of replacing them seems like an inequitable trade-off, and it seems like there are other desires at play that have varying relevance to dependency management.

[archseer]

Does this approach help? Don't we have to clone just as many repos? Or is it that right now, we must clone everything, but this way we don't?

You can modify the list of repositories before running the script, or use a custom list.

We can also avoid git altogether by fetching https://github.com/<repo>/archive/<commit_sha>.zip for grammars hosted on github. That's what NixOS does for fetchFromGithub to speed things up. It would make #1536 even easier to do.

Can't they do this now? If I'm not mistaken, users are currently able to do this by just building the .so, and putting that and the queries in the runtime folder, right? Would this change that?

Right, but nobody actually does that since you need to figure out the correct compiler incantation to do so. Having a script for it would make this a viable option.

This is how vim, VSCode, and probably others do it.

Neovim does, but it requires you to have a properly set up compiler so it can compile the grammars, there's some downsides there (as opposed to fetching them from a release). VSCode doesn't actually use tree-sitter.

What we really want in the long term is tree-sitter/tree-sitter#930 stretch goal where grammars are .wasm files. Then all the .wasm grammars can be downloaded and checked against a checksum.

There is nothing to tell you so after doing a git pull of the latest master, for example.

We could still hook into this in build.rs or add a Makefile or some sort of build wrapper. The current process is not without problems: If you pull in new grammars, they won't get rebuilt since helix-syntax/src doesn't get changed. You need to cargo clean -p helix-syntax to trigger a grammar build. That might be fixable with https://doc.rust-lang.org/cargo/reference/build-scripts.html#rerun-if-changed though

[dead10ck]

You can modify the list of repositories before running the script, or use a custom list.

Hm, if we're editing the file that's checked in, this will cause probably more accidental commits than submodules. Maybe the script could check for a ~/.config/helix/revisions.txt instead first?

This is how vim, VSCode, and probably others do it.

Neovim does, but it requires you to have a properly set up compiler so it can compile the grammars, there's some downsides there (as opposed to fetching them from a release). VSCode doesn't actually use tree-sitter.

Is this different than the reality now? If your grammars don't all have prebuilt binaries (do any of them?), then your machine must be capable of compiling them.

What we really want in the long term is tree-sitter/tree-sitter#930 stretch goal where grammars are .wasm files. Then all the .wasm grammars can be downloaded and checked against a checksum.

This would be nice, and very similar in spirit to the plugin system imagined for helix, but even when/if that lands, are we going to be able to expect all tree-sitter grammars from the community that we depend on to compile and distribute prebuilt wasm files? There are dozens. There isn't going to be a tree-sitter distribution platform. The more likely scenario is that many will still have to be compiled from source, right? I imagine the same will be true of the plugin system.

[the-mikedavis]

The script ends up being about twice as fast as submodules, even with the --depth=1 (I think submodules are doing a shallow clone and then a shallow fetch, so twice the work).

I think you're right @dead10ck that this change doesn't really significantly improve the grammars situation, plus it's not great that it introduces a bespoke CLI and there's no git helper to tell you you're out of date. But this is meant to be a stepping stone for future improvements, namely the one people are most interested in afaict: being able to have a per-user subset of grammars. For that, I think having the script look for ~/.config/helix/revisions.txt is a reasonable next step.

... it now becomes much more easy to not realize that your grammars are out of date... Just all of a sudden the highlighting and tree-sitter navigation is broken and you have no idea it's because the grammar and queries were updated

Thanks for bringing this up, this will be a problem. There needs to be a way to specify a version of the queries used with a particular version of the grammar or else query analysis will fail when the grammar makes a breaking change to the set of nodes. The most minor "fix" to this would be to improve the error message that happens when query analysis fails and have it say that you need to ./scripts/grammars sync.

I think a more maintainable solution would be to do what nvim-treesitter does and have the queries in a repository with the revisions, so when you install/update a language it pulls in a version of the grammar and a version of the queries that work together. This approach might also work well in a WASM-based future: we could have a github actions automation that compiles each language's grammar into the necessary WASM and then the user simply downloads the WASM and the queries (by hand, with a script, or maybe a built-in command).

[pickfire]

Won't this just introduce an additional step to manually edit this file and put in the submodule? I think this may be harder for new contributors to use rather than just git submodule add.

[the-mikedavis]

Adding it here removes any submodule steps, so no need to git submodule add and set it as shallow. It's not really an improvement though: you end up still needing to ./scripts/grammars sync once you add it to the file, so it ends up being the same number of steps to add a new grammar.

One area where it helps a lot is when a grammar changes remotes (like #830). The script handles that just with an update to revisions.txt but with submodules it's a total nightmare, you have to go edit multiple files under .git/ to fix the submodule. This is usually only relevant if you need to fork a grammar and make some changes when adding it, I don't know how common that is for others 🙃

The main thing the script improves is being able to have a per-user subset of grammars (albeit it's not implemented in this branch), which I can't think of a way to do well with submodules.

[the-mikedavis]

I ported the helix-syntax/build.rs builder to an xtask instead of adding a build command for the scripts: some grammars have subdirectories to build which I think need a more expressive language to describe than what revisions.txt can offer.

Still to do: update the flake to read revisions.txt and fetch the grammars.

[archseer]

need a more expressive language to describe than what revisions.txt can offer.

Here's how the code looks inside the tree-sitter CLI if it helps: https://github.com/tree-sitter/tree-sitter/blob/15860392c5952e3ee6f93524fc75de5ef14493c0/cli/loader/src/lib.rs#L301-L444

It seems to assume a certain structure, but I'm guessing that doesn't work for everything.

[the-mikedavis]

I think tree-sitter ends up consulting a grammar repository's package.json to detect repositories that have grammars in subdirectories (see here), and I think in helix that package.json info ends up rewritten in languages.toml.

Maybe it's better to have the revisions and remotes that are currently in revisions.txt be in languages.toml instead, so all of your language configuration goes in one place? Writing a toml parser in bash sounds like a fun project but writing a toml parser in windows batch script would probably drive me crazy 😅: maybe the fetching part of this script should be an xtask as well? It seems a bit wasteful to use xtask for basically shelling out to git, but I think it'd be easier to maintain cross-platform. Plus it'd be reusable when/if adding this functionality to helix runtime or making a new CLI to do it.

What do you think?