Add more parsers

[Freed-Wu]

• ada
• agda
• angular
• apex
• arduino
• asm
• astro
• authzed
• awk
• bash
• bass
• beancount
• bibtex
• bicep
• bitbake
• blueprint
• c
• c_sharp
• cairo
• capnp
• chatito
• clojure
• cmake
• comment
• commonlisp
• cooklang
• corn
• cpon
• cpp
• css
• csv
• cuda
• cue
• d
• dart
• devicetree
• dhall
• diff
• disassembly
• dockerfile
• dot
• doxygen
• dtd
• ebnf
• eds
• eex
• elixir
• elm
• elsa
• elvish
• embedded_template
• erlang
• facility
• faust
• fennel
• fidl
• firrtl
• fish
• foam
• forth
• fortran
• fsh
• func
• fusion
• gdscript
• gdshader
• git_config
• git_rebase
• gitattributes
• gitcommit
• gitignore
• gleam
• glimmer
• glsl
• gn
• gnuplot
• go
• godot_resource
• gomod
• gosum
• gowork
• gpg
• graphql
• groovy
• gstlaunch
• hack
• hare
• haskell
• haskell_persistent
• hcl
• heex
• hjson
• hlsl
• hlsplaylist
• hocon
• hoon
• html
• htmldjango
• http
• hurl
• hyprlang
• ini
• ispc
• janet_simple
• java
• javascript
• jq
• jsdoc
• json
• json5
• jsonc
• jsonnet
• julia
• kconfig
• kdl
• kotlin
• kusto
• lalrpop
• latex
• ledger
• leo
• linkerscript
• liquidsoap
• llvm
• lua
• luadoc
• luap
• luau
• m68k
• make
• markdown
• markdown_inline
• matlab
• menhir
• mermaid
• meson
• mlir
• muttrc
• nasm
• nickel
• nim
• nim_format_string
• ninja
• nix
• norg
• nqc
• objc
• objdump
• ocaml
• ocaml_interface
• ocamllex
• odin
• org
• pascal
• passwd
• pem
• perl
• php
• php_only
• phpdoc
• pioasm
• po
• pod
• poe_filter
• pony
• printf
• prisma
• promql
• properties
• proto
• prql
• psv
• pug
• puppet
• purescript
• pymanifest
• python
• ql
• qmldir
• qmljs
• query
• r
• racket
• rasi
• rbs
• re2c
• readline
• regex
• rego
• requirements
• rnoweb
• robot
• ron
• rst
• ruby
• rust
• scala
• scfg
• scheme
• scss
• slang
• slint
• smali
• smithy
• snakemake
• solidity
• soql
• sosl
• sourcepawn
• sparql
• sql
• squirrel
• ssh_config
• starlark
• strace
• styled
• supercollider
• surface
• svelte
• swift
• sxhkdrc
• systemtap
• t32
• tablegen
• tcl
• teal
• templ
• terraform
• textproto
• thrift
• tiger
• tlaplus
• tmux
• todotxt
• toml
• tsv
• tsx
• turtle
• twig
• typescript
• typoscript
• typst
• udev
• ungrammar
• unison
• usd
• uxntal
• v
• vala
• verilog
• vhs
• vim
• vimdoc
• vue
• wgsl
• wgsl_bevy
• wing
• xcompose
• xml
• yaml
• yang
• yuck
• zathurarc
• zig

Use parsers.json and lockfile.json to get information required by Language.build_library()
Add .github/workflows/update.yml to update parsers.json and lockfile.json

Continue #33
@sansmoraxz lockfile.json lock the commit

[Freed-Wu]

Also a lot of them are missing src/parser.c. (Written in JS)

Those will fail to build.

In github action, tree-sitter and npm will be installed to fix this problem.

[Freed-Wu]

tree-sitter/py-tree-sitter#189 affect many parsers.

[Freed-Wu]

Now it can work

[heijligers]

Big BUMP to have this PR integrated. Aider with Swift support would rock.

[heijligers]

Any updated what would be required to merge/approve this PR?
An eager Aider user. Thanks!

[ret2libc]

Would love to have this merged as well! Left a small comment.
Moreover, would it make sense to first just move to this new approach of handling parsers and only later add additional parsers so that the two things can be reviewed in an easier/more thorough way?

Great work anyway :)

[cmosguy]

@Freed-Wu
Something is wrong now, when I checkout your parsers branch and try to do an install I am getting an error during the python build.py process:

Initialized empty Git repository in /workdir/py-tree-sitter-languages/vendor/wing/.git/
remote: Enumerating objects: 3043, done.
remote: Counting objects: 100% (3043/3043), done.
remote: Compressing objects: 100% (2235/2235), done.
remote: Total 3043 (delta 924), reused 1813 (delta 671), pack-reused 0
Receiving objects: 100% (3043/3043), 23.79 MiB | 7.44 MiB/s, done.
Resolving deltas: 100% (924/924), done.
From https://github.com/winglang/wing
 * branch            992e76b445311e13ff18470542f5ca972fb28567 -> FETCH_HEAD
Note: switching to '992e76b445311e13ff18470542f5ca972fb28567'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 992e76b feat(sdk): implement `entries()` method inside Map (#5475)
Error processing rule _escape_sequence

Caused by:
    regex parse error:
        u{[0-9a-fA-F]+}
          ^
    error: repetition quantifier expects a valid decimal
Traceback (most recent call last):
  File "/workdir/py-tree-sitter-languages/build.py", line 56, in <module>
    subprocess.check_call(["tree-sitter", "generate"], cwd=vendor)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['tree-sitter', 'generate']' returned non-zero exit status 1.

Is this because something is wrong with the downstream of the wing language parser?

[Freed-Wu]

Can you disable wing then try again?

[sansmoraxz]

Whatever git commit you locking down the build step to should be tested across multiple archs and os.

Maybe try to build through gh actions in a separate branch.

[ret2libc]

I think I saw a similar error with newer tree sitter on another project

[Freed-Wu]

a similar error

I have updated and now I haven't found any error.

Any updated what would be required to merge/approve this PR?

Wait @grantjenks on #54

[cmosguy]

@Freed-Wu can you clarify something for me?

Can you adjust this project to run the python build.py and compile everything in as a github action? Only you can enable because you own the repo.

That way we can just include this repo in the requirements.txt file and the languages.so library will be included.

[sansmoraxz]

@cmosguy I believe that is out of the scope of this project.

From README:

py-tree-sitter-languages provides binary Python wheels for all tree sitter languages. The binary wheels remove the need to download and compile support for individual languages.

This project aims to provide precompiled tree sitter packages so you don't need to depend on a C compiler.

[cmosguy]

Thanks @sansmoraxz - I think I am confused then. The pipeline for this project compiles and generates .whl files that are used during compilation. If you look at this workflow.yaml and the setup.py it builds multiple .whl files that should be used for installation.

My point is that because @grantjenks has absconded from this project, we need someone to basically take over the reigns now and create a new project repo and PiPy project so that others can use these python bindings. A github action workflow will build the .whl files and deploy them to PiPy, but we need to transfer ownership to @Freed-Wu probably since I think he can manage this better.

What do you say @Freed-Wu ? Maybe create a whole new repo since @grantjenks is not interested in this anymore?

[FeepingCreature]

Okay, on the basis of "if nobody does anything then nothing will happen" (and also cause I really wanted aider D support), I've created a new Python package called 'tree-sitter-languages-freed-wu-pr' https://pypi.org/project/tree-sitter-languages-freed-wu-pr/ https://github.com/FeepingCreature/py-tree-sitter-languages

It changes the package name to tree_sitter_languages_freed_wu_pr to avoid collisions with this repo.

I don't know what @grantjenks or @Freed-Wu are up to, or if they're going to come back to this, but that should at least let other projects move forward for now.

I'm not committing to maintaining it. I'm especially not committing to maintaining it after the forty attempts to get the Github release action to work. But hopefully it should work at least as well as the last official release.

Changes from this PR:

• The rename
• On Windows and aarch64 only the "legacy" tree-sitter-languages (before this PR) are built.
• on Windows, because the number of files passed to the linker exceeds the commandline length limit and I didn't feel like bisecting exactly where the cutoff was
• on aarch64, because the Github release flow uses qemu and with all the new languages it's so slow that it times out the runner.
• Downgraded tree-sitter-cpp because not all py wheel targets support static_assert
• Removed just because it failed to build
• Removed norg because it requires C++14 which is not available on the MacOS target
• Removed groovy because the repo ships a .so file which annoys the wheel builder, anyway who does that?

But otherwise, ie. on Linux and Mac, all the languages from this PR should be in there and working.

If you want, you can look at the failed build actions here.