A few things that made me avoid bindgen in my project

[HadrienG2]

I have ended up writing manual bindings in a project where it feels like bindgen should have been the right tool, because there were too many things I did not like about (my understanding of) how bindgen works.

I am not sure which of these aspects of bindgen are conscious design choices, which are involuntary limitations that would be worth fixing someday, and which are simple misunderstandings from my part (not actual bindgen limitations, can be overcome by someone who understands the library more than my quick reading of the manual + experimentation with the CLI allowed).

So here are my notes in the hope that they will be useful to you:

• I don't like dealing with other -sys crates that were autogenerated using bindgen because they are fully undocumented, which makes me constantly round trip between the crate and underlying C lib documentation. In my ideal world, bindgen would be able to translate doxygen into rustdoc. The translation does not have to be very good to be a huge improvement over what we have right now, e.g. it would already be good to only parse the markdown part and leave uninterpreted doxygen directives around.
  • I would also like to be able to append extra stuff to the auto-translated docs of some functions, as C libraries are traditionally quite bad at e.g. articulating their safety invariants and upstream may not accept doc contributions in this area.
• I don't want to impose an extra libclang dependency on my users, and I'm binding a library whose headers will vary from one system to another so baking a pre-generated binding into my repository is not an option. In my ideal world, bindgen would provide some optional crate feature where it brings its own libclang binary (possibly by leveraging some existing, well-respected source of LLVM binary releases).
• I don't like that many things in bindgen are controlled using regexes over C entity names, because regexes on open sets are a footgun that is likely to age badly as the target C project evolves. In my ideal world, a less brittle filtering system would be available.
• I don't like that bindgen produces lots of irrelevant junk from e.g. libc. I am aware that C's header inclusion model is not making things easy here, but in my ideal world, things which people will commonly want to exclude, like glibc headers, should have ready-made exclusion lists that are easy for bindgen users to apply.
  • Bonus point for having an easy way to filter out everything from #include <something.h> with angle brackets, which are conventionally almost always external dependencies of the library of interest whose declarations should not end up in the present binding but in a separate binding for that external dependency.

[emilio]

Thanks for the feedback!

* I don't like dealing with other `-sys` crates that were autogenerated using bindgen because they are fully undocumented, which makes me constantly round trip between the crate and underlying C lib documentation. In my ideal world, bindgen would be able to translate doxygen into rustdoc. The translation does not have to be very good to be a huge improvement over what we have right now, e.g. it would already be good to only parse the markdown part and leave uninterpreted doxygen directives around.

We do try to generate documentation, see the documentation for the generate_comments option. If there's a case where that isn't working for you, then that'd be worth a report so it can be looked at.

We don't try to do any fancy doxygen -> rustdoc conversion though. Not opposed to add that.

  * I would also like to be able to append extra stuff to the auto-translated docs of some functions, as C libraries are traditionally quite bad at e.g. articulating their safety invariants and upstream may not accept doc contributions in this area.

That we don't currently support. No reason it can't be added, but hadn't come up before afaict.

* I don't want to impose an extra `libclang` dependency on my users, and I'm binding a library whose headers will vary from one system to another so baking a pre-generated binding into my repository is not an option. In my ideal world, bindgen would provide some optional crate feature where it brings its own libclang binary (possibly by leveraging some existing, well-respected source of LLVM binary releases).

Yeah, so that's trickier. I don't think we want to make the bindgen crate include all of LLVM and build it, or update a pre-built version to crates.io for all platforms.

We have a static feature but that is not suitable if you're building bindgen on the target machine, or using it as a library. Maybe our pre-built releases could have a statically-linked LLVM tho. Not sure if that works for your use case?

Alternatively you can bundle LLVM and point bindgen at it with LIBCLANG_PATH, right?

* I don't like that many things in bindgen are controlled using regexes over C entity names, because regexes on open sets are a footgun that is likely to age badly as the target C project evolves. In my ideal world, a less brittle filtering system would be available.

We need to deal with C++ which can be a lot more complicated...

* I don't like that bindgen produces lots of irrelevant junk from e.g. libc. I am aware that C's header inclusion model is not making things easy here, but in my ideal world, things which people will commonly want to exclude, like glibc headers, should have ready-made exclusion lists that are easy for bindgen users to apply.

You can filter with the allowlist_* APIs right? The general idea is to use allowlist_ for the things you want / need to generate, and bindgen will only generate those and their dependencies. But you can also filter by file with allowlist_file.

  * Bonus point for having an easy way to filter out everything from `#include <something.h>` with angle brackets, which are conventionally almost always external dependencies of the library of interest whose declarations should not end up in the present binding but in a separate binding for that external dependency.

See above. Does that not work for your use case? If not it'd be good to know why.

[HadrienG2]

Thanks for your reply!

We do try to generate documentation, see the documentation for the generate_comments option. If there's a case where that isn't working for you, then that'd be worth a report so it can be looked at.

After closer investigation, some docs are translated and some aren't. It is not immediately clear why. I'll try to minimize some test cases and submit issues.

We don't try to do any fancy doxygen -> rustdoc conversion though. Not opposed to add that.

I'll create some issues for easy quality of life improvements like stripping leading \brief and \short. Not sure if it would be a good idea for bindgen to attempt translations of more complex doxygen patterns as this can easily become a rabbit hole...

I would also like to be able to append extra stuff to the auto-translated docs of some functions, as C libraries are traditionally quite bad at e.g. articulating their safety invariants and upstream may not accept doc contributions in this area.

That we don't currently support. No reason it can't be added, but hadn't come up before afaict.

I'll file a feature request for this.

I don't want to impose an extra libclang dependency on my users, and I'm binding a library whose headers will vary from one system to another so baking a pre-generated binding into my repository is not an option. In my ideal world, bindgen would provide some optional crate feature where it brings its own libclang binary (possibly by leveraging some existing, well-respected source of LLVM binary releases).

Yeah, so that's trickier. I don't think we want to make the bindgen crate include all of LLVM and build it, or update a pre-built version to crates.io for all platforms.

Yeah, the sanest course of action would probably be to download and unpack the relevant binary release from LLVM's github releases or some similarly respected binaries source when the relevant (optional, off by default) feature is on. The main problem is that these are huge, so a little bot that automatically extracts libclang-only packages from them would be useful.

We have a static feature but that is not suitable if you're building bindgen on the target machine, or using it as a library. Maybe our pre-built releases could have a statically-linked LLVM tho. Not sure if that works for your use case?

Since I'm writing a binding to some C library and cannot use AoT bindgen because there are OS-dependent definitions in the header file, I don't think this option would work for me.

Alternatively you can bundle LLVM and point bindgen at it with LIBCLANG_PATH, right?

I'll try this using the above strategy. If it works well, would you be open to a bindgen integration PR so other projects can benefit from it?

• I don't like that bindgen produces lots of irrelevant junk from e.g. libc. I am aware that C's header inclusion model is not making things easy here, but in my ideal world, things which people will commonly want to exclude, like glibc headers, should have ready-made exclusion lists that are easy for bindgen users to apply.

You can filter with the allowlist_* APIs right? The general idea is to use allowlist_ for the things you want / need to generate, and bindgen will only generate those and their dependencies. But you can also filter by file with allowlist_file.

• Bonus point for having an easy way to filter out everything from #include <something.h> with angle brackets, which are conventionally almost always external dependencies of the library of interest whose declarations should not end up in the present binding but in a separate binding for that external dependency.

See above. Does that not work for your use case? If not it'd be good to know why.

I've been experimenting with the CLI so far, so I need to play a bit with the local allowlist API equivalents, but I spontaneously get the impression that this is not so easy. For example, the bindgen output includes every define from GCC and glibc even though hwloc (the library I am binding) doesn't use most of them. Will play with it some more and ask additional questions here or file issues as appropriate.

[HadrienG2]

Follow-up after further investigation:

• Issues have been created/bumped for things that look like clear bugs or feature requests.
• Turns out allowlist and blocklist could do what I want, with some care and a bit of extra scripting ;)
• Missing docs came from clang stripping item docs in system headers, it only took a -fretain-comments-from-system-headers clang arg to fix them.
• Waiting for your input on the best strategy for pre-built libclang (is this wanted as an upstream bindgen feature?), will start experimenting with it meanwhile.

[HadrienG2]

I have just checked that bindgen only needs the libclang.so binary from the official LLVM binary distribution, which would cut the download size to 36MB after gzip compression or 23MB after xz compression. Still a relatively big download, but quite a large improvement from the original 1 GB.

It's probably possible to do better with a custom clang build that contains just what bindgen needs and nothing more, but then the required infrastructure becomes a lot more complex. Right now, this is all you need to do for each LLVM archive you want to minifiy:

mkdir libclang
cd libclang
# Need to adapt the URL to the LLVM release and OS of interest
curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.6/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04.tar.xz | tar --strip-components=2 --wildcards -xJ 'clang+llvm-*/lib/libclang.so*'
cd ..
# Using xz compression here takes more time, but produces 1/3 smaller archives
tar -czf libclang.tar.gz libclang
rm -r libclang

[pvdrz]

👋 I'm here to drop my 2 cents about certain topics

Regarding the allowlist/blocklist API: yeah I get it, regexes aren't the prettiest, but, as you mentioned, the lack of modularity in C leaves us with few choices to filter items. C++ is less prone to the so it could be reasonable to have a more structured API there but I'm not sure how widespread its use would be.

Regarding the prebuilt clang, I feel this is a discussion we've had in the past without a clear answer so it might be worth it to open an issue/discussion for it. I think it has a legitimate use case but it also brings the question of which libclang version should be used by default.

[HadrienG2]

Regarding the prebuilt clang, I feel this is a discussion we've had in the past without a clear answer so it might be worth it to open an issue/discussion for it. I think it has a legitimate use case but it also brings the question of which libclang version should be used by default.

I think #2767 and #918 are closely related to this idea, so let me just link to this part of the discussion there...