C API extensions

[samansmink]

This PR:

• Reworks the DuckDB C API header duckdb.h to use code-generation
• Adds a new C Extension API based on a struct-of-functionpointers design similar to sqlite

Why?

The C Extension API solves 3 major problems with extensions:

Linking DuckDB into every extension

Currently every extension has a (part of) DuckDB linked into it statically. When an extension's init function is called, a pointer to the DuckDB instance is passed to the extension code. However, to interact with
this pointer to the instance in any meaningful way, requires to be able to call into DuckDB code. There are two ways this can be achieved when extending the C++ API: Firstly, by relying on the RTLD_GLOBAL flag. This flag
can be set when dlopening binaries to ensure the dlopened extension can resolve its missing symbols against those in the parent process, i.e. DuckDB. Unfortunately we can't always control whether DuckDB itself is opened with RTLD global: this would require us to hack this into every client (because the default on MacOS is RTLD_LOCAL). The second way is what we currently do: statically link a copy of DuckDB into every extension. While this is super wasteful with regards to the binary size, it turns out to be quite effective in distributing extensions to many platforms.

Now using the extension C API implementation described here, this problem would be circumvented completely: the extension can call into duckdb using only function pointers: no linking required there! This makes the extensions really small! for example the extension in my demo branch compiles to a 17K binary.

Rebuilding extensions for new DuckDB versions

C++ API Extension are now tied tightly to the DuckDB version: Because we make no guarantees about C++ API stability and also because of how we link DuckDB into every extension binary, all extensions need to be recompiled for new versions of DuckDB. With C api extensions, this is no longer the case. We can simply build the extension once, and as long as DuckDB supports that version of the C API, the extension binary will be loadable.

Requiring C++ for writing your extensions

Cleanly writing extensions in C, Rust, Go will be possible building on top of the C API. By using the "passing a c struct" method from SQLite, we remove the need to link with duckdb during build time whatsoever, making things much simpler, allowing extension to be created purely using the extension header file.

Implementation

Overview

The basic idea of this is that instead of relying on dynamic or static linking to call into DuckDB functions, the extension init function gets passed a struct containing all available functions. This would
look a little like:

typedef struct {
	// ...
	idx_t (*duckdb_data_chunk_get_size)(duckdb_data_chunk chunk);
	duckdb_vector (*duckdb_data_chunk_get_vector)(duckdb_data_chunk chunk, idx_t col_idx);
	// ...
} duckdb_ext_api_v1;

The code for a C Extension API based extension is very straightforward, due to some macros:

#include "duckdb_extension.h"

DUCKDB_EXTENSION_ENTRYPOINT(duckdb_connection connection, duckdb_extension_info info, duckdb_extension_access *access) {
    // Register extension stuff here
}

Checkout the capi_demo extension added in this PR for an example.

Code generation

I've split up the C API across two files:

• duckdb.h is the original capi header file
• duckdb_extension.h is the header that extensions need to include, it contains the struct-of-function-pointers and typedefs to the original extension names.

The code is generated from json files which are in src/include/duckdb/main/capi/header_generation/. There are 2 types of json files. function definition files define all possible functions that are in the C API. These can potentially be deprecated. api definition files contain the definition (and order) of the C API struct.

[samansmink]

@Mause @Mytherin @carlopi @Maxxen got a first version of c api for extensions, would love to get some feedback before spending more time here

[rustyconover]

This looks fine from the perspective of someone experienced with the extension API. It won’t significantly affect API consumers from the Rust side.

I appreciate that this PR allows the extension API to evolve. I suggest adding the ability for DuckDB to support older extension API versions by passing multiple structures upon extension initialization. In the init function, can you extend it so DuckDB can send a list of supported extension API structures, and the extension can choose which version to use?

For example, duckdb_function_set_error() only works with table-returning functions, but it should be called duckdb_table_function_set_error(), given the existence of duckdb_scalar_function_set_error(). This could be addressed in a newer version of the API, where as existing extensions could just keep working like nothing ever changed.

Currently, we lack many features of the C++ extension API in the C API. Some examples are:

1. Interacting with duckdb_vector is difficult for extension writers. GenericExecutor and similar tools make this much simpler.
2. The C API cannot define new types and set aliases for them.
3. There is no way to perform predicate pushdown from the C API for table-returning functions.

Closing these gaps by making the extension API easier to extend would be beneficial.

[Mytherin]

Interacting with duckdb_vector is difficult for extension writers. GenericExecutor and similar tools make this much simpler.

The reason tools like GenericExecutor work well in C++ is due to templating/inlining - that unfortunately doesn't really work in C. As such you would incur function call overheads (that get optimized away in the C++ versions) - which would make such wrappers a lot slower than they would need to be.

I also think we should limit convenience wrappers in the C API and focus primarily on feature parity/performance. When the C API is sufficiently extended we can design wrappers in other languages that build on top of the C API (e.g. Rust/Go, maybe C++ itself) that build on top of this API and provide a better usability experience for writing extensions. Ideally few people would actually need to interact with the C API outside of the developers maintaining the wrappers at that point, making the convenience wrappers at that layer moot.

[Maxxen]

I like the idea of maybe having multiple structs be returned as we could compartmentalize parts of the API. e.g. scalar and table functions could have their own API structs that can be evolved independently. Although I guess we would need a common "core" api for types and values anyway so maybe this just causes a bunch of unnecessary complexity.

However, I wonder if it's not better to start from scratch with the extension API instead of having it be derived from the current C-API...? As this is a new interface we don't have to worry about breaking ABI, we can remove deprecated apis and generally rethink stuff (like e.g. error handling) with the benefit of hindsight and with a (perhaps) clearer goal of what functionality extensions need. We can also omit other client-related stuff like result collection etc. I know Im not alone in being annoyed at some of the inconsistency that has accrued. I know this might sound daunting but I think we could come up with a draft if we just schedule an afternoon of mob-programming in the office.

[samansmink]

@Maxxen

However, I wonder if it's not better to start from scratch with the extension API instead of having it be derived from the current C-API...?

So for this demo I just added every function that is not deprecated to the v0 api struct. I think its a fair point that this would be and execellent point to reconsider if there are parts of the C CAPI we are not happy with and simply not include them in the struct.

table functions could have their own API structs that can be evolved independently.

That's an interesting idea, but it would indeed make things a bit more complex. Note also that even though we can then more easily change the apis, we can not make a lot of changes anyway, because we need to support both the before and after of every change we make, so we shouldn't be making lots of changes anyway.

All in all I'm not super convinced that splitting the structs is worth the effort but let's discuss tomorrow!

@rustyconover
Thanks for the feedback! Passing an array of structs is an approach I had not considered yet, that does sound like an interesting option. Right now what i've done is use the binary footer that is added to every extension to store a semantic version number of the extension api. Before loading an extension, DuckDB will look up which version of the extension API the extension requires and pass it the struct it expects. I think that also solves what you want to achieve by passing an array of structs?

[carlopi]

On Load, I would try hard to re-design the init function so that it takes parameters that are compatible between C++ and C API.
Given extension (up to this moment) are valid only for a given version, I think this is the right moment to do this, and rely on metadata to avoid that sane users will not mix and match.

I think it's better to have a single uniform init function that try to find a given function.
If we go with different functions, the main INIT function should still be declared and return failure in a way that is understood by the system (unsure how, since you can't throw, maybe adding a return parameter then)

Possibly for later, unsure what's the ideas here:

• unsure if CAPI extensions can be statically loaded at this moment, of what's the path there
• on Install, where are CAPI extension found ( either together with regular ones OR in a different subtree like capi/v0/$platform/$extension_name.duckdb_extension )
• on Install, where are CAPI extension saved locally (also here, either as a regular one OR as in a separate folder)

[samansmink]

On Load, I would try hard to re-design the init function so that it takes parameters that are compatible between C++ and C API.

Hmm technically c++ extensions can already use the C API init function I wrote here, they can just fetch the DatabaseInstance from the connection directly. Maybe we can indeed unify this right away.

I think it's better to have a single uniform init function that try to find a given function.
If we go with different functions, the main INIT function should still be declared and return failure in a way that is understood by the system (unsure how, since you can't throw, maybe adding a return parameter then)

i think i prefer letting duckdb handle the compatibility checking, but i agree that the hard dependency on having the metadata present is not ideal. I would say we have these options:

• add some macro magic to the duckdb_extension.h to verify the C api version in the init function extension side
• do some sort of handshake with a _version function that returns the C api extension version
• rely on the extension metadata as i've done in this PR

I went with #3, but i think either of these options could work

unsure if CAPI extensions can be statically loaded at this moment, of what's the path there

we could and probably should have this. Adding it shouldn't be too hard though

on Install, where are CAPI extension found .... on Install, where are CAPI extension saved locally

good point, i've left this to a future PR. it's implementation-wise completely separate from this PR, i think its just a matter of where to look for extensions for downloading and where to store/load them from

[Mytherin]

Thanks! Looks great - some comments from my side

[Mytherin]

This can be a duckdb_database, and shouldn't be a reference

[Mytherin]

Maybe we can do something like this instead to keep things extensible (ha);

void demo_capi_init_capi(duckdb_extension_info *info, duckdb_extension_access *access) {
	auto db = access->get_database(info);
	auto api = access->get_api(info, DUCKDB_API_VERSION);
	if (!api) {
		return;
	}
    ...
}

[Mytherin]

This could be wrapped into DUCKDB_EXTENSION_LOAD_API

[Mytherin]

duckdb_extension_info info(db);
duckdb_extension_access access_methods;

ext.Load(&info, &access_methods);
if (info.LoadedUnsupportedVersion()) {
	throw ...
}

[Mytherin]

Can we remove all deprecated methods from the struct?

[Mytherin]

This shouldn't be in duckdb.h

[samansmink]

I've processed the comments here. To summarize the changes:

• removed the DUCKDB_API_0_3_1 etc. version defines from duckdb.h Note that ive also removed the DUCKDB_NO_EXTENSION_FUNCTIONS flag.
• refactored the api struct definition file into multiple files to make it more explicit that you can only add new ones, not edit existing ones.
• reworked the way we enforce the versioning of the api struct: this now works using defines that prevent compilation if a function is used from a version higher than the one selected.
• the API struct is now created on stack and copied into the extension global variable
• there is now a "dev" version which can hold newly added functions, these functions are not promised to be stable and cannot be used by extensions that want binary compatibility with multiple versions of duckdb.
• there is an exclude list for functions that are not in the api struct to make that explicit. This is now enforced by the api generation script
• the api generation script is refactored slightly

[Mytherin]

Thanks!

[jangorecki]

I wonder if we could get a follow up of this PR that would cover

• extension written in C, rather than C++ as now in extension/demo_capi (which actually looks like C code, but has cpp file extension?)
• readme that describes structure of the new interface

[samansmink]

Hey @jangorecki yea there will be follow ups here! Both documentation and testing are lacking still.

As things materialize here, I'm also planning to write a blogpost to describe the whole thing in more detail.

[a10y]

Hey! Just curious, are there any plans to add the filters to the C API? I have a file format (implemented in Rust) that I'd like to write an extension for reading. The format has statistics/zone-maps so it could support predicate pushdown, but AFAICT that's currently not exposed as part of the table function C API.

[Mytherin]

Yes, that is definitely on the roadmap

[Florents-Tselai]

@samansmink

I'm trying this out in an SQLite-like way, but I'm hitting a few roadblocks.

Here is the closes I've managed (see https://github.com/Florents-Tselai/mycext/actions/runs/10849069915/job/30107489251#step:6:5)

IO Error: Extension "./mycext.duckdb_extension" could not be loaded: ./mycext.duckdb_extension: undefined symbol: duckdb_register_scalar_function

I'm also building 1.1.0 from the source and using Makefile.
Emulating the extension writing process for SQLite/Postgres here:

On my Mac though, things look a bit better in that the init function is indeed called,
but the scalar function doesn't appear in the catalog.

aaaadding myaddfunc
Catalog Error: Scalar Function with name myaddfunc does not exist!
Did you mean "mad"?
LINE 1: select myaddfunc(1,2)

Am I missing something obvious here? (i.e. have to pass extra flags when building from src)
Am I too naive to expect this to work that simply?

[samansmink]

@Florents-Tselai did you follow the example extension? https://github.com/duckdb/duckdb/tree/main/extension/demo_capi

[ddevienne]

Hi. The _extension.h header is missing from the amalgamation source release. Normal?

[samansmink]

@ddevienne I guess it should be added there indeed. For now you can just grab the header from https://github.com/duckdb/duckdb/blob/v1.1.1/src/include/duckdb_extension.h

pr: #14086

[yiyuanliu]

How can I determine if a value is null in the C API? It seems there isn't a method for this currently. If I call duckdb_get_xxx methods directly on a null value, it crashes.