2024-09-19 kernel meeting notes

[zachschuermann]

summary

lots of discussion on arrow transitive deps. planning on splitting crates and/or specifying arrow dependency enum

pull requests/ongoing threads

• removing default features! change default features to be none #339
• todo

attendees

@nicklan @rtyler @zachschuermann @hntd187 @scovich @OussamaSaoudi-db

notes stream

arrow transitive dependencies

arrow dependency difficulty. can we support the last N (N=2?) versions of arrow?
tyler: supporting multiple versions through feature flags is ew
as long as we bump major version (0.x) for arrow updates
tyler concern is cfg-if

if we churn so much, we might need to split the crate (also note these aren't mutually exlusive)

we could split crate first then add config flags if needed. are there issues where people want to bump core version but not engine and are they compatible?

arrow-rs: problem is ABI change not necessarily API change. from ABI point, pyarrow/arrow-rs have maintained compat.

kernel schema/ arrow schema required ABI compat.

does arrow break binary compat regularly but not source compat? would be nice for the feature flag story.

if there are no code changes at all: can we say "bring any arrow >= 52 etc.?"

cross the bridge with feature flags if we get there? (a place where we need source changes)

tyler: enumerate the actual arrow versions we allow (arrow = 51, 52, 53) and then manually add versions at the top as they are released. have to have testing for bottom end of the range.

tyler advocating for integration tests (separate crate to test kernel) - external to cargo workspace. need outside workspace because cargo.lock is shared.

conclusion: split out engine. but first do arrow version enumeration. and we have new crate (outside workspace) to test.