Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilation errors when disabling default features in datafusion-common #7624

Closed
jonmmease opened this issue Sep 22, 2023 · 2 comments · Fixed by #7625
Closed

Compilation errors when disabling default features in datafusion-common #7624

jonmmease opened this issue Sep 22, 2023 · 2 comments · Fixed by #7625
Labels
bug Something isn't working

Comments

@jonmmease
Copy link
Contributor

Describe the bug

When depending on datafusion-common with default-features = false, there are compilation errors related to parquet functionality.

To Reproduce

In the datafusion-expr crate, add default-features = false to the datafusion-common dependency. Then try to build the datafusion-expr crate.

There are compilation errors such as:

error[E0433]: failed to resolve: use of undeclared crate or module `parquet`
  --> datafusion/common/src/file_options/parse_utils.rs:20:5
   |
20 | use parquet::{
   |     ^^^^^^^ use of undeclared crate or module `parquet`

error[E0433]: failed to resolve: use of undeclared crate or module `parquet`
  --> datafusion/common/src/file_options/parquet_writer.rs:20:5
   |
20 | use parquet::file::properties::{WriterProperties, WriterPropertiesBuilder};
   |     ^^^^^^^ use of undeclared crate or module `parquet`

error[E0433]: failed to resolve: use of undeclared crate or module `parquet`
  --> datafusion/common/src/file_options/parse_utils.rs:43:13
   |
43 | ) -> Result<parquet::basic::Encoding> {
   |             ^^^^^^^ use of undeclared crate or module `parquet`

error[E0433]: failed to resolve: use of undeclared crate or module `parquet`
  --> datafusion/common/src/file_options/parse_utils.rs:46:23
   |
46 |         "plain" => Ok(parquet::basic::Encoding::PLAIN),
   |                       ^^^^^^^ use of undeclared crate or module `parquet`
   |
...

Expected behavior

It would be possible to compile datafusion-common with the default parquet feature disabled.

Additional context

In VegaFusion I compile datafusion-common into WASM, and the parquet crate is not (easily?) compatible with WASM due to some non-rust compression dependencies.

I'll have a PR with suggested fix up shortly

@alamb
Copy link
Contributor

alamb commented Sep 23, 2023

In VegaFusion I compile datafusion-common into WASM, and the parquet crate is not (easily?) compatible with WASM due to some non-rust compression dependencies.

@jonmmease in order to avoid such errors in the future, perhaps we can encode the usecase into a CI test

Can you share the command / commands you use to compile for wasm (specifically what target)? Here is what we use for sqlparser-rs

https://github.com/sqlparser-rs/sqlparser-rs/blob/7723ea56c5119c7d1a15233c18eb2aaf48b60dc0/.github/workflows/rust.yml#L48-L56

But I

@jonmmease
Copy link
Contributor Author

That's a good idea. I made a PR in #7633 that adds a minimal wasm library crate that depends on the various DataFusion crates that are currently wasm compatible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants