Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs.rs cannot build datafusion-proto crate #3538

Closed
andygrove opened this issue Sep 19, 2022 · 10 comments · Fixed by #3580 or #3950
Closed

docs.rs cannot build datafusion-proto crate #3538

andygrove opened this issue Sep 19, 2022 · 10 comments · Fixed by #3580 or #3950
Labels
bug Something isn't working

Comments

@andygrove
Copy link
Member

Describe the bug

[INFO] [stderr]    Compiling datafusion-proto v12.0.0 (/opt/rustwide/workdir)
[INFO] [stderr] error: failed to run custom build command for `datafusion-proto v12.0.0 (/opt/rustwide/workdir)`
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/datafusion-proto-4369f54fa6f25371/build-script-build` (exit status: 1)
[INFO] [stderr]   --- stdout
[INFO] [stderr]   cargo:rerun-if-env-changed=FORCE_REBUILD
[INFO] [stderr]   cargo:rerun-if-changed=proto/datafusion.proto
[INFO] [stderr]   Running: "/usr/bin/protoc" "--include_imports" "--include_source_info" "-o" "/tmp/prost-buildQPnxTC/prost-descriptor-set" "-I" "proto" "proto/datafusion.proto"
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr]   Error: "protobuf compilation failed: Read-only file system (os error 30)"

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.
@andygrove andygrove added the bug Something isn't working label Sep 19, 2022
@andygrove
Copy link
Member Author

@avantgardnerio @alamb This is one side-effect of the recent change to generate protobuf into the src directory. I don't have a strong opinion on how much of an issue this is, although it would be nice to have docs.

@avantgardnerio
Copy link
Contributor

@andygrove do you have a link to the build in question? The error message isn't sufficient in and of itself, but I feel like this is fixable.

@andygrove
Copy link
Member Author

@andygrove do you have a link to the build in question? The error message isn't sufficient in and of itself, but I feel like this is fixable.

https://docs.rs/crate/datafusion-proto/12.0.0

@avantgardnerio
Copy link
Contributor

Got it, so https://docs.rs/crate/datafusion-proto/12.0.0/builds/630103

@avantgardnerio
Copy link
Contributor

This looks broken because the source directory is closed to a readonly file system https://docs.rs/about/builds#read-only-directories

But should be fixable using conditional compilation flags https://docs.rs/about/builds#cross-compiling

scsmithr added a commit to GlareDB/glaredb that referenced this issue Sep 21, 2022
@scsmithr
fix: Remove datafusion-proto crate
8ac39db 
We're not using if for anything yet, and this release seems to break building
container images.

```
error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > Caused by:
       >   process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1)
       >   --- stdout
       >   cargo:rerun-if-env-changed=FORCE_REBUILD
       >   cargo:rerun-if-changed=proto/datafusion.proto
       >   Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto"
       >
       >   --- stderr
       >   Error: "protobuf compilation failed: Permission denied (os error 13)"
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'.
```

Possibly related: apache/datafusion#3538
@scsmithr scsmithr mentioned this issue Sep 21, 2022
Merged
scsmithr added a commit to GlareDB/glaredb that referenced this issue Sep 21, 2022
@scsmithr
fix: Remove datafusion-proto crate (#119)
4d0d6db 
We're not using if for anything yet, and this release seems to break building
container images.

```
error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > Caused by:
       >   process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1)
       >   --- stdout
       >   cargo:rerun-if-env-changed=FORCE_REBUILD
       >   cargo:rerun-if-changed=proto/datafusion.proto
       >   Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto"
       >
       >   --- stderr
       >   Error: "protobuf compilation failed: Permission denied (os error 13)"
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'.
```

Possibly related: apache/datafusion#3538
@avantgardnerio avantgardnerio mentioned this issue Sep 21, 2022
Merged
justinrubek added a commit to GlareDB/glaredb that referenced this issue Sep 27, 2022
@scsmithr @RustomMS @justinrubek
Get further with extended query protocol (#127)
3a2154c 
* chore: Update to datafusion 12 (#114)

* First draft of key layout using RocksDB (#116)

* First draft of key layout using RocksDB

* Additional future considerations from Sean

* feat: Add information_schema (#115)

* feat: Add information_schema

Fixes #98

Cloud will be making requests directly to the database to get info about the
contents of the database, including schemas, tables, and columns.

* fix: Remove datafusion-proto crate (#119)

We're not using if for anything yet, and this release seems to break building
container images.

```
error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > Caused by:
       >   process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1)
       >   --- stdout
       >   cargo:rerun-if-env-changed=FORCE_REBUILD
       >   cargo:rerun-if-changed=proto/datafusion.proto
       >   Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto"
       >
       >   --- stderr
       >   Error: "protobuf compilation failed: Permission denied (os error 13)"
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'.
```

Possibly related: apache/datafusion#3538

* feat: Implement raft via gRPC (#63)

* Replace toy-rpc with tonic gRPC

* implement glaredb cli for raft nodes and client

* current progress

* implement begin, allocate_table, and get_schema

* implement scan

* implement insert

* cleanup

* comment out old tests

* clean up ConsensusClient

* Implement change membership command

* rewrite cluster tests to use RaftClientSource

* add protoc to CI

* switch raft to in-memory implementation

* Remove application logic from raft cluster tests

* cargo fmt

* add tracing to RPC impls

* Remove lemur from raft crate

* remove raft_client example

* Apply suggestions from code review

Co-authored-by: Sean Smith <[email protected]>

* remove protoc from ci

* Remove lemur_impl from raft crate

* Store tonic clients instead of endpoint in ConsensusClient

* use shared n_retries

* Add default num_retries

* Apply suggestions from code review

Co-authored-by: Rustom Shareef <[email protected]>

* moved some mod.rs modules into their parent directories

* implement ConsensusClient retry to find leader using macro

* Fix missing delimiter

* fix clippy issues

* rewrite retry_rpc_on_leader macro to evaluate to an expression

* remove panics in rpc server impls

Co-authored-by: Sean Smith <[email protected]>
Co-authored-by: Rustom Shareef <[email protected]>

* build(nix): Use crane to cache cargo dependencies (#121)

* Add crane

* switch rust toolchain to come from fenix

* touch buildscript before executing cargo build

* add clippy and build checks

* rename arrowstore build script

* rename raft build script

* Send back BindComplete intead of ParseComplete

Also moved sending results into its own function since we need to send results
back after Execute commands complete.

* Add logical plan stub for SETting runtime vars

Also fixes logic for checking pg message length.

Co-authored-by: Rustom Shareef <[email protected]>
Co-authored-by: Justin Rubek <[email protected]>
justinrubek added a commit to GlareDB/glaredb that referenced this issue Sep 27, 2022
@justinrubek @scsmithr @RustomMS
feat(pgsrv): Implement start of postgres extended query protocol (#117)
f980994 
* add extended query parse message

* parse step

* bind step

* Some Describe + Some Drop table

* close server on Terminate message

* Get further with extended query protocol (#127)

* chore: Update to datafusion 12 (#114)

* First draft of key layout using RocksDB (#116)

* First draft of key layout using RocksDB

* Additional future considerations from Sean

* feat: Add information_schema (#115)

* feat: Add information_schema

Fixes #98

Cloud will be making requests directly to the database to get info about the
contents of the database, including schemas, tables, and columns.

* fix: Remove datafusion-proto crate (#119)

We're not using if for anything yet, and this release seems to break building
container images.

```
error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > Caused by:
       >   process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1)
       >   --- stdout
       >   cargo:rerun-if-env-changed=FORCE_REBUILD
       >   cargo:rerun-if-changed=proto/datafusion.proto
       >   Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto"
       >
       >   --- stderr
       >   Error: "protobuf compilation failed: Permission denied (os error 13)"
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'.
```

Possibly related: apache/datafusion#3538

* feat: Implement raft via gRPC (#63)

* Replace toy-rpc with tonic gRPC

* implement glaredb cli for raft nodes and client

* current progress

* implement begin, allocate_table, and get_schema

* implement scan

* implement insert

* cleanup

* comment out old tests

* clean up ConsensusClient

* Implement change membership command

* rewrite cluster tests to use RaftClientSource

* add protoc to CI

* switch raft to in-memory implementation

* Remove application logic from raft cluster tests

* cargo fmt

* add tracing to RPC impls

* Remove lemur from raft crate

* remove raft_client example

* Apply suggestions from code review

Co-authored-by: Sean Smith <[email protected]>

* remove protoc from ci

* Remove lemur_impl from raft crate

* Store tonic clients instead of endpoint in ConsensusClient

* use shared n_retries

* Add default num_retries

* Apply suggestions from code review

Co-authored-by: Rustom Shareef <[email protected]>

* moved some mod.rs modules into their parent directories

* implement ConsensusClient retry to find leader using macro

* Fix missing delimiter

* fix clippy issues

* rewrite retry_rpc_on_leader macro to evaluate to an expression

* remove panics in rpc server impls

Co-authored-by: Sean Smith <[email protected]>
Co-authored-by: Rustom Shareef <[email protected]>

* build(nix): Use crane to cache cargo dependencies (#121)

* Add crane

* switch rust toolchain to come from fenix

* touch buildscript before executing cargo build

* add clippy and build checks

* rename arrowstore build script

* rename raft build script

* Send back BindComplete intead of ParseComplete

Also moved sending results into its own function since we need to send results
back after Execute commands complete.

* Add logical plan stub for SETting runtime vars

Also fixes logic for checking pg message length.

Co-authored-by: Rustom Shareef <[email protected]>
Co-authored-by: Justin Rubek <[email protected]>

* fix clippy issues

* replaced dbg usage with tracing

* updated some comments

* comment terminate message

Co-authored-by: Sean Smith <[email protected]>
Co-authored-by: Rustom Shareef <[email protected]>
@tustvold tustvold reopened this Oct 25, 2022
@tustvold
Copy link
Contributor

Docs builds are still failing

@tustvold tustvold mentioned this issue Oct 25, 2022
Merged
@avantgardnerio
Copy link
Contributor

Docs builds are still failing

Can you explain what you mean please? I see the results and the build log contains no errors (only warnings).

@alamb
Copy link
Contributor

alamb commented Oct 25, 2022

Perhaps https://docs.rs/crate/datafusion-proto/13.0.0

With https://docs.rs/crate/datafusion-proto/13.0.0/builds/646416 failure:

@alamb
Copy link
Contributor

alamb commented Oct 25, 2022

BTW the fact that generated code definition jump to works in datafusion-proto is awesome!

@avantgardnerio
Copy link
Contributor

avantgardnerio commented Oct 25, 2022 •
edited
Loading

With https://docs.rs/crate/datafusion-proto/13.0.0/builds/646416 failure:

For the record:

[INFO] [stderr]   Running: "/usr/bin/protoc" "--include_imports" "--include_source_info" "-o" "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/datafusion-proto-7d222b4b383775ea/out/proto_descriptor.bin" "-I" "proto" "proto/datafusion.proto"
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr]   thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" }', build.rs:76:10

It looks like my previous fix got the main build to work on docs.rs, but specifically not the datafusion-proto crate. And unfortunately docs.rs will only let us generate into the OUT_DIR.

I see 3 possible solutions:

  1. generate into OUT_DIR, but only for docs.rs (getting complicated and painful here)
  2. we always generate to OUT_DIR and as a CLion user, I enable the experimental feature that make it work. vscode users aren't supported
  3. We merge Vendor Generated Protobuf Code (#3947) #3950

at this point I'm fine with the 3rd option. I just want to keep momentum going.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
4 participants
@alamb @andygrove @tustvold @avantgardnerio