First draft of key layout using RocksDB #116
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scsmithr
approved these changes
Sep 20, 2022
RustomMS
force-pushed
the
storage-doc
branch
from
4047b54 to
2c486be
Compare
justinrubek
added a commit
that referenced
this pull request
Sep 27, 2022
* chore: Update to datafusion 12 (#114) * First draft of key layout using RocksDB (#116) * First draft of key layout using RocksDB * Additional future considerations from Sean * feat: Add information_schema (#115) * feat: Add information_schema Fixes #98 Cloud will be making requests directly to the database to get info about the contents of the database, including schemas, tables, and columns. * fix: Remove datafusion-proto crate (#119) We're not using if for anything yet, and this release seems to break building container images. ``` error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101; last 10 log lines: > Caused by: > process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1) > --- stdout > cargo:rerun-if-env-changed=FORCE_REBUILD > cargo:rerun-if-changed=proto/datafusion.proto > Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto" > > --- stderr > Error: "protobuf compilation failed: Permission denied (os error 13)" > warning: build failed, waiting for other jobs to finish... For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'. ``` Possibly related: apache/datafusion#3538 * feat: Implement raft via gRPC (#63) * Replace toy-rpc with tonic gRPC * implement glaredb cli for raft nodes and client * current progress * implement begin, allocate_table, and get_schema * implement scan * implement insert * cleanup * comment out old tests * clean up ConsensusClient * Implement change membership command * rewrite cluster tests to use RaftClientSource * add protoc to CI * switch raft to in-memory implementation * Remove application logic from raft cluster tests * cargo fmt * add tracing to RPC impls * Remove lemur from raft crate * remove raft_client example * Apply suggestions from code review Co-authored-by: Sean Smith <[email protected]> * remove protoc from ci * Remove lemur_impl from raft crate * Store tonic clients instead of endpoint in ConsensusClient * use shared n_retries * Add default num_retries * Apply suggestions from code review Co-authored-by: Rustom Shareef <[email protected]> * moved some mod.rs modules into their parent directories * implement ConsensusClient retry to find leader using macro * Fix missing delimiter * fix clippy issues * rewrite retry_rpc_on_leader macro to evaluate to an expression * remove panics in rpc server impls Co-authored-by: Sean Smith <[email protected]> Co-authored-by: Rustom Shareef <[email protected]> * build(nix): Use crane to cache cargo dependencies (#121) * Add crane * switch rust toolchain to come from fenix * touch buildscript before executing cargo build * add clippy and build checks * rename arrowstore build script * rename raft build script * Send back BindComplete intead of ParseComplete Also moved sending results into its own function since we need to send results back after Execute commands complete. * Add logical plan stub for SETting runtime vars Also fixes logic for checking pg message length. Co-authored-by: Rustom Shareef <[email protected]> Co-authored-by: Justin Rubek <[email protected]>
justinrubek
added a commit
that referenced
this pull request
Sep 27, 2022
* add extended query parse message * parse step * bind step * Some Describe + Some Drop table * close server on Terminate message * Get further with extended query protocol (#127) * chore: Update to datafusion 12 (#114) * First draft of key layout using RocksDB (#116) * First draft of key layout using RocksDB * Additional future considerations from Sean * feat: Add information_schema (#115) * feat: Add information_schema Fixes #98 Cloud will be making requests directly to the database to get info about the contents of the database, including schemas, tables, and columns. * fix: Remove datafusion-proto crate (#119) We're not using if for anything yet, and this release seems to break building container images. ``` error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101; last 10 log lines: > Caused by: > process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1) > --- stdout > cargo:rerun-if-env-changed=FORCE_REBUILD > cargo:rerun-if-changed=proto/datafusion.proto > Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto" > > --- stderr > Error: "protobuf compilation failed: Permission denied (os error 13)" > warning: build failed, waiting for other jobs to finish... For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'. ``` Possibly related: apache/datafusion#3538 * feat: Implement raft via gRPC (#63) * Replace toy-rpc with tonic gRPC * implement glaredb cli for raft nodes and client * current progress * implement begin, allocate_table, and get_schema * implement scan * implement insert * cleanup * comment out old tests * clean up ConsensusClient * Implement change membership command * rewrite cluster tests to use RaftClientSource * add protoc to CI * switch raft to in-memory implementation * Remove application logic from raft cluster tests * cargo fmt * add tracing to RPC impls * Remove lemur from raft crate * remove raft_client example * Apply suggestions from code review Co-authored-by: Sean Smith <[email protected]> * remove protoc from ci * Remove lemur_impl from raft crate * Store tonic clients instead of endpoint in ConsensusClient * use shared n_retries * Add default num_retries * Apply suggestions from code review Co-authored-by: Rustom Shareef <[email protected]> * moved some mod.rs modules into their parent directories * implement ConsensusClient retry to find leader using macro * Fix missing delimiter * fix clippy issues * rewrite retry_rpc_on_leader macro to evaluate to an expression * remove panics in rpc server impls Co-authored-by: Sean Smith <[email protected]> Co-authored-by: Rustom Shareef <[email protected]> * build(nix): Use crane to cache cargo dependencies (#121) * Add crane * switch rust toolchain to come from fenix * touch buildscript before executing cargo build * add clippy and build checks * rename arrowstore build script * rename raft build script * Send back BindComplete intead of ParseComplete Also moved sending results into its own function since we need to send results back after Execute commands complete. * Add logical plan stub for SETting runtime vars Also fixes logic for checking pg message length. Co-authored-by: Rustom Shareef <[email protected]> Co-authored-by: Justin Rubek <[email protected]> * fix clippy issues * replaced dbg usage with tracing * updated some comments * comment terminate message Co-authored-by: Sean Smith <[email protected]> Co-authored-by: Rustom Shareef <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.