From 8b2c3e1cd8f01f222d36ca0091354cea89bea5b2 Mon Sep 17 00:00:00 2001 From: Evgenii Khramkov Date: Fri, 15 Mar 2024 10:06:32 +0900 Subject: [PATCH 1/3] Update acknowledgments (#109) * Update acknowledgments * Remove spiceai packages --- .../content/en/acknowledgements/_index.md | 66 +++++++++++-------- 1 file changed, 39 insertions(+), 27 deletions(-) diff --git a/spiceaidocs/content/en/acknowledgements/_index.md b/spiceaidocs/content/en/acknowledgements/_index.md index 7b3203a4c..64c7d153b 100644 --- a/spiceaidocs/content/en/acknowledgements/_index.md +++ b/spiceaidocs/content/en/acknowledgements/_index.md @@ -78,16 +78,18 @@ gopkg.in/yaml.v3, https://github.com/go-yaml/yaml/blob/v3.0.1/LICENSE, MIT | name | version | authors | repository | license | license_file | description | |------|---------|---------|------------|---------|--------------|-------------| | name | version | authors | repository | license | license_file | description | -| app | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | All loaded components from the root Spicepod and its dependencies | -| arrow | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Rust implementation of Apache Arrow | -| arrow-flight | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Arrow Flight | +| arrow | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Rust implementation of Apache Arrow | +| arrow-flight | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Arrow Flight | | arrow-ipc | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for the Arrow IPC format | -| arrow-json | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for parsing JSON format into the Arrow format | -| arrow_sql_gen | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | +| arrow-ipc | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for the Arrow IPC format | +| arrow-json | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for parsing JSON format to and from the Arrow format | | async-stream | 0.3.5 | Carl Lerche | https://github.com/tokio-rs/async-stream | MIT | | Asynchronous streams using async & await notation | | async-trait | 0.1.77 | David Tolnay | https://github.com/dtolnay/async-trait | Apache-2.0 OR MIT | | Type erasure for async trait methods | +| aws-config | 1.1.8 | AWS Rust SDK Team |Russell Cohen | https://github.com/smithy-lang/smithy-rs | Apache-2.0 | | AWS SDK config and credential provider implementations. | +| aws-types | 1.1.8 | AWS Rust SDK Team |Russell Cohen | https://github.com/smithy-lang/smithy-rs | Apache-2.0 | | Cross-service types for the AWS SDK. | | axum | 0.6.20 | | https://github.com/tokio-rs/axum | MIT | | Web framework that focuses on ergonomics and modularity | | axum | 0.7.4 | | https://github.com/tokio-rs/axum | MIT | | Web framework that focuses on ergonomics and modularity | +| base64 | 0.13.1 | Alice Maz |Marshall Pierce | https://github.com/marshallpierce/rust-base64 | Apache-2.0 OR MIT | | encodes and decodes base64 as bytes or utf8 | | base64 | 0.21.7 | Alice Maz |Marshall Pierce | https://github.com/marshallpierce/rust-base64 | Apache-2.0 OR MIT | | encodes and decodes base64 as bytes or utf8 | | base64 | 0.22.0 | Alice Maz |Marshall Pierce | https://github.com/marshallpierce/rust-base64 | Apache-2.0 OR MIT | | encodes and decodes base64 as bytes or utf8 | | bb8 | 0.7.1 | Dirkjan Ochtman |Kyle Huey | https://github.com/djc/bb8 | MIT | | Full-featured async (tokio-based) connection pool (like r2d2) | @@ -95,53 +97,63 @@ gopkg.in/yaml.v3, https://github.com/go-yaml/yaml/blob/v3.0.1/LICENSE, MIT | bigdecimal | 0.3.1 | Andrew Kubera | https://github.com/akubera/bigdecimal-rs | Apache-2.0 OR MIT | | Arbitrary precision decimal numbers | | bigdecimal | 0.4.3 | Andrew Kubera | https://github.com/akubera/bigdecimal-rs | Apache-2.0 OR MIT | | Arbitrary precision decimal numbers | | bytes | 1.5.0 | Carl Lerche |Sean McArthur | https://github.com/tokio-rs/bytes | MIT | | Types and traits for working with bytes | -| clap | 4.5.1 | | https://github.com/clap-rs/clap | Apache-2.0 OR MIT | | A simple to use, efficient, and full-featured Command Line Argument Parser | -| datafusion | 34.0.0 | Apache Arrow | https://github.com/apache/arrow-datafusion | Apache-2.0 | | DataFusion is an in-memory query engine that uses Apache Arrow as the memory model | +| clap | 4.5.2 | | https://github.com/clap-rs/clap | Apache-2.0 OR MIT | | A simple to use, efficient, and full-featured Command Line Argument Parser | +| csv | 1.3.0 | Andrew Gallant | https://github.com/BurntSushi/rust-csv | MIT OR Unlicense | | Fast CSV parsing with support for serde. | +| datafusion | 35.0.0 | Apache Arrow | https://github.com/apache/arrow-datafusion | Apache-2.0 | | DataFusion is an in-memory query engine that uses Apache Arrow as the memory model | +| deltalake | 0.17.0 | Qingping Hou | https://github.com/delta-io/delta.rs | Apache-2.0 | | Native Delta Lake implementation in Rust | | dirs | 5.0.1 | Simon Ochsenreither | https://github.com/soc/dirs-rs | Apache-2.0 OR MIT | | A tiny low-level library that provides platform-specific standard locations of directories for config, cache and other data on Linux, Windows, macOS and Redox by leveraging the mechanisms defined by the XDG base/user directory specifications on Linux, the Known Folder API on Windows, and the Standard Directory guidelines on macOS. | -| duckdb | 0.9.2 | wangfenjin | https://github.com/wangfenjin/duckdb-rs | MIT | | Ergonomic wrapper for DuckDB | -| flight_client | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | -| flight_datafusion | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | -| flightpublisher | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS Flight Publisher | -| flightrepl | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS Flight Query Repl | -| flightsubscriber | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS Flight Subscriber | +| duckdb | 0.10.0 | wangfenjin | https://github.com/wangfenjin/duckdb-rs | MIT | | Ergonomic wrapper for DuckDB | | fundu | 2.0.0 | Joining7943 | https://github.com/fundu-rs/fundu | MIT | | Configurable, precise and fast rust string parser to a Duration | | futures | 0.3.30 | | https://github.com/rust-lang/futures-rs | Apache-2.0 OR MIT | | An implementation of futures and streams featuring zero allocations, composability, and iterator-like interfaces. | | futures-core | 0.3.30 | | https://github.com/rust-lang/futures-rs | Apache-2.0 OR MIT | | The core traits and types in for the `futures` library. | | indexmap | 1.9.3 | | https://github.com/bluss/indexmap | Apache-2.0 OR MIT | | A hash table with consistent order and fast iteration. | -| indexmap | 2.2.3 | | https://github.com/indexmap-rs/indexmap | Apache-2.0 OR MIT | | A hash table with consistent order and fast iteration. | +| indexmap | 2.2.5 | | https://github.com/indexmap-rs/indexmap | Apache-2.0 OR MIT | | A hash table with consistent order and fast iteration. | +| itertools | 0.10.5 | bluss | https://github.com/rust-itertools/itertools | Apache-2.0 OR MIT | | Extra iterator adaptors, iterator methods, free functions, and macros. | +| itertools | 0.11.0 | bluss | https://github.com/rust-itertools/itertools | Apache-2.0 OR MIT | | Extra iterator adaptors, iterator methods, free functions, and macros. | +| itertools | 0.12.1 | bluss | https://github.com/rust-itertools/itertools | Apache-2.0 OR MIT | | Extra iterator adaptors, iterator methods, free functions, and macros. | +| keyring | 2.3.2 | Walther Chen |Daniel Brotsky | https://github.com/hwchen/keyring-rs.git | Apache-2.0 OR MIT | | Cross-platform library for managing passwords/credentials | | metrics | 0.22.1 | Toby Lawrence | https://github.com/metrics-rs/metrics | MIT | | A lightweight metrics facade. | | metrics-exporter-prometheus | 0.13.1 | Toby Lawrence | https://github.com/metrics-rs/metrics | MIT | | A metrics-compatible exporter for sending metrics to Prometheus. | -| ndarray | 0.15.6 | "Ulrik Sverdrup ""bluss""|Jim Turner" | https://github.com/rust-ndarray/ndarray | Apache-2.0 OR MIT | | An n-dimensional array for general elements and for numerics. Lightweight array views and slicing; views support chunking and splitting. | +| ndarray | 0.15.6 | Ulrik Sverdrup bluss|Jim Turner | https://github.com/rust-ndarray/ndarray | Apache-2.0 OR MIT | | An n-dimensional array for general elements and for numerics. Lightweight array views and slicing; views support chunking and splitting. | | ndarray-npy | 0.8.1 | Jim Turner | https://github.com/jturner314/ndarray-npy | Apache-2.0 OR MIT | | .npy and .npz file format support for ndarray | | notify | 6.1.1 | Félix Saparelli |Daniel Faust |Aron Heinecke | https://github.com/notify-rs/notify.git | CC0-1.0 | | Cross-platform filesystem notification library | +| object_store | 0.9.1 | | https://github.com/apache/arrow-rs/tree/master/object_store | Apache-2.0 OR MIT | | A generic object store interface for uniformly interacting with AWS S3, Google Cloud Storage, Azure Blob Storage and local files. | +| once_cell | 1.19.0 | Aleksey Kladov | https://github.com/matklad/once_cell | Apache-2.0 OR MIT | | Single assignment cells and lazy values. | | opentelemetry-proto | 0.4.0 | | https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-proto | Apache-2.0 | | Protobuf generated files and transformations. | | parquet | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Parquet implementation in Rust | -| postgres | 0.19.7 | Steven Fackler | https://github.com/sfackler/rust-postgres | Apache-2.0 OR MIT | | A native, synchronous PostgreSQL client | +| parquet | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Parquet implementation in Rust | +| prost | 0.11.9 | Dan Burkert |Lucio Franco | https://github.com/tokio-rs/prost | Apache-2.0 | | A Protocol Buffers implementation for the Rust Language. | +| prost | 0.12.3 | Dan Burkert |Lucio Franco | https://github.com/tokio-rs/prost | Apache-2.0 | | A Protocol Buffers implementation for the Rust Language. | +| quick-xml | 0.23.1 | | https://github.com/tafia/quick-xml | MIT | | High performance xml reader and writer | +| quick-xml | 0.31.0 | | https://github.com/tafia/quick-xml | MIT | | High performance xml reader and writer | | r2d2 | 0.8.10 | Steven Fackler | https://github.com/sfackler/r2d2 | Apache-2.0 OR MIT | | A generic connection pool | | regex | 1.10.3 | The Rust Project Developers|Andrew Gallant | https://github.com/rust-lang/regex | Apache-2.0 OR MIT | | An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs. | -| reqwest | 0.11.24 | Sean McArthur | https://github.com/seanmonstar/reqwest | Apache-2.0 OR MIT | | higher level HTTP client library | -| runtime | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | The spiced runtime | +| reqwest | 0.11.26 | Sean McArthur | https://github.com/seanmonstar/reqwest | Apache-2.0 OR MIT | | higher level HTTP client library | +| rustls | 0.19.1 | Joseph Birr-Pixton | https://github.com/ctz/rustls | Apache-2.0 OR ISC OR MIT | | Rustls is a modern TLS library written in Rust. | | rustls | 0.21.10 | | https://github.com/rustls/rustls | Apache-2.0 OR ISC OR MIT | | Rustls is a modern TLS library written in Rust. | +| rustls-native-certs | 0.5.0 | Joseph Birr-Pixton | https://github.com/ctz/rustls-native-certs | Apache-2.0 OR ISC OR MIT | | rustls-native-certs allows rustls to use the platform native certificate store | | rustls-native-certs | 0.6.3 | | https://github.com/ctz/rustls-native-certs | Apache-2.0 OR ISC OR MIT | | rustls-native-certs allows rustls to use the platform native certificate store | | rustls-pemfile | 1.0.4 | | https://github.com/rustls/pemfile | Apache-2.0 OR ISC OR MIT | | Basic .pem file parser for keys and certificates | | rustyline | 13.0.0 | Katsu Kawakami | https://github.com/kkawakam/rustyline | MIT | | Rustyline, a readline implementation based on Antirez's Linenoise | | sea-query | 0.30.7 | Chris Tsang |Billy Chan |Ivan Krivosheev | https://github.com/SeaQL/sea-query | Apache-2.0 OR MIT | | 🔱 A dynamic query builder for MySQL, Postgres and SQLite | -| serde | 1.0.196 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/serde | Apache-2.0 OR MIT | | A generic serialization/deserialization framework | -| serde_json | 1.0.113 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/json | Apache-2.0 OR MIT | | A JSON serialization file format | -| serde_yaml | 0.9.31 | David Tolnay | https://github.com/dtolnay/serde-yaml | Apache-2.0 OR MIT | | YAML data format for Serde | +| secrecy | 0.8.0 | Tony Arcieri | https://github.com/iqlusioninc/crates/tree/main/secrecy | Apache-2.0 OR MIT | | Wrapper types and traits for secret management which help ensure they aren't accidentally copied, logged, or otherwise exposed (as much as possible), and also ensure secrets are securely wiped from memory when dropped. | +| serde | 1.0.197 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/serde | Apache-2.0 OR MIT | | A generic serialization/deserialization framework | +| serde_json | 1.0.114 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/json | Apache-2.0 OR MIT | | A JSON serialization file format | +| serde_yaml | 0.9.32 | David Tolnay | https://github.com/dtolnay/serde-yaml | Apache-2.0 OR MIT | | YAML data format for Serde | | snafu | 0.7.5 | Jake Goulding | https://github.com/shepmaster/snafu | Apache-2.0 OR MIT | | An ergonomic error handling library | -| snafu | 0.8.0 | Jake Goulding | https://github.com/shepmaster/snafu | Apache-2.0 OR MIT | | An ergonomic error handling library | -| spiced | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS | -| spicepod | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | -| sql_provider_datafusion | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | +| snafu | 0.8.2 | Jake Goulding | https://github.com/shepmaster/snafu | Apache-2.0 OR MIT | | An ergonomic error handling library | | time | 0.3.34 | Jacob Pratt |Time contributors | https://github.com/time-rs/time | Apache-2.0 OR MIT | | Date and time library. Fully interoperable with the standard library. Mostly compatible with #![no_std]. | | tokio | 1.36.0 | Tokio Contributors | https://github.com/tokio-rs/tokio | MIT | | An event-driven, non-blocking I/O platform for writing asynchronous I/O backed applications. | +| tokio-postgres | 0.7.10 | Steven Fackler | https://github.com/sfackler/rust-postgres | Apache-2.0 OR MIT | | A native, asynchronous PostgreSQL client | | tokio-stream | 0.1.14 | Tokio Contributors | https://github.com/tokio-rs/tokio | MIT | | Utilities to work with `Stream` and `tokio`. | -| toml | 0.8.10 | Alex Crichton | https://github.com/toml-rs/toml | Apache-2.0 OR MIT | | A native Rust encoder and decoder of TOML-formatted files and streams. Provides implementations of the standard Serialize/Deserialize traits for TOML data to facilitate deserializing and serializing Rust structures. | +| toml | 0.8.11 | Alex Crichton | https://github.com/toml-rs/toml | Apache-2.0 OR MIT | | A native Rust encoder and decoder of TOML-formatted files and streams. Provides implementations of the standard Serialize/Deserialize traits for TOML data to facilitate deserializing and serializing Rust structures. | | tonic | 0.9.2 | Lucio Franco | https://github.com/hyperium/tonic | MIT | | A gRPC over HTTP/2 implementation focused on high performance, interoperability, and flexibility. | | tonic | 0.10.2 | Lucio Franco | https://github.com/hyperium/tonic | MIT | | A gRPC over HTTP/2 implementation focused on high performance, interoperability, and flexibility. | +| tonic | 0.11.0 | Lucio Franco | https://github.com/hyperium/tonic | MIT | | A gRPC over HTTP/2 implementation focused on high performance, interoperability, and flexibility. | +| tonic-health | 0.11.0 | James Nugent | https://github.com/hyperium/tonic | MIT | | Health Checking module of `tonic` gRPC implementation. | | tracing | 0.1.40 | Eliza Weisman |Tokio Contributors | https://github.com/tokio-rs/tracing | MIT | | Application-level tracing for Rust. | | tracing-subscriber | 0.3.18 | Eliza Weisman |David Barsky |Tokio Contributors | https://github.com/tokio-rs/tracing | MIT | | Utilities for implementing and composing `tracing` subscribers. | | tract-core | 0.21.1 | Mathieu Poumeyrol | https://github.com/snipsco/tract | Apache-2.0 OR MIT | | Tiny, no-nonsense, self contained, TensorFlow and ONNX inference | | tract-onnx | 0.21.1 | Mathieu Poumeyrol | https://github.com/snipsco/tract | Apache-2.0 OR MIT | | Tiny, no-nonsense, self contained, TensorFlow and ONNX inference | -| uuid | 1.7.0 | Ashley Mannix|Christopher Armstrong|Dylan DPC|Hunar Roop Kahlon | https://github.com/uuid-rs/uuid | Apache-2.0 OR MIT | | A library to generate and parse UUIDs. | \ No newline at end of file +| url | 2.5.0 | The rust-url developers | https://github.com/servo/rust-url | Apache-2.0 OR MIT | | URL library for Rust, based on the WHATWG URL Standard | +| uuid | 1.7.0 | Ashley Mannix|Christopher Armstrong|Dylan DPC|Hunar Roop Kahlon | https://github.com/uuid-rs/uuid | Apache-2.0 OR MIT | | A library to generate and parse UUIDs. | From d48ce5b911c927b877df7e5f9237324f772fa35a Mon Sep 17 00:00:00 2001 From: jeadie Date: Fri, 15 Mar 2024 12:29:32 +1000 Subject: [PATCH 2/3] update S3 connector docs --- .../content/en/reference/Connectors/s3.md | 58 +++++++++---------- 1 file changed, 28 insertions(+), 30 deletions(-) diff --git a/spiceaidocs/content/en/reference/Connectors/s3.md b/spiceaidocs/content/en/reference/Connectors/s3.md index c9280cfba..2e3408ba3 100644 --- a/spiceaidocs/content/en/reference/Connectors/s3.md +++ b/spiceaidocs/content/en/reference/Connectors/s3.md @@ -1,44 +1,42 @@ --- type: docs -title: "S3 Parquet Dataset" -linkTitle: "S3 Parquet Dataset" -description: 'S3 Parquet Dataset YAML reference' +title: "S3 Data Connector" +linkTitle: "S3 Data Connector" +description: 'S3 Data Connector YAML reference' weight: 80 --- -Parquet data in S3-compatible storage can be queried in mesh mode or accelerated locally. - -# `dataset` - -```yaml -- from: s3://s3-bucket-name/path/to/parquet/file.parquet - name: s3_file - auth: s3 - params: - endpoint: https://your.minio.host - region: "" - acceleration: - enabled: true - refresh_mode: full -``` +S3 as a connector for federated SQL query across Parquet files stored in S3, or S3-compatible storage solutions (e.g. Minio, Cloudflare R2). ## `params` -- `endpoint`: Check `with_endpoint` in [object store](https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html) -- `region`: Check `with_region` in [object store](https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html) - -## `acceleration` - -Optional. - -When `null`, query will run in mesh mode; parquet data will be read directly from the object store for each query. - -Only `full` refresh mode is supported. +- `endpoint`: The S3 endpoint, or equivalent (e.g. Minio endpoint), for the S3-compatible storage. +- `region`: Region of the S3 bucket, if region specific. ## `auth` Check [Secrets]({{}}). Required attribbutes: -- `key` -- `secret` +- `key`: The access key authorised to access the S3 data (e.g. `AWS_ACCESS_KEY_ID` for AWS) +- `secret`The secret key authorised to access the S3 data (e.g. `AWS_SECRET_ACCESS_KEY` for AWS) + + +## Example +### Minio +```yaml +- from: s3://s3-bucket-name/path/to/parquet/cool_dataset.parquet + name: cool_dataset + params: + endpoint: https://my.minio.server + region: "us-east-1" # Best practice for Minio +``` + +#### S3 +```yaml +- from: s3://my-startups-data/path/to/parquet/cool_dataset.parquet + name: cool_dataset + params: + endpoint: http://my-startups-data.s3.amazonaws.com + region: "ap-southeast-2" +``` \ No newline at end of file From d04185efed46f70b8503aa3f526b8bc5a11e741d Mon Sep 17 00:00:00 2001 From: jeadie Date: Fri, 15 Mar 2024 12:30:25 +1000 Subject: [PATCH 3/3] Revert "Update acknowledgments (#109)" This reverts commit 8b2c3e1cd8f01f222d36ca0091354cea89bea5b2. --- .../content/en/acknowledgements/_index.md | 66 ++++++++----------- 1 file changed, 27 insertions(+), 39 deletions(-) diff --git a/spiceaidocs/content/en/acknowledgements/_index.md b/spiceaidocs/content/en/acknowledgements/_index.md index 64c7d153b..7b3203a4c 100644 --- a/spiceaidocs/content/en/acknowledgements/_index.md +++ b/spiceaidocs/content/en/acknowledgements/_index.md @@ -78,18 +78,16 @@ gopkg.in/yaml.v3, https://github.com/go-yaml/yaml/blob/v3.0.1/LICENSE, MIT | name | version | authors | repository | license | license_file | description | |------|---------|---------|------------|---------|--------------|-------------| | name | version | authors | repository | license | license_file | description | -| arrow | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Rust implementation of Apache Arrow | -| arrow-flight | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Arrow Flight | +| app | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | All loaded components from the root Spicepod and its dependencies | +| arrow | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Rust implementation of Apache Arrow | +| arrow-flight | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Arrow Flight | | arrow-ipc | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for the Arrow IPC format | -| arrow-ipc | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for the Arrow IPC format | -| arrow-json | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for parsing JSON format to and from the Arrow format | +| arrow-json | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Support for parsing JSON format into the Arrow format | +| arrow_sql_gen | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | | async-stream | 0.3.5 | Carl Lerche | https://github.com/tokio-rs/async-stream | MIT | | Asynchronous streams using async & await notation | | async-trait | 0.1.77 | David Tolnay | https://github.com/dtolnay/async-trait | Apache-2.0 OR MIT | | Type erasure for async trait methods | -| aws-config | 1.1.8 | AWS Rust SDK Team |Russell Cohen | https://github.com/smithy-lang/smithy-rs | Apache-2.0 | | AWS SDK config and credential provider implementations. | -| aws-types | 1.1.8 | AWS Rust SDK Team |Russell Cohen | https://github.com/smithy-lang/smithy-rs | Apache-2.0 | | Cross-service types for the AWS SDK. | | axum | 0.6.20 | | https://github.com/tokio-rs/axum | MIT | | Web framework that focuses on ergonomics and modularity | | axum | 0.7.4 | | https://github.com/tokio-rs/axum | MIT | | Web framework that focuses on ergonomics and modularity | -| base64 | 0.13.1 | Alice Maz |Marshall Pierce | https://github.com/marshallpierce/rust-base64 | Apache-2.0 OR MIT | | encodes and decodes base64 as bytes or utf8 | | base64 | 0.21.7 | Alice Maz |Marshall Pierce | https://github.com/marshallpierce/rust-base64 | Apache-2.0 OR MIT | | encodes and decodes base64 as bytes or utf8 | | base64 | 0.22.0 | Alice Maz |Marshall Pierce | https://github.com/marshallpierce/rust-base64 | Apache-2.0 OR MIT | | encodes and decodes base64 as bytes or utf8 | | bb8 | 0.7.1 | Dirkjan Ochtman |Kyle Huey | https://github.com/djc/bb8 | MIT | | Full-featured async (tokio-based) connection pool (like r2d2) | @@ -97,63 +95,53 @@ gopkg.in/yaml.v3, https://github.com/go-yaml/yaml/blob/v3.0.1/LICENSE, MIT | bigdecimal | 0.3.1 | Andrew Kubera | https://github.com/akubera/bigdecimal-rs | Apache-2.0 OR MIT | | Arbitrary precision decimal numbers | | bigdecimal | 0.4.3 | Andrew Kubera | https://github.com/akubera/bigdecimal-rs | Apache-2.0 OR MIT | | Arbitrary precision decimal numbers | | bytes | 1.5.0 | Carl Lerche |Sean McArthur | https://github.com/tokio-rs/bytes | MIT | | Types and traits for working with bytes | -| clap | 4.5.2 | | https://github.com/clap-rs/clap | Apache-2.0 OR MIT | | A simple to use, efficient, and full-featured Command Line Argument Parser | -| csv | 1.3.0 | Andrew Gallant | https://github.com/BurntSushi/rust-csv | MIT OR Unlicense | | Fast CSV parsing with support for serde. | -| datafusion | 35.0.0 | Apache Arrow | https://github.com/apache/arrow-datafusion | Apache-2.0 | | DataFusion is an in-memory query engine that uses Apache Arrow as the memory model | -| deltalake | 0.17.0 | Qingping Hou | https://github.com/delta-io/delta.rs | Apache-2.0 | | Native Delta Lake implementation in Rust | +| clap | 4.5.1 | | https://github.com/clap-rs/clap | Apache-2.0 OR MIT | | A simple to use, efficient, and full-featured Command Line Argument Parser | +| datafusion | 34.0.0 | Apache Arrow | https://github.com/apache/arrow-datafusion | Apache-2.0 | | DataFusion is an in-memory query engine that uses Apache Arrow as the memory model | | dirs | 5.0.1 | Simon Ochsenreither | https://github.com/soc/dirs-rs | Apache-2.0 OR MIT | | A tiny low-level library that provides platform-specific standard locations of directories for config, cache and other data on Linux, Windows, macOS and Redox by leveraging the mechanisms defined by the XDG base/user directory specifications on Linux, the Known Folder API on Windows, and the Standard Directory guidelines on macOS. | -| duckdb | 0.10.0 | wangfenjin | https://github.com/wangfenjin/duckdb-rs | MIT | | Ergonomic wrapper for DuckDB | +| duckdb | 0.9.2 | wangfenjin | https://github.com/wangfenjin/duckdb-rs | MIT | | Ergonomic wrapper for DuckDB | +| flight_client | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | +| flight_datafusion | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | +| flightpublisher | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS Flight Publisher | +| flightrepl | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS Flight Query Repl | +| flightsubscriber | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS Flight Subscriber | | fundu | 2.0.0 | Joining7943 | https://github.com/fundu-rs/fundu | MIT | | Configurable, precise and fast rust string parser to a Duration | | futures | 0.3.30 | | https://github.com/rust-lang/futures-rs | Apache-2.0 OR MIT | | An implementation of futures and streams featuring zero allocations, composability, and iterator-like interfaces. | | futures-core | 0.3.30 | | https://github.com/rust-lang/futures-rs | Apache-2.0 OR MIT | | The core traits and types in for the `futures` library. | | indexmap | 1.9.3 | | https://github.com/bluss/indexmap | Apache-2.0 OR MIT | | A hash table with consistent order and fast iteration. | -| indexmap | 2.2.5 | | https://github.com/indexmap-rs/indexmap | Apache-2.0 OR MIT | | A hash table with consistent order and fast iteration. | -| itertools | 0.10.5 | bluss | https://github.com/rust-itertools/itertools | Apache-2.0 OR MIT | | Extra iterator adaptors, iterator methods, free functions, and macros. | -| itertools | 0.11.0 | bluss | https://github.com/rust-itertools/itertools | Apache-2.0 OR MIT | | Extra iterator adaptors, iterator methods, free functions, and macros. | -| itertools | 0.12.1 | bluss | https://github.com/rust-itertools/itertools | Apache-2.0 OR MIT | | Extra iterator adaptors, iterator methods, free functions, and macros. | -| keyring | 2.3.2 | Walther Chen |Daniel Brotsky | https://github.com/hwchen/keyring-rs.git | Apache-2.0 OR MIT | | Cross-platform library for managing passwords/credentials | +| indexmap | 2.2.3 | | https://github.com/indexmap-rs/indexmap | Apache-2.0 OR MIT | | A hash table with consistent order and fast iteration. | | metrics | 0.22.1 | Toby Lawrence | https://github.com/metrics-rs/metrics | MIT | | A lightweight metrics facade. | | metrics-exporter-prometheus | 0.13.1 | Toby Lawrence | https://github.com/metrics-rs/metrics | MIT | | A metrics-compatible exporter for sending metrics to Prometheus. | -| ndarray | 0.15.6 | Ulrik Sverdrup bluss|Jim Turner | https://github.com/rust-ndarray/ndarray | Apache-2.0 OR MIT | | An n-dimensional array for general elements and for numerics. Lightweight array views and slicing; views support chunking and splitting. | +| ndarray | 0.15.6 | "Ulrik Sverdrup ""bluss""|Jim Turner" | https://github.com/rust-ndarray/ndarray | Apache-2.0 OR MIT | | An n-dimensional array for general elements and for numerics. Lightweight array views and slicing; views support chunking and splitting. | | ndarray-npy | 0.8.1 | Jim Turner | https://github.com/jturner314/ndarray-npy | Apache-2.0 OR MIT | | .npy and .npz file format support for ndarray | | notify | 6.1.1 | Félix Saparelli |Daniel Faust |Aron Heinecke | https://github.com/notify-rs/notify.git | CC0-1.0 | | Cross-platform filesystem notification library | -| object_store | 0.9.1 | | https://github.com/apache/arrow-rs/tree/master/object_store | Apache-2.0 OR MIT | | A generic object store interface for uniformly interacting with AWS S3, Google Cloud Storage, Azure Blob Storage and local files. | -| once_cell | 1.19.0 | Aleksey Kladov | https://github.com/matklad/once_cell | Apache-2.0 OR MIT | | Single assignment cells and lazy values. | | opentelemetry-proto | 0.4.0 | | https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-proto | Apache-2.0 | | Protobuf generated files and transformations. | | parquet | 49.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Parquet implementation in Rust | -| parquet | 50.0.0 | Apache Arrow | https://github.com/apache/arrow-rs | Apache-2.0 | | Apache Parquet implementation in Rust | -| prost | 0.11.9 | Dan Burkert |Lucio Franco | https://github.com/tokio-rs/prost | Apache-2.0 | | A Protocol Buffers implementation for the Rust Language. | -| prost | 0.12.3 | Dan Burkert |Lucio Franco | https://github.com/tokio-rs/prost | Apache-2.0 | | A Protocol Buffers implementation for the Rust Language. | -| quick-xml | 0.23.1 | | https://github.com/tafia/quick-xml | MIT | | High performance xml reader and writer | -| quick-xml | 0.31.0 | | https://github.com/tafia/quick-xml | MIT | | High performance xml reader and writer | +| postgres | 0.19.7 | Steven Fackler | https://github.com/sfackler/rust-postgres | Apache-2.0 OR MIT | | A native, synchronous PostgreSQL client | | r2d2 | 0.8.10 | Steven Fackler | https://github.com/sfackler/r2d2 | Apache-2.0 OR MIT | | A generic connection pool | | regex | 1.10.3 | The Rust Project Developers|Andrew Gallant | https://github.com/rust-lang/regex | Apache-2.0 OR MIT | | An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs. | -| reqwest | 0.11.26 | Sean McArthur | https://github.com/seanmonstar/reqwest | Apache-2.0 OR MIT | | higher level HTTP client library | -| rustls | 0.19.1 | Joseph Birr-Pixton | https://github.com/ctz/rustls | Apache-2.0 OR ISC OR MIT | | Rustls is a modern TLS library written in Rust. | +| reqwest | 0.11.24 | Sean McArthur | https://github.com/seanmonstar/reqwest | Apache-2.0 OR MIT | | higher level HTTP client library | +| runtime | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | The spiced runtime | | rustls | 0.21.10 | | https://github.com/rustls/rustls | Apache-2.0 OR ISC OR MIT | | Rustls is a modern TLS library written in Rust. | -| rustls-native-certs | 0.5.0 | Joseph Birr-Pixton | https://github.com/ctz/rustls-native-certs | Apache-2.0 OR ISC OR MIT | | rustls-native-certs allows rustls to use the platform native certificate store | | rustls-native-certs | 0.6.3 | | https://github.com/ctz/rustls-native-certs | Apache-2.0 OR ISC OR MIT | | rustls-native-certs allows rustls to use the platform native certificate store | | rustls-pemfile | 1.0.4 | | https://github.com/rustls/pemfile | Apache-2.0 OR ISC OR MIT | | Basic .pem file parser for keys and certificates | | rustyline | 13.0.0 | Katsu Kawakami | https://github.com/kkawakam/rustyline | MIT | | Rustyline, a readline implementation based on Antirez's Linenoise | | sea-query | 0.30.7 | Chris Tsang |Billy Chan |Ivan Krivosheev | https://github.com/SeaQL/sea-query | Apache-2.0 OR MIT | | 🔱 A dynamic query builder for MySQL, Postgres and SQLite | -| secrecy | 0.8.0 | Tony Arcieri | https://github.com/iqlusioninc/crates/tree/main/secrecy | Apache-2.0 OR MIT | | Wrapper types and traits for secret management which help ensure they aren't accidentally copied, logged, or otherwise exposed (as much as possible), and also ensure secrets are securely wiped from memory when dropped. | -| serde | 1.0.197 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/serde | Apache-2.0 OR MIT | | A generic serialization/deserialization framework | -| serde_json | 1.0.114 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/json | Apache-2.0 OR MIT | | A JSON serialization file format | -| serde_yaml | 0.9.32 | David Tolnay | https://github.com/dtolnay/serde-yaml | Apache-2.0 OR MIT | | YAML data format for Serde | +| serde | 1.0.196 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/serde | Apache-2.0 OR MIT | | A generic serialization/deserialization framework | +| serde_json | 1.0.113 | Erick Tryzelaar |David Tolnay | https://github.com/serde-rs/json | Apache-2.0 OR MIT | | A JSON serialization file format | +| serde_yaml | 0.9.31 | David Tolnay | https://github.com/dtolnay/serde-yaml | Apache-2.0 OR MIT | | YAML data format for Serde | | snafu | 0.7.5 | Jake Goulding | https://github.com/shepmaster/snafu | Apache-2.0 OR MIT | | An ergonomic error handling library | -| snafu | 0.8.2 | Jake Goulding | https://github.com/shepmaster/snafu | Apache-2.0 OR MIT | | An ergonomic error handling library | +| snafu | 0.8.0 | Jake Goulding | https://github.com/shepmaster/snafu | Apache-2.0 OR MIT | | An ergonomic error handling library | +| spiced | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | Spice OSS | +| spicepod | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | +| sql_provider_datafusion | 0.7.0-alpha | | https://github.com/spiceai/spiceai | Apache-2.0 | | | | time | 0.3.34 | Jacob Pratt |Time contributors | https://github.com/time-rs/time | Apache-2.0 OR MIT | | Date and time library. Fully interoperable with the standard library. Mostly compatible with #![no_std]. | | tokio | 1.36.0 | Tokio Contributors | https://github.com/tokio-rs/tokio | MIT | | An event-driven, non-blocking I/O platform for writing asynchronous I/O backed applications. | -| tokio-postgres | 0.7.10 | Steven Fackler | https://github.com/sfackler/rust-postgres | Apache-2.0 OR MIT | | A native, asynchronous PostgreSQL client | | tokio-stream | 0.1.14 | Tokio Contributors | https://github.com/tokio-rs/tokio | MIT | | Utilities to work with `Stream` and `tokio`. | -| toml | 0.8.11 | Alex Crichton | https://github.com/toml-rs/toml | Apache-2.0 OR MIT | | A native Rust encoder and decoder of TOML-formatted files and streams. Provides implementations of the standard Serialize/Deserialize traits for TOML data to facilitate deserializing and serializing Rust structures. | +| toml | 0.8.10 | Alex Crichton | https://github.com/toml-rs/toml | Apache-2.0 OR MIT | | A native Rust encoder and decoder of TOML-formatted files and streams. Provides implementations of the standard Serialize/Deserialize traits for TOML data to facilitate deserializing and serializing Rust structures. | | tonic | 0.9.2 | Lucio Franco | https://github.com/hyperium/tonic | MIT | | A gRPC over HTTP/2 implementation focused on high performance, interoperability, and flexibility. | | tonic | 0.10.2 | Lucio Franco | https://github.com/hyperium/tonic | MIT | | A gRPC over HTTP/2 implementation focused on high performance, interoperability, and flexibility. | -| tonic | 0.11.0 | Lucio Franco | https://github.com/hyperium/tonic | MIT | | A gRPC over HTTP/2 implementation focused on high performance, interoperability, and flexibility. | -| tonic-health | 0.11.0 | James Nugent | https://github.com/hyperium/tonic | MIT | | Health Checking module of `tonic` gRPC implementation. | | tracing | 0.1.40 | Eliza Weisman |Tokio Contributors | https://github.com/tokio-rs/tracing | MIT | | Application-level tracing for Rust. | | tracing-subscriber | 0.3.18 | Eliza Weisman |David Barsky |Tokio Contributors | https://github.com/tokio-rs/tracing | MIT | | Utilities for implementing and composing `tracing` subscribers. | | tract-core | 0.21.1 | Mathieu Poumeyrol | https://github.com/snipsco/tract | Apache-2.0 OR MIT | | Tiny, no-nonsense, self contained, TensorFlow and ONNX inference | | tract-onnx | 0.21.1 | Mathieu Poumeyrol | https://github.com/snipsco/tract | Apache-2.0 OR MIT | | Tiny, no-nonsense, self contained, TensorFlow and ONNX inference | -| url | 2.5.0 | The rust-url developers | https://github.com/servo/rust-url | Apache-2.0 OR MIT | | URL library for Rust, based on the WHATWG URL Standard | -| uuid | 1.7.0 | Ashley Mannix|Christopher Armstrong|Dylan DPC|Hunar Roop Kahlon | https://github.com/uuid-rs/uuid | Apache-2.0 OR MIT | | A library to generate and parse UUIDs. | +| uuid | 1.7.0 | Ashley Mannix|Christopher Armstrong|Dylan DPC|Hunar Roop Kahlon | https://github.com/uuid-rs/uuid | Apache-2.0 OR MIT | | A library to generate and parse UUIDs. | \ No newline at end of file