-
-
Notifications
You must be signed in to change notification settings - Fork 528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add proxy connection type #1881
Merged
Merged
Changes from 22 commits
Commits
Show all changes
54 commits
Select commit
Hold shift + click to select a range
dc14d28
feat: Add proxy connection type
langyo eda0bff
feat: Add proxy database's proxy functions trait.
langyo 88f4177
fix: Remove some unused impl to fix the unit test
langyo 0ff1952
test: Create the proxy by empty declaration.
langyo 5e2e3fa
test: Try to genereate query and exec commands.
langyo ef357ff
perf: Add more query debug trait for debugging.
langyo 0d67fe2
chore: Add the example for wasi + proxy.
langyo 32c3813
chore: Try to read string from wasmtime vm.
langyo f35725a
chore: Sucks, but how to do without tokio::spawn?
langyo bbcd987
chore: Complete the basic memory read logic.
langyo 460c978
chore: Abandon the WASI demo, native demo first...
langyo c354c83
refactor: Use single proxy connection generator
langyo ac8816f
refactor: Rename the inner structs' name
langyo 1ab8a55
fix: Fix CI clippy and unit test
langyo ffff54f
Merge commit 'bd9a0e9b7e8f55d4ed91511f4b088849b94d3087' into proxy-conn
langyo c54ed1e
fix: Rename the example.
langyo bf91484
chore: Try to embed surrealdb for proxy test.
langyo d593d7b
fix: Transfer the query result correctly.
langyo a3d6a0d
refactor: Rename the example.
langyo 9638926
chore: Ready to add example for wasmtime proxy.
langyo 4e51c86
feat: Try to compile sea-orm into wasm binary.
langyo f818e66
fix: WASM targets can't use sqlx.
langyo 82b0075
fix: Try to fix CI by remove toml.
langyo 05416c7
fix: Try to fix CI by remove toml.
langyo 9803092
Merge branch 'proxy-conn' of https://github.com/langyo/sea-orm into p…
langyo 9a1c594
fix: Move vm to the example's root dir.
langyo b87ce92
fix: Add a pre-build script.
langyo 933f952
chore: Add README.
langyo 5ee2174
Merge commit 'e9acabd847d34f5fe257dba1b0b95647853c8af0' into proxy-conn
langyo 6bae09f
fix: Try to fix CI.
langyo 2fc95b0
feat: Add proxy logic in wasm module.
langyo c2e4c2a
fix: Try to run the wasi module.
langyo fc20fa9
refactor: Bump wasmtime to 14.
langyo f0d3741
fix: Now we can use async traits on wasmtime.
langyo defa6bf
build: Use build.rs instead of dynamic command.
langyo dc2faa4
feat: Add the execute result's transfer logic.
langyo cda01be
fix: Convert sqlx query result for sea-query.
langyo 6b762f3
fix: Now we can transfer wasm's query to outside.
langyo a36dfe6
refactor: Convert to ProxyRow first.
langyo 0898707
Merge commit '42babea318ffed731aa916522208d136174025af' into proxy-conn
langyo 2c90866
fix: Multiple time library reference.
langyo ecbc2f3
feat: Add a new proxy example which uses GlueSQL.
langyo c482b36
test: Add the test cases for three new examples.
langyo b1274dc
ci: Add wasm component's compiler for unit test.
langyo c90d0d0
ci: Add wasi target.
langyo 3282413
ci: It may needs wasi target twice...
langyo 0cb42c2
feat: Add more keys for proxy execute result.
langyo 956dfa8
fix: Use custom id type instead of json value.
langyo 25662ea
fix: Wrong reference type.
langyo a1693cf
fix: Rewrite the transformer.
langyo 9bac6e9
perf: Add ToString trait for proxy exec result.
langyo 151eb1d
revert: Again.
langyo ba09d84
revert: Back to the basic proxy exec result.
langyo de66032
Merge branch 'master' into proxy-conn
langyo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| [package] | ||
| name = "sea-orm-proxy-surrealdb-example" | ||
| version = "0.1.0" | ||
| authors = ["Langyo <[email protected]>"] | ||
| edition = "2021" | ||
| publish = false | ||
|
|
||
| [workspace] | ||
|
|
||
| [dependencies] | ||
| async-std = { version = "1.12", features = ["attributes", "tokio1"] } | ||
| serde_json = { version = "1" } | ||
| serde = { version = "1" } | ||
| futures = { version = "0.3" } | ||
| async-stream = { version = "0.3" } | ||
| futures-util = { version = "0.3" } | ||
|
|
||
| sea-orm = { path = "../../", features = [ | ||
| "sqlx-all", | ||
| "proxy", | ||
| "runtime-async-std-native-tls", | ||
| "debug-print", | ||
| ] } | ||
| surrealdb = { version = "1", features = ["kv-mem"] } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| pub mod post; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| use sea_orm::entity::prelude::*; | ||
| use serde::{Deserialize, Serialize}; | ||
|
|
||
| #[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel, Deserialize, Serialize)] | ||
| #[sea_orm(table_name = "posts")] | ||
| pub struct Model { | ||
| #[sea_orm(primary_key)] | ||
| pub id: String, | ||
|
|
||
| pub title: String, | ||
| pub text: String, | ||
| } | ||
|
|
||
| #[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] | ||
| pub enum Relation {} | ||
|
|
||
| impl ActiveModelBehavior for ActiveModel {} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| //! Proxy connection example. | ||
|
|
||
| #![deny(missing_docs)] | ||
|
|
||
| mod entity; | ||
|
|
||
| use std::{ | ||
| collections::BTreeMap, | ||
| sync::{Arc, Mutex}, | ||
| }; | ||
|
|
||
| use sea_orm::{ | ||
| ActiveValue::Set, Database, DbBackend, DbErr, EntityTrait, ProxyDatabaseTrait, ProxyExecResult, | ||
| ProxyRow, Statement, | ||
| }; | ||
| use surrealdb::{ | ||
| engine::local::{Db, Mem}, | ||
| Surreal, | ||
| }; | ||
|
|
||
| use entity::post::{ActiveModel, Entity}; | ||
|
|
||
| #[derive(Debug)] | ||
| struct ProxyDb { | ||
| mem: Surreal<Db>, | ||
| } | ||
|
|
||
| impl ProxyDatabaseTrait for ProxyDb { | ||
| fn query(&self, statement: Statement) -> Result<Vec<ProxyRow>, DbErr> { | ||
| println!("SQL query: {:?}", statement); | ||
| let sql = statement.sql.clone(); | ||
| let mut ret = async_std::task::block_on(async { | ||
| // Surrealdb's grammar is not compatible with sea-orm's | ||
| // so we need to remove the extra clauses | ||
| // from "SELECT `from`.`col` FROM `from` WHERE `from`.`col` = xx" | ||
| // to "SELECT `col` FROM `from` WHERE `col` = xx" | ||
|
|
||
| // Get the first index of "FROM" | ||
| let from_index = sql.find("FROM").unwrap(); | ||
| // Get the name after "FROM" | ||
| let from_name = sql[from_index + 5..].split(' ').next().unwrap(); | ||
| // Delete the name before all the columns | ||
| let new_sql = sql.replace(&format!("{}.", from_name), ""); | ||
|
|
||
| self.mem.query(new_sql).await | ||
| }) | ||
| .unwrap(); | ||
|
|
||
| // Convert the result to sea-orm's format | ||
| let ret: Vec<serde_json::Value> = ret.take(0).unwrap(); | ||
| println!("SQL query result: {}", serde_json::to_string(&ret).unwrap()); | ||
| let ret = ret | ||
| .iter() | ||
| .map(|row| { | ||
| let mut map = serde_json::Map::new(); | ||
| for (k, v) in row.as_object().unwrap().iter() { | ||
| if k == "id" { | ||
| // Get `tb` and `id` columns from surrealdb | ||
| // and convert them to sea-orm's `id` | ||
| let tb = v.as_object().unwrap().get("tb").unwrap().to_string(); | ||
| let id = v | ||
| .as_object() | ||
| .unwrap() | ||
| .get("id") | ||
| .unwrap() | ||
| .get("String") | ||
| .unwrap(); | ||
|
|
||
| // Remove the quotes | ||
| let tb = tb.to_string().replace("\"", ""); | ||
| let id = id.to_string().replace("\"", ""); | ||
|
|
||
| map.insert("id".to_owned(), format!("{}:{}", tb, id).into()); | ||
| continue; | ||
| } | ||
|
|
||
| map.insert(k.to_owned(), v.to_owned()); | ||
| } | ||
| serde_json::Value::Object(map) | ||
| }) | ||
| .map(|v: serde_json::Value| { | ||
| let mut ret: BTreeMap<String, sea_orm::Value> = BTreeMap::new(); | ||
| for (k, v) in v.as_object().unwrap().iter() { | ||
| ret.insert( | ||
| k.to_owned(), | ||
| match v { | ||
| serde_json::Value::Bool(b) => { | ||
| sea_orm::Value::TinyInt(if *b { Some(1) } else { Some(0) }) | ||
| } | ||
| serde_json::Value::Number(n) => { | ||
| if n.is_i64() { | ||
| sea_orm::Value::BigInt(Some(n.as_i64().unwrap())) | ||
| } else if n.is_u64() { | ||
| sea_orm::Value::BigUnsigned(Some(n.as_u64().unwrap())) | ||
| } else if n.is_f64() { | ||
| sea_orm::Value::Double(Some(n.as_f64().unwrap())) | ||
| } else { | ||
| unreachable!() | ||
| } | ||
| } | ||
| serde_json::Value::String(s) => { | ||
| sea_orm::Value::String(Some(Box::new(s.to_owned()))) | ||
| } | ||
| _ => sea_orm::Value::Json(Some(Box::new(v.to_owned()))), | ||
| }, | ||
| ); | ||
| } | ||
| ProxyRow { values: ret } | ||
| }) | ||
| .collect::<Vec<_>>(); | ||
|
|
||
| Ok(ret) | ||
| } | ||
|
|
||
| fn execute(&self, statement: Statement) -> Result<ProxyExecResult, DbErr> { | ||
| async_std::task::block_on(async { | ||
| if let Some(values) = statement.values { | ||
| // Replace all the '?' with the statement values | ||
| let mut new_sql = statement.sql.clone(); | ||
| let mark_count = new_sql.matches('?').count(); | ||
| for (i, v) in values.0.iter().enumerate() { | ||
| if i >= mark_count { | ||
| break; | ||
| } | ||
| new_sql = new_sql.replacen('?', &v.to_string(), 1); | ||
| } | ||
| println!("SQL execute: {}", new_sql); | ||
|
|
||
| self.mem.query(new_sql).await | ||
| } else { | ||
| self.mem.query(statement.sql).await | ||
| } | ||
| }) | ||
| .unwrap(); | ||
|
|
||
| Ok(ProxyExecResult { | ||
| last_insert_id: 1, | ||
| rows_affected: 1, | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| #[async_std::main] | ||
| async fn main() { | ||
| let mem = Surreal::new::<Mem>(()).await.unwrap(); | ||
| mem.use_ns("test").use_db("post").await.unwrap(); | ||
|
|
||
| let db = Database::connect_proxy( | ||
| DbBackend::MySql, | ||
| Arc::new(Mutex::new(Box::new(ProxyDb { mem }))), | ||
| ) | ||
| .await | ||
| .unwrap(); | ||
|
|
||
| println!("Initialized"); | ||
|
|
||
| let data = ActiveModel { | ||
| title: Set("Homo".to_owned()), | ||
| text: Set("いいよ、来いよ".to_owned()), | ||
| ..Default::default() | ||
| }; | ||
| Entity::insert(data).exec(&db).await.unwrap(); | ||
| let data = ActiveModel { | ||
| title: Set("Homo".to_owned()), | ||
| text: Set("そうだよ".to_owned()), | ||
| ..Default::default() | ||
| }; | ||
| Entity::insert(data).exec(&db).await.unwrap(); | ||
| let data = ActiveModel { | ||
| title: Set("Homo".to_owned()), | ||
| text: Set("悔い改めて".to_owned()), | ||
| ..Default::default() | ||
| }; | ||
| Entity::insert(data).exec(&db).await.unwrap(); | ||
|
|
||
| let list = Entity::find().all(&db).await.unwrap().to_vec(); | ||
| println!("Result: {:?}", list); | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| [workspace] | ||
| members = ["module", "vm"] | ||
| resolver = "2" | ||
|
|
||
| [profile.release] | ||
| lto = true | ||
| opt-level = 'z' | ||
| codegen-units = 1 | ||
| panic = "abort" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| [package] | ||
| name = "sea-orm-proxy-wasmtime-example-module" | ||
| version = "0.1.0" | ||
| authors = ["Langyo <[email protected]>"] | ||
| edition = "2021" | ||
| publish = false | ||
|
|
||
| [dependencies] | ||
| ron = "0.8" | ||
| serde = { version = "1", features = ["derive"] } | ||
| anyhow = "1" | ||
| tokio_wasi = { version = "1", features = ["full"] } | ||
|
|
||
| sea-orm = { path = "../../../", default-features = false, features = [ | ||
| "macros", | ||
| "proxy", | ||
| "runtime-tokio", | ||
| ] } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This position temporarily disables sqlx because rustls, the upstream dependency of sqlx, cannot compile to any wasm target for the time being. The mainly cause is that rustls-dependent rings need to cross-compile code from C language.
Recently, ring may support compilation of the wasm platform by compiling old cipher suites with BoringC instead, but this will take a long time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Related to briansmith/ring#1745 and rustls/rustls#521