Entity Generator #11

billy1624 · 2021-06-01T14:57:48Z

Generating entity file for each db table.

Entity
Model
- mapping db column type to Rust type
- find_* helper function
Column
- ColumnTrait column def
PrimaryKey
Relation
- RelationTrait relation def
Related

Work plan in progress...

use crate as sea_orm;
use crate::entity::prelude::*;

#[derive(Copy, Clone, Default, Debug, DeriveEntity)]
#[table = "cake"]
pub struct Entity;

#[derive(Clone, Debug, PartialEq, DeriveModel, DeriveActiveModel)]
pub struct Model {
    pub id: i32,
    pub name: String,
}

#[derive(Copy, Clone, Debug, EnumIter, DeriveColumn)]
pub enum Column {
    Id,
    Name,
}

#[derive(Copy, Clone, Debug, EnumIter, DerivePrimaryKey)]
pub enum PrimaryKey {
    Id,
}

#[derive(Copy, Clone, Debug, EnumIter)]
pub enum Relation {
    Fruit,
}

impl ColumnTrait for Column {
    type EntityName = Entity;

    fn def(&self) -> ColumnType {
        match self {
            Self::Id => ColumnType::Integer(None),
            Self::Name => ColumnType::String(None),
        }
    }
}

impl RelationTrait for Relation {
    fn def(&self) -> RelationDef {
        match self {
            Self::Fruit => Entity::has_many(super::fruit::Entity)
                .from(Column::Id)
                .to(super::fruit::Column::CakeId)
                .into(),
        }
    }
}

impl Related<super::fruit::Entity> for Entity {
    fn to() -> RelationDef {
        Relation::Fruit.def()
    }
}

impl Related<super::filling::Entity> for Entity {
    fn to() -> RelationDef {
        super::cake_filling::Relation::Filling.def()
    }

    fn via() -> Option<RelationDef> {
        Some(super::cake_filling::Relation::Cake.def().rev())
    }
}

impl Model {
    pub fn find_fruit(&self) -> Select<super::fruit::Entity> {
        Entity::find_related().belongs_to::<Entity>(self)
    }

    pub fn find_filling(&self) -> Select<super::filling::Entity> {
        Entity::find_related().belongs_to::<Entity>(self)
    }
}

The text was updated successfully, but these errors were encountered:

tyt2y3 · 2021-06-03T11:50:36Z

What is the implementation strategy we have in mind?

billy1624 · 2021-06-03T13:42:19Z

Collect schema info using sea-schema.

Sample table info collected by it.

TableDef {
    info: TableInfo {
        name: "cake_filling",
        engine: InnoDb,
        auto_increment: None,
        char_set: Utf8Mb4,
        collation: Utf8Mb4GeneralCi,
        comment: "",
    },
    columns: [
        ColumnInfo {
            name: "cake_id",
            col_type: Int(
                NumericAttr {
                    maximum: Some(
                        11,
                    ),
                    decimal: None,
                    unsigned: None,
                    zero_fill: None,
                },
            ),
            null: false,
            key: Primary,
            default: None,
            extra: ColumnExtra {
                auto_increment: false,
                on_update_current_timestamp: false,
                generated: false,
                default_generated: false,
            },
            expression: None,
            comment: "",
        },
        ColumnInfo {
            name: "filling_id",
            col_type: Int(
                NumericAttr {
                    maximum: Some(
                        11,
                    ),
                    decimal: None,
                    unsigned: None,
                    zero_fill: None,
                },
            ),
            null: false,
            key: Primary,
            default: None,
            extra: ColumnExtra {
                auto_increment: false,
                on_update_current_timestamp: false,
                generated: false,
                default_generated: false,
            },
            expression: None,
            comment: "",
        },
    ],
    indexes: [
        IndexInfo {
            unique: false,
            name: "filling_id",
            parts: [
                IndexPart {
                    column: "filling_id",
                    order: Ascending,
                    sub_part: None,
                },
            ],
            nullable: false,
            idx_type: BTree,
            comment: "",
            functional: false,
        },
        IndexInfo {
            unique: true,
            name: "PRIMARY",
            parts: [
                IndexPart {
                    column: "cake_id",
                    order: Ascending,
                    sub_part: None,
                },
                IndexPart {
                    column: "filling_id",
                    order: Ascending,
                    sub_part: None,
                },
            ],
            nullable: false,
            idx_type: BTree,
            comment: "",
            functional: false,
        },
    ],
    foreign_keys: [
        ForeignKeyInfo {
            name: "cake_filling_ibfk_1",
            columns: [
                "cake_id",
            ],
            referenced_table: "cake",
            referenced_columns: [
                "id",
            ],
            on_update: Restrict,
            on_delete: Restrict,
        },
        ForeignKeyInfo {
            name: "cake_filling_ibfk_2",
            columns: [
                "filling_id",
            ],
            referenced_table: "filling",
            referenced_columns: [
                "id",
            ],
            on_update: Restrict,
            on_delete: Restrict,
        },
    ],
},

Mapping above table info into specification for orm entity, column & relation...

First pass
- map sea-schema TableRef info into entity, column & relation
- append inverse relation into inverse_relations
Second pass
- put inverse_relations back to corresponding entity relation

Data structure for generating entity file.

// Can actually be implemented as two runtime variable `entities` and `inverse_relations`
pub struct EntityGenerator {
    entities: Vec<EntitySpec>,
    inverse_relations: HashMap<String, Vec<RelationSpec>>,  // Store many-to-one relation in first pass
}

pub struct EntitySpec {
    table_name: String,
    columns: Vec<ColumnSpec>,
    relations: Vec<RelationSpec>,
}

pub struct ColumnSpec {
    name: String,
    rs_type: String,  // Rust type provided by sea-schema (this type info is currently not supported)
    col_type: ColumnType,  // Infer from `rs_type`
    is_primary_key: bool,
}

pub struct RelationSpec {
    name: String,
    ref_table: String,
    columns: Vec<String>,
    ref_columns: Vec<String>,
    rel_type: RelationType,  // Relation parse directly from sea-schema is one-to-one, otherwise is many-to-one since it's from `inverse_relations`
}

Having above info should be enough to generate the entity file.

Code generation...

having both API & CLI interface to initiate entity generation which required input...
- db connection string
- output path for entity files
using quote crate to construct struct, enum and impl trait...
formatting identifier (e.g. fruit, Fruit, id, Id) with heck crate to change between case
collecting each entity file content as string and write to corresponding entity file
formatting output file with rustfmt

billy1624 · 2021-06-03T15:35:42Z

See latest commit for the POC

tyt2y3 · 2021-06-03T18:37:37Z

Several comments.

I don't quite understand why first / second pass is needed. Can you explain?
The code generation part should be moved to a new crate, while it's okay to keep data structures (EntitySpec) inside the ORM
Instead of looking at the raw output of SeaSchema, perhaps we can use SeaQuery's TableStatement generated by SeaSchema's writer. It is engine agnostic already (not actually, still using some custom names, but can be easily changed) and has a simpler structure

billy1624 · 2021-06-07T15:39:49Z

What I mean by "second pass", just some implementation details
https://github.com/SeaQL/sea-orm/blob/codegen/sea-orm-codegen/src/entity/base.rs#L106
I will move all to a new crate called sea-orm-codegen
Sure! Now I have a working codegen draft, and thinking how to map the db columns type into corresponding Rust type and ColumnType

billy1624 · 2021-06-07T15:43:03Z

Still need plenty of refactoring but you can take a glance at the codegen branch

tyt2y3 · 2021-06-08T13:14:53Z

Oh so this is like a procedural macro but actually write code to a file?

billy1624 · 2021-06-08T13:27:32Z

Correct!

billy1624 · 2021-06-08T13:28:08Z

Will refactor the code now, too messy loll

tyt2y3 · 2021-06-08T13:28:53Z

Makes sense. But we'd better separate the discover -> transform -> generate stages.

billy1624 · 2021-06-08T13:31:08Z

Without keeping any internal state?

Sth like this?

input_0 -> Discover -> state_1
state_1 -> Transform -> state_2
state_2 -> Generate -> codes...

tyt2y3 · 2021-06-08T13:32:34Z

It's easier to unit test if intermediate states have data structures.

billy1624 · 2021-06-08T13:35:55Z

Nice! Will do it this way then

billy1624 · 2021-06-08T15:42:23Z

Refactored

tyt2y3 · 2021-06-09T07:57:52Z

Seems good so far

billy1624 · 2021-06-09T11:45:49Z

Going to develop the CLI now

Any suggestions?

tyt2y3 · 2021-06-09T14:18:06Z

diesel_cli is worth looking into.

billy1624 · 2021-06-10T14:34:27Z

For now, I'm mapping db column into one of basic rust type listed below. Do you think this is reasonable?

String
u32
f32
Vec<u8>
bool

It's difficult to map, for example, an integer db column into specific usize / u8 / u32 ...

sea-orm/sea-orm-codegen/src/entity/column.rs

Lines 21 to 44 in 2f6b216

    
           pub fn get_rs_type(&self) -> Ident { 
        
               match self.col_type { 
        
                   ColumnType::Char(_) 
        
                   | ColumnType::String(_) 
        
                   | ColumnType::Text 
        
                   | ColumnType::DateTime(_) 
        
                   | ColumnType::Timestamp(_) 
        
                   | ColumnType::Time(_) 
        
                   | ColumnType::Date 
        
                   | ColumnType::Json 
        
                   | ColumnType::JsonBinary 
        
                   | ColumnType::Custom(_) => format_ident!("String"), 
        
                   ColumnType::TinyInteger(_) 
        
                   | ColumnType::SmallInteger(_) 
        
                   | ColumnType::Integer(_) 
        
                   | ColumnType::BigInteger(_) => format_ident!("u32"), 
        
                   ColumnType::Float(_) 
        
                   | ColumnType::Double(_) 
        
                   | ColumnType::Decimal(_) 
        
                   | ColumnType::Money(_) => format_ident!("f32"), 
        
                   ColumnType::Binary(_) => format_ident!("Vec<u8>"), 
        
                   ColumnType::Boolean => format_ident!("bool"), 
        
               } 
        
           }

tyt2y3 · 2021-06-10T14:38:19Z

May be follow sea query

https://github.com/SeaQL/sea-query/blob/78ba5d7242dfe5e18e94736b4f522a837f609206/src/value.rs#L152-L162

billy1624 · 2021-06-10T14:58:23Z

sea-schema mysql writer only convert column type into sea_query::ColumnType::Custom.

So, I'm thinking is there any good way to generically convert sea-schema column info into sea_query::ColumnDef with specific sea_query::ColumnType (with no sea_query::ColumnType::Custom).

However, I understand that sea-schema has no responsibility to perform the bolded task. As it does the job to write accurate SQL table create statement.

https://github.com/SeaQL/sea-schema/blob/f5866dc64b9e39a7c41417b647a508d68a57c3a2/src/mysql/writer/column.rs#L8

tyt2y3 · 2021-06-10T15:01:57Z

I agree we can improve SeaSchema to output more specific ColumnDef provided that there is no data loss.

billy1624 · 2021-06-11T14:11:52Z

Checkout the quick demo of sea-orm-cli 33f16e1

tyt2y3 · 2021-06-11T16:17:53Z

Can't try without the sea query commits :(

billy1624 · 2021-06-11T16:19:14Z

Checkout sea-query/readonly branch

tyt2y3 · 2021-06-12T08:33:58Z

I tried. Quite good so far.
Several comments:

This doesn't compile:

    fn def(&self) -> RelationDef {
        match self {}
    }

And we should not use std::rc::Rc directly. We should use sea_query::SeaRc instead.
But I guess Alias will be removed anyway.

And we should also generate some doc comment:

//! SeaORM Entity. Generated by sea-orm 0.1.0

tyt2y3 · 2021-06-12T11:14:38Z

Oh, and it'd also be a good idea to generate the entities/mod.rs and entities/prelude.rs

billy1624 · 2021-06-13T01:35:51Z

Ah ha good idea

tyt2y3 · 2021-06-19T13:02:00Z

Several amendments required:

adding unique() and null() on ColumnDef
can we use rev() to define reverse relation?
if we detect that the foreign key column is unique, we should make the relation has_one()
(Explain: the distinction between one-to-one and one-to-many is whether the reference key is unique)
can we be smart enough to detect junction table (where it has two or more foreign keys)?

As ref:

impl Related<super::filling::Entity> for Entity {
    fn to() -> RelationDef {
        super::cake_filling::Relation::Filling.def()
    }

    fn via() -> Option<RelationDef> {
        Some(super::cake_filling::Relation::Cake.def().rev())
    }
}

But for 4, if we determined that it is too hard, we can delay it post release.

billy1624 · 2021-06-20T15:26:25Z

What do u mean by "reverse relation"?

For cake and fruit

Each fruit has one cake
Each cake has many fruit

Because foreign key is on fruit, so "Each cake has many fruit" is the inverse relation?

Then, we have...

// fruit.rs
impl Related<super::cake::Entity> for Entity {
    fn to() -> RelationDef {
        Relation::Cake.def()
    }
}

// cake.rs
impl Related<super::fruit::Entity> for Entity {
    fn to() -> RelationDef {
        super::Fruit::Relation::Cake.def().rev()
    }
}

tyt2y3 · 2021-06-21T13:25:37Z

Basically yes. Each relation should have only one owner where we define it.
Others should only reuse that definition.
Put it simple, don't repeat yourself.

billy1624 · 2021-06-21T14:37:34Z

Basically yes. Each relation should have only one owner where we define it.
Others should only reuse that definition.
Put it simple, don't repeat yourself.

Then basically we won't see has_many loll

tyt2y3 · 2021-06-21T16:07:28Z

How about

impl RelationTrait for Relation {
    fn def(&self) -> RelationDef {
        match self {
            Self::Fruit => Entity::has_many(super::fruit::Entity).into()
        }
    }
}

That means we actually require that the one side relation exists, and automatically define the has_many relation.

tyt2y3 · 2021-06-21T17:33:40Z

Done. See d77a7c8

tyt2y3 · 2021-06-21T17:34:43Z

To clarify again, if the foreign key column also has an unique constraint, then the reverse of it would also be has_one. Otherwise it is has_many.

Note: our current test schema does not have has_one.

billy1624 · 2021-06-22T03:22:40Z

Done. See d77a7c8

Ah ha! cool

billy1624 · 2021-06-22T03:23:07Z

To clarify again, if the foreign key column also has an unique constraint, then the reverse of it would also be has_one. Otherwise it is has_many.

Note: our current test schema does not have has_one.

Got it!

billy1624 · 2021-06-22T03:33:20Z

To clarify again, if the foreign key column also has an unique constraint, then the reverse of it would also be has_one. Otherwise it is has_many.

Note: our current test schema does not have has_one.

But then this special has_one still need to define something like... on both end

impl RelationTrait for Relation {
    fn def(&self) -> RelationDef {
        match self {
            Self::Cake => Entity::has_one(super::cake::Entity)
                .from(Column::CakeId)
                .to(super::cake::Column::Id)
                .into(),
        }
    }
}

tyt2y3 · 2021-06-22T06:19:04Z

I will add a belongs_to similar to has_many

billy1624 · 2021-06-22T06:21:48Z

I will add a belongs_to similar to has_many

Nice!

billy1624 · 2021-06-22T07:55:52Z

@tyt2y3

billy1624 · 2021-06-22T15:40:45Z

Several amendments required:

can we use rev() to define reverse relation?

Is this still valid? After we have belongs_to, has_one & has_many

Check 9a25bb9

tyt2y3 · 2021-07-03T07:28:21Z

How can we detect 'junction table' and generate many-to-many relations with via?

Perhaps:

We detect that a table has two or more belongs_to relations
We N choose 2 from the relations, and append has_many for each

tyt2y3 · 2021-07-03T07:29:38Z

Btw, the examples cli and codegen, one of them are redundant?

billy1624 · 2021-07-05T11:34:52Z

Yes, in terms of output

Just two different way to invoke codegen

tyt2y3 · 2021-07-06T07:03:58Z

Then we can remove one of them

billy1624 · 2021-07-07T12:40:11Z

How can we detect 'junction table' and generate many-to-many relations with via?

Perhaps:

We detect that a table has two or more belongs_to relations

We N choose 2 from the relations, and append has_many for each

For 2, yes we can do that when we found all "junction table".

For 1, I think the problem is to define what we mean by "junction table"?

Table that meets one of the following criteria?

table with exactly two foreign keys, and primary key is composite of these two foreign keys.
table with exactly two foreign keys. Together with an id primary key column in the table. And table can also have other additional non-key (not a primary key nor foreign key) columns, if there is any.

tyt2y3 · 2021-07-08T04:54:48Z

table with exactly two foreign keys, and primary key is composite of these two foreign keys

I agree, may be we can handle this common enough M-to-N junction case first

This was referenced Jun 1, 2021

0.1.0 To Do Tasks #1

Closed

Mapping db column type to Rust type SeaQL/sea-schema#4

Closed

billy1624 self-assigned this Jun 2, 2021

billy1624 linked a pull request Jun 18, 2021 that will close this issue

Codegen: Entity Generator #23

Merged

tyt2y3 closed this as completed in #23 Jun 19, 2021

tyt2y3 reopened this Jun 19, 2021

billy1624 mentioned this issue Jun 23, 2021

Codegen amendment #30

Merged

tyt2y3 closed this as completed in #30 Jun 26, 2021

tyt2y3 reopened this Jul 3, 2021

billy1624 mentioned this issue Jul 6, 2021

Codegen unit test #38

Merged

billy1624 mentioned this issue Jul 20, 2021

Entity generator handling M-to-N junction #51

Merged

tyt2y3 closed this as completed in #51 Jul 22, 2021

addap mentioned this issue Jan 13, 2023

Relations generated differ between Postgres and Sqlite #1393

Closed

Entity Generator #11

Entity Generator #11

Comments

billy1624 commented Jun 1, 2021 • edited Loading

tyt2y3 commented Jun 3, 2021

billy1624 commented Jun 3, 2021

billy1624 commented Jun 3, 2021

tyt2y3 commented Jun 3, 2021

billy1624 commented Jun 7, 2021

billy1624 commented Jun 7, 2021

tyt2y3 commented Jun 8, 2021 • edited Loading

billy1624 commented Jun 8, 2021

billy1624 commented Jun 8, 2021

tyt2y3 commented Jun 8, 2021

billy1624 commented Jun 8, 2021

tyt2y3 commented Jun 8, 2021 • edited Loading

billy1624 commented Jun 8, 2021

billy1624 commented Jun 8, 2021

tyt2y3 commented Jun 9, 2021

billy1624 commented Jun 9, 2021

tyt2y3 commented Jun 9, 2021

billy1624 commented Jun 10, 2021

tyt2y3 commented Jun 10, 2021 • edited Loading

billy1624 commented Jun 10, 2021

tyt2y3 commented Jun 10, 2021

billy1624 commented Jun 11, 2021

tyt2y3 commented Jun 11, 2021

billy1624 commented Jun 11, 2021

tyt2y3 commented Jun 12, 2021 • edited Loading

tyt2y3 commented Jun 12, 2021 • edited Loading

billy1624 commented Jun 13, 2021

tyt2y3 commented Jun 19, 2021 • edited Loading

billy1624 commented Jun 20, 2021

tyt2y3 commented Jun 21, 2021 • edited Loading

billy1624 commented Jun 21, 2021

tyt2y3 commented Jun 21, 2021 • edited Loading

tyt2y3 commented Jun 21, 2021

tyt2y3 commented Jun 21, 2021 • edited Loading

billy1624 commented Jun 22, 2021

billy1624 commented Jun 22, 2021

billy1624 commented Jun 22, 2021

tyt2y3 commented Jun 22, 2021

billy1624 commented Jun 22, 2021

billy1624 commented Jun 22, 2021

billy1624 commented Jun 22, 2021 • edited Loading

tyt2y3 commented Jul 3, 2021 • edited Loading

tyt2y3 commented Jul 3, 2021

billy1624 commented Jul 5, 2021

tyt2y3 commented Jul 6, 2021

billy1624 commented Jul 7, 2021

tyt2y3 commented Jul 8, 2021 • edited Loading

billy1624 commented Jun 1, 2021 •

edited

Loading

tyt2y3 commented Jun 8, 2021 •

edited

Loading

tyt2y3 commented Jun 8, 2021 •

edited

Loading

tyt2y3 commented Jun 10, 2021 •

edited

Loading

tyt2y3 commented Jun 12, 2021 •

edited

Loading

tyt2y3 commented Jun 12, 2021 •

edited

Loading

tyt2y3 commented Jun 19, 2021 •

edited

Loading

tyt2y3 commented Jun 21, 2021 •

edited

Loading

tyt2y3 commented Jun 21, 2021 •

edited

Loading

tyt2y3 commented Jun 21, 2021 •

edited

Loading

billy1624 commented Jun 22, 2021 •

edited

Loading

tyt2y3 commented Jul 3, 2021 •

edited

Loading

tyt2y3 commented Jul 8, 2021 •

edited

Loading