Proposal: A language-independent tool for managing and migrating database schemas

[haoxins]

I want to share a proposal for a language-independent tool for managing and migrating schemas.
The ideas for this come from the awesome project Atlas.

There will be 3 key features for the end users.

Schema as code

• Ref: https://atlasgo.io/getting-started/

For example, we can define the schemas as below:

schema.hcl

space "account_transaction" {
  vertex "account" {
    id = string
    prop "email" {
      type = string
    }
  }
  vertex "transaction" {
    id = string
    prop "amount" {
      type = float
    }
  }
  edge "account_transaction" {
    id = string
    src = account
    dst = transaction
    prop "timestamp" {
      type = int
      ttl_col = true
    }
  }
}

With this schema file, we can apply this to the database by the commands below:

# inspect first
nebula-schema inspect schema.hcl --host nebula_host --port nebula_port --user user_name --pass user_pass
# apply
nebula-schema apply schema.hcl --host nebula_host --port nebula_port --user user_name --pass user_pass

Integrates with the Nebula client

If we choose Golang to implement the application, we can also provide special providers for different languages.
Such as https://github.com/ariga/atlas-provider-hibernate

Integrates with the Nebula operator

Since we already have the K8s operator for Nebula, https://github.com/vesoft-inc/nebula-operator.
We can integrate schema management into the operator.
See https://atlasgo.io/integrations/kubernetes/operator

[yixinglu]

This looks very interesting! I believe nebula community will like this job very much. However, there are one question about the definition of edge. I see that it defines the type of src/dst, which is consistent with the upcoming GQL standard, but there is no such constraint in nebula.

[haoxins]

This looks very interesting! I believe nebula community will like this job very much.

This is great!

there are one question about the definition of edge. I see that it defines the type of src/dst, which is consistent with the upcoming GQL standard, but there is no such constraint in nebula.

haha, I just wrote a rough example for understanding, if we have an agreement on starting this project, we can start to define the standard more carefully.

which is consistent with the upcoming GQL standard, but there is no such constraint in nebula.

1. I haven't read the GQL standard, and I haven't found a page containing the whole GQL standard, can you share one link or pdf?
2. I wrote the example that copied the idea from nebula-exchange's config, it seems Nebula has the src/dst constraint?

[HarrisChu]

# inspect first
nebula-schema inspect schema.hcl --host nebula_host --port nebula_port --user user_name --pass user_pass
# apply
nebula-schema apply schema.hcl --host nebula_host --port nebula_port --user user_name --pass user_pass

like this very much
how about export command?

#e.g.
nebula-schema export  --host nebula_host --port nebula_port --user user_name --pass user_pass --space xxx --output schema.hcl 

[haoxins]

+1

[wey-gu]

Dear @haoxins

Thanks!

This is a NEXT LEVEL missing piece for NebulaGraph.(We had been thinking of things like integration with bytebase, etc) and Atlas looks awesome to me!

On top of the abstraction of schema/ddl as code, the integration on client/operators pushes it to a level I didn't even think of.

I think your starting work on this will bring the NebulaGraph community to a whole new level.

I think the schema-aC is version-wised, so we could start with the one that edges' src/dst vertex without type constraints, but in next major version, where we honor ISO GQL, will do(but the schema will be different in other perspectives, too).

While, indeed, for now when you are defining the Schema-aC DSL, considerations based on ISO GQL may make future adaptation easier(that's why you're mentioning its PDF version right?)

@HarrisChu @yixinglu @xtcyclist what do you think? BTW could you help share @haoxins a readable version of ISO GQL standard?

[wey-gu]

We are more than happy to either you put it under your org/id or eventually consider putting it under nebula-contrib tent

[haoxins]

While, indeed, for now when you are defining the Schema-aC DSL, considerations based on ISO GQL may make future adaptation easier(that's why you're mentioning its PDF version right?)

Yes!

[haoxins]

We are more than happy to either you put it under your org/id or eventually consider putting it under nebula-contrib tent

make sense. So that

1. we start with our org and then migrate it to nebula-contrib tent or
2. we init one empty project under nebula-contrib tent and start the work?

[wey-gu]

It's recommended to start from your own id/org, so you could merge things fast(contrib org will enforce 2+ approvel merge and branch protections)

But feel free to discuss here with ref of that report when you consider needed:)

[haoxins]

It's recommended to start from your own id/org, so you could merge things fast(contrib org will enforce 2+ approvel merge and branch protections)

Make sense

[haoxins]

Add a reference for the feedback from the author of Atlas.

ariga/atlas#2443 (comment)

[wey-gu]

Amazing, this is what Open-Source is all about.

Let's collaborate to make that happen🫡