Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BREAKING] Namespace dgraph internal types/predicates with dgraph. #5185

Merged
merged 33 commits into from
Jun 5, 2020

Conversation

animesh2049
Copy link
Contributor

@animesh2049 animesh2049 commented Apr 13, 2020

Dgraph's internal types User, Group and Rule were too generic. Prefixed them with dgraph.type so that there is no chance of collision with user-defined types.

Also, reserved dgraph. as the namespace for dgraph's internal types/predicates so that no user could create any such type/predicate.

Fixes #4878.
Fixes #DGRAPH-1090.


This change is Reviewable

Docs Preview: Dgraph Preview

@animesh2049 animesh2049 requested review from manishrjain and a team as code owners April 13, 2020 14:09
x/x.go Outdated Show resolved Hide resolved
x/x.go Outdated Show resolved Hide resolved
edgraph/access_ee.go Outdated Show resolved Hide resolved
edgraph/access_ee.go Outdated Show resolved Hide resolved
edgraph/access_ee.go Outdated Show resolved Hide resolved
Copy link
Contributor

@pawanrawal pawanrawal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a test case showing that the user can't create or update types with the prefix dgraph.type. as those are reserved types. Do we already have one?
Would this be a breaking change and would require migrating data? How would users migrate from older versions?

Reviewed 1 of 9 files at r1, 4 of 9 files at r2, 4 of 4 files at r3.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @animesh2049 and @manishrjain)

x/x.go Outdated Show resolved Hide resolved
@animesh2049 animesh2049 changed the title Prefix internal types with dgraph.type [BREAKING] Prefix internal types with dgraph.type Apr 13, 2020
Copy link
Contributor Author

@animesh2049 animesh2049 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we do restrict Alter of internal types. I have also added tests for the same. Now we need to figure out a way to migrate users from older version.

Reviewable status: 5 of 12 files reviewed, 6 unresolved discussions (waiting on @animesh2049, @manishrjain, and @pawanrawal)

Copy link
Contributor

@abhimanyusinghgaur abhimanyusinghgaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 5 of 12 files reviewed, 7 unresolved discussions (waiting on @animesh2049, @manishrjain, and @pawanrawal)

a discussion (no related file):
I was thinking should we reserve dgraph. as a namespace for the internal types and predicates that we may create in future? So that people don't create types/predicates starting with dgraph.. Otherwise, they may create such a type/predicates now, and later if we are introducing that type/predicates internally, then those people will have to change their schema, which won't be a good experience for them.


Copy link
Contributor

@pawanrawal pawanrawal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 2 files at r4, 5 of 5 files at r5.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @abhimanyusinghgaur, @animesh2049, and @manishrjain)

a discussion (no related file):

Previously, abhimanyusinghgaur (Abhimanyu Singh Gaur) wrote…

I was thinking should we reserve dgraph. as a namespace for the internal types that we may create in future? So that people don't create types starting with dgraph.. Otherwise, they may create such a type now, and later if we are introducing that type internally, then those people will have to change their schema, which won't be a good experience for them.

Yeah, agree. We should ideally not allow the user to create/alter predicates/types prefixed with dgraph.


Copy link
Contributor

@abhimanyusinghgaur abhimanyusinghgaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 6 of 15 files reviewed, 5 unresolved discussions (waiting on @manishrjain, @MichaelJCompton, @pawanrawal, and @vvbalaji-dgraph)

a discussion (no related file):

Previously, pawanrawal (Pawan Rawal) wrote…

We need to figure out a migration story for this as this would require migration of ACL data to work for old users.

Created a story in JIRA for this.



edgraph/server.go, line 266 at r6 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Add more comments about how pre-defined are just a subset of reserved namespace so that this is easier to understand and follow. Also, say that we allow schema updates for predefined predicate if they are the same as before but disallow for reserved predicates that are not pre-defined.

Done


query/mutation_test.go, line 34 at r6 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Should this be TestAlteringReservedTypesAndPredicatesShouldFail?

Done


worker/proposal.go, line 169 at r6 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Please add a comment here saying that we don't allow mutations for reserved predicates if the schema for them doesn't already exist.

Done


x/keys.go, line 573 at r6 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Add a comment that this would be a subset of IsReservedPredicate

Done


x/keys.go, line 610 at r6 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Comment is old

Done

Copy link
Contributor

@pawanrawal pawanrawal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 11 files at r6, 8 of 9 files at r7.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @manishrjain, @MichaelJCompton, and @vvbalaji-dgraph)

a discussion (no related file):

Previously, abhimanyusinghgaur (Abhimanyu Singh Gaur) wrote…

Created a story in JIRA for this.

:lgtm:


Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @animesh2049, @manishrjain, @MichaelJCompton, and @vvbalaji-dgraph)


x/keys.go, line 591 at r7 (raw file):

reservedPredicateMap

maybe this map should be renamed to predefinedPredicateMap?

Copy link
Contributor

@abhimanyusinghgaur abhimanyusinghgaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 14 of 16 files reviewed, 1 unresolved discussion (waiting on @manishrjain, @martinmr, @MichaelJCompton, @pawanrawal, and @vvbalaji-dgraph)


x/keys.go, line 591 at r7 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…
reservedPredicateMap

maybe this map should be renamed to predefinedPredicateMap?

Renamed it to starAllPredicateMap as it was being used only for finding predicates which were needed when doing edge expansion for queries containing * in nquads.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if the dgraph. prefix is the right one. Perhaps use _dgraph. or something.

Reviewable status: 12 of 16 files reviewed, 3 unresolved discussions (waiting on @animesh2049, @manishrjain, @martinmr, @MichaelJCompton, @pawanrawal, and @vvbalaji-dgraph)


edgraph/access_ee.go, line 828 at r8 (raw file):

	acl info by applying filter of userid and groupid to acl predicates. A query like
	Conversion pattern:
		* me(func: type(dgraph.type.Group)) ->

Perhaps use _dgraph. as the name.


worker/proposal.go, line 172 at r8 (raw file):

				// already exist.
				if x.IsReservedPredicate(edge.Attr) {
					return errors.Errorf("Can't store predicate `%s` as it is prefixed with `dgraph.`"+

100 chars.

Copy link
Contributor

@abhimanyusinghgaur abhimanyusinghgaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 11 of 16 files reviewed, 3 unresolved discussions (waiting on @manishrjain, @martinmr, @MichaelJCompton, @pawanrawal, and @vvbalaji-dgraph)


edgraph/access_ee.go, line 828 at r8 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Perhaps use _dgraph. as the name.

As discussed, keeping it to dgraph. because changing it to _dgraph. will force all users to migrate data.


worker/proposal.go, line 172 at r8 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

100 chars.

Done

@abhimanyusinghgaur abhimanyusinghgaur merged commit aef7072 into master Jun 5, 2020
@abhimanyusinghgaur abhimanyusinghgaur deleted the animesh2049/internal-types branch June 16, 2020 21:53
abhimanyusinghgaur added a commit that referenced this pull request Jun 19, 2020
This PR fixes the issue where live loader was not able to import the schema exported using export request. It does so, by allowing the alter schema request for pre-defined types to go through if the type in the request is same as the initial internal type.

This issue was introduced as a result of #5185.

This PR also disallows a pre-defined type from being dropped.

Fixes #DGRAPH-1694.
abhimanyusinghgaur added a commit that referenced this pull request Jun 19, 2020
This PR fixes the issue where live loader was not able to import the schema exported using export request. It does so, by allowing the alter schema request for pre-defined types to go through if the type in the request is same as the initial internal type.

This issue was introduced as a result of #5185.

This PR also disallows a pre-defined type from being dropped.

Fixes #DGRAPH-1694.

(cherry picked from commit 76ae60b)
danielmai pushed a commit that referenced this pull request Jun 19, 2020
This PR fixes the issue where live loader was not able to import the schema exported using export request. It does so, by allowing the alter schema request for pre-defined types to go through if the type in the request is same as the initial internal type.

This issue was introduced as a result of #5185.

This PR also disallows a pre-defined type from being dropped.

Fixes #DGRAPH-1694.

(cherry picked from commit 76ae60b)
abhimanyusinghgaur added a commit that referenced this pull request Jul 1, 2020
Fixes #DGRAPH-1617.

This PR adds the capability to upgrade to v20.07.0 from v20.03.0+
It has only considered the breaking changes introduced in #5185.

It also introduces a generic system to handle upgrades.

At present, upgrade tool expects to handle only breaking ACL changes, and does not expect to handle badger data format changes. The way it is supposed to work is by updating the Dgraph binary to new version, start Dgraph cluster with existing data directories (which could also have been restored by enterprise backup/restore), and then running the upgrade tool by specifying the version you want to upgrade from and the version you want to upgrade to.
abhimanyusinghgaur added a commit that referenced this pull request Jul 6, 2020
Fixes #DGRAPH-1617.

This PR adds the capability to upgrade to v20.07.0 from v20.03.0+
It has only considered the breaking changes introduced in #5185.

It also introduces a generic system to handle upgrades.

At present, upgrade tool expects to handle only breaking ACL changes, and does not expect to handle badger data format changes. The way it is supposed to work is by updating the Dgraph binary to new version, start Dgraph cluster with existing data directories (which could also have been restored by enterprise backup/restore), and then running the upgrade tool by specifying the version you want to upgrade from and the version you want to upgrade to.

(cherry picked from commit d5892dc)
abhimanyusinghgaur added a commit that referenced this pull request Jul 6, 2020
Fixes #DGRAPH-1617.

This PR adds the capability to upgrade to v20.07.0 from v20.03.0+
It has only considered the breaking changes introduced in #5185.

It also introduces a generic system to handle upgrades.

At present, upgrade tool expects to handle only breaking ACL changes, and does not expect to handle badger data format changes. The way it is supposed to work is by updating the Dgraph binary to new version, start Dgraph cluster with existing data directories (which could also have been restored by enterprise backup/restore), and then running the upgrade tool by specifying the version you want to upgrade from and the version you want to upgrade to.

(cherry picked from commit d5892dc)
arijitAD pushed a commit that referenced this pull request Jul 14, 2020
Fixes #DGRAPH-1617.

This PR adds the capability to upgrade to v20.07.0 from v20.03.0+
It has only considered the breaking changes introduced in #5185.

It also introduces a generic system to handle upgrades.

At present, upgrade tool expects to handle only breaking ACL changes, and does not expect to handle badger data format changes. The way it is supposed to work is by updating the Dgraph binary to new version, start Dgraph cluster with existing data directories (which could also have been restored by enterprise backup/restore), and then running the upgrade tool by specifying the version you want to upgrade from and the version you want to upgrade to.
dna2github pushed a commit to dna2fork/dgraph that referenced this pull request Jul 18, 2020
…graph-io#5185)

Dgraph's internal types User, Group and Rule were too generic. Prefixed them with dgraph.type so that there is no chance of collision with user-defined types.

Also, reserved dgraph. as the namespace for dgraph's internal types/predicates so that no user could create any such type/predicate.

Fixes dgraph-io#4878.
Fixes #DGRAPH-1090.
dna2github pushed a commit to dna2fork/dgraph that referenced this pull request Jul 18, 2020
This PR fixes the issue where live loader was not able to import the schema exported using export request. It does so, by allowing the alter schema request for pre-defined types to go through if the type in the request is same as the initial internal type.

This issue was introduced as a result of dgraph-io#5185.

This PR also disallows a pre-defined type from being dropped.

Fixes #DGRAPH-1694.
dna2github pushed a commit to dna2fork/dgraph that referenced this pull request Jul 18, 2020
Fixes #DGRAPH-1617.

This PR adds the capability to upgrade to v20.07.0 from v20.03.0+
It has only considered the breaking changes introduced in dgraph-io#5185.

It also introduces a generic system to handle upgrades.

At present, upgrade tool expects to handle only breaking ACL changes, and does not expect to handle badger data format changes. The way it is supposed to work is by updating the Dgraph binary to new version, start Dgraph cluster with existing data directories (which could also have been restored by enterprise backup/restore), and then running the upgrade tool by specifying the version you want to upgrade from and the version you want to upgrade to.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Namespace dgraph internal types
6 participants