Alpha server start issue with ludicrous mode #5601
Labels
status/accepted
We accept to investigate/work on it.
Comments
|
I looked into as to why this started happening. So, when I use master at 28617a6, I get the following error in ludicrous_mode: This error keeps occurring for a while, and after that it goes away (within a minute), and alpha is able to boot up properly with the initial node for GraphQL schema. I had checked history till commit 4b264e4, and the same behaviour was there. Now, when I use master one commit after the above commit at aef7072, I get following kinds of error in ludicrous_mode: These errors keep repeating, with the first error occurring more often than the second. I tried waiting for some time, but these errors never went away. This causes alpha to not be able to boot up. Now, the reason why this behaviour is happening with the second commit and not the first one, is because the second commit restricts the mutation for predicates starting with And, I think the reason why it doesn't happen with the first commit is although the full initial schema is not there yet, but because of no restriction on the mutations on predicates starting with In order to solve it, I guess the only way is to make sure that the initial proposal for internal types and predicates gets applied properly, irrespective of the ludicrous_mode constraints, as indicated by the first error. So, in a way the commit aef7072 breaks ludicrous_mode, but the issue lies somewhere else. |
ashish-goswami
added a commit
that referenced
this issue
Jul 8, 2020
Fixes: #5601 Fixes: DGRAPH-1669 Currently we are unable to start alpha server in ludicrous mode, because of reasons explained below. Things we do at alpha start: At start we insert dgraph.type, dgraph.graphql.schema and dgraph.graphql.xid predicates as our initial schema. We also insert dgraph.graphql type(having predicates dgraph.graphql.schema and dgraph.graphql.xid) as initial types. We also upsert data related to graphql schema(which results in upserts for predicates dgraph.graphql.schema and dgraph.graphql.xid). While inserting types we verify that predicates used in the types should either already be present in schema or should be part of same mutation where we are trying to insert this type. In ludicrous mode we mark any proposal done without error as soon as it is retrieved as part of commited entries, as opposed to after applying proposal (this is done in normal mode). Events which led to alpha getting stuck: We propose schema mutation for dgraph.type, which is marked as done immediately. This proposal is still getting applied. Since we marked above proposal done without error, we propose next schema mutation for dgraph.graphql.schema, which again is marked done without error. But applying of this proposal fails as indexing was already in progress for dgraph.type. But we never retry this schema mutation. Next we propose dgraph.graphql.xid, which is applied successfully(indexing is done until now for dgraph.type). When we insert dgraph.graphql type, verifyTypes() fails, because it never finds dgraphq.graphql.schema predicate. Proposed solution: Assumption that proposal is never failed while applying, hence it should be marked as done without error immediately, should hold only for data mutations and not schema mutations. This PR checks if proposal is data mutation(having edges > 0), while marking it done immediately. This leads to retrying of schema proposal if they fail once because indexing is already in progress. Note: We didn't observe this before commit(aef7072), because predicate dgraph.graphql.schema was getting inserted via graphql upserts(#3 of start steps). Now after commit(aef7072) we cannot insert data for predicates starting with dgraph. prefix unless those are already present.
ashish-goswami
added a commit
that referenced
this issue
Jul 9, 2020
Fixes: #5601 Fixes: DGRAPH-1669 Currently we are unable to start alpha server in ludicrous mode, because of reasons explained below. Things we do at alpha start: At start we insert dgraph.type, dgraph.graphql.schema and dgraph.graphql.xid predicates as our initial schema. We also insert dgraph.graphql type(having predicates dgraph.graphql.schema and dgraph.graphql.xid) as initial types. We also upsert data related to graphql schema(which results in upserts for predicates dgraph.graphql.schema and dgraph.graphql.xid). While inserting types we verify that predicates used in the types should either already be present in schema or should be part of same mutation where we are trying to insert this type. In ludicrous mode we mark any proposal done without error as soon as it is retrieved as part of commited entries, as opposed to after applying proposal (this is done in normal mode). Events which led to alpha getting stuck: We propose schema mutation for dgraph.type, which is marked as done immediately. This proposal is still getting applied. Since we marked above proposal done without error, we propose next schema mutation for dgraph.graphql.schema, which again is marked done without error. But applying of this proposal fails as indexing was already in progress for dgraph.type. But we never retry this schema mutation. Next we propose dgraph.graphql.xid, which is applied successfully(indexing is done until now for dgraph.type). When we insert dgraph.graphql type, verifyTypes() fails, because it never finds dgraphq.graphql.schema predicate. Proposed solution: Assumption that proposal is never failed while applying, hence it should be marked as done without error immediately, should hold only for data mutations and not schema mutations. This PR checks if proposal is data mutation(having edges > 0), while marking it done immediately. This leads to retrying of schema proposal if they fail once because indexing is already in progress. Note: We didn't observe this before commit(aef7072), because predicate dgraph.graphql.schema was getting inserted via graphql upserts(#3 of start steps). Now after commit(aef7072) we cannot insert data for predicates starting with dgraph. prefix unless those are already present. (cherry picked from commit d3c16be)
ashish-goswami
added a commit
that referenced
this issue
Jul 10, 2020
Fixes: #5601 Fixes: DGRAPH-1669 Currently we are unable to start alpha server in ludicrous mode, because of reasons explained below. Things we do at alpha start: At start we insert dgraph.type, dgraph.graphql.schema and dgraph.graphql.xid predicates as our initial schema. We also insert dgraph.graphql type(having predicates dgraph.graphql.schema and dgraph.graphql.xid) as initial types. We also upsert data related to graphql schema(which results in upserts for predicates dgraph.graphql.schema and dgraph.graphql.xid). While inserting types we verify that predicates used in the types should either already be present in schema or should be part of same mutation where we are trying to insert this type. In ludicrous mode we mark any proposal done without error as soon as it is retrieved as part of commited entries, as opposed to after applying proposal (this is done in normal mode). Events which led to alpha getting stuck: We propose schema mutation for dgraph.type, which is marked as done immediately. This proposal is still getting applied. Since we marked above proposal done without error, we propose next schema mutation for dgraph.graphql.schema, which again is marked done without error. But applying of this proposal fails as indexing was already in progress for dgraph.type. But we never retry this schema mutation. Next we propose dgraph.graphql.xid, which is applied successfully(indexing is done until now for dgraph.type). When we insert dgraph.graphql type, verifyTypes() fails, because it never finds dgraphq.graphql.schema predicate. Proposed solution: Assumption that proposal is never failed while applying, hence it should be marked as done without error immediately, should hold only for data mutations and not schema mutations. This PR checks if proposal is data mutation(having edges > 0), while marking it done immediately. This leads to retrying of schema proposal if they fail once because indexing is already in progress. Note: We didn't observe this before commit(aef7072), because predicate dgraph.graphql.schema was getting inserted via graphql upserts(#3 of start steps). Now after commit(aef7072) we cannot insert data for predicates starting with dgraph. prefix unless those are already present. (cherry picked from commit d3c16be)
arijitAD
pushed a commit
that referenced
this issue
Jul 14, 2020
Fixes: #5601 Fixes: DGRAPH-1669 Currently we are unable to start alpha server in ludicrous mode, because of reasons explained below. Things we do at alpha start: At start we insert dgraph.type, dgraph.graphql.schema and dgraph.graphql.xid predicates as our initial schema. We also insert dgraph.graphql type(having predicates dgraph.graphql.schema and dgraph.graphql.xid) as initial types. We also upsert data related to graphql schema(which results in upserts for predicates dgraph.graphql.schema and dgraph.graphql.xid). While inserting types we verify that predicates used in the types should either already be present in schema or should be part of same mutation where we are trying to insert this type. In ludicrous mode we mark any proposal done without error as soon as it is retrieved as part of commited entries, as opposed to after applying proposal (this is done in normal mode). Events which led to alpha getting stuck: We propose schema mutation for dgraph.type, which is marked as done immediately. This proposal is still getting applied. Since we marked above proposal done without error, we propose next schema mutation for dgraph.graphql.schema, which again is marked done without error. But applying of this proposal fails as indexing was already in progress for dgraph.type. But we never retry this schema mutation. Next we propose dgraph.graphql.xid, which is applied successfully(indexing is done until now for dgraph.type). When we insert dgraph.graphql type, verifyTypes() fails, because it never finds dgraphq.graphql.schema predicate. Proposed solution: Assumption that proposal is never failed while applying, hence it should be marked as done without error immediately, should hold only for data mutations and not schema mutations. This PR checks if proposal is data mutation(having edges > 0), while marking it done immediately. This leads to retrying of schema proposal if they fail once because indexing is already in progress. Note: We didn't observe this before commit(aef7072), because predicate dgraph.graphql.schema was getting inserted via graphql upserts(#3 of start steps). Now after commit(aef7072) we cannot insert data for predicates starting with dgraph. prefix unless those are already present.
dna2github
pushed a commit
to dna2fork/dgraph
that referenced
this issue
Jul 18, 2020
Fixes: dgraph-io#5601 Fixes: DGRAPH-1669 Currently we are unable to start alpha server in ludicrous mode, because of reasons explained below. Things we do at alpha start: At start we insert dgraph.type, dgraph.graphql.schema and dgraph.graphql.xid predicates as our initial schema. We also insert dgraph.graphql type(having predicates dgraph.graphql.schema and dgraph.graphql.xid) as initial types. We also upsert data related to graphql schema(which results in upserts for predicates dgraph.graphql.schema and dgraph.graphql.xid). While inserting types we verify that predicates used in the types should either already be present in schema or should be part of same mutation where we are trying to insert this type. In ludicrous mode we mark any proposal done without error as soon as it is retrieved as part of commited entries, as opposed to after applying proposal (this is done in normal mode). Events which led to alpha getting stuck: We propose schema mutation for dgraph.type, which is marked as done immediately. This proposal is still getting applied. Since we marked above proposal done without error, we propose next schema mutation for dgraph.graphql.schema, which again is marked done without error. But applying of this proposal fails as indexing was already in progress for dgraph.type. But we never retry this schema mutation. Next we propose dgraph.graphql.xid, which is applied successfully(indexing is done until now for dgraph.type). When we insert dgraph.graphql type, verifyTypes() fails, because it never finds dgraphq.graphql.schema predicate. Proposed solution: Assumption that proposal is never failed while applying, hence it should be marked as done without error immediately, should hold only for data mutations and not schema mutations. This PR checks if proposal is data mutation(having edges > 0), while marking it done immediately. This leads to retrying of schema proposal if they fail once because indexing is already in progress. Note: We didn't observe this before commit(aef7072), because predicate dgraph.graphql.schema was getting inserted via graphql upserts(dgraph-io#3 of start steps). Now after commit(aef7072) we cannot insert data for predicates starting with dgraph. prefix unless those are already present.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of Dgraph are you using?
master(222b35a)
Have you tried reproducing the issue with the latest release?
What is the hardware spec (RAM, OS)?
Steps to reproduce the issue (command/config used to run Dgraph).
Alpha node continuously prints below logs:
Expected behaviour and actual result.
The text was updated successfully, but these errors were encountered: