Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local and endpoint migration for renamed and removed entities. #554

Merged
merged 11 commits into from
Sep 14, 2021

Conversation

sa-bpelakh
Copy link
Collaborator

This is in anticipation of #535, based on the most recent version.

In order to run these, you will need to make sure that onto-tool is upgraded to v1.4.1.

I ran the local migration using the following command:

onto_tool bundle -v input test-migration -v output test-migration/migrated -v report test-migration/report migration/v10.0/migrate_local.yaml

which updates all .ttl and .owl files in the input directory (not recursively, is that an issue?) and places the updated RDF in the output directory. ValidationReport RDF is placed in the report directory.

To run the migration against a SPARQL endpoint:

onto_tool bundle -v user USER -v password PWD -v endpoint https://agraph.semanticarts.com/catalogs/demo-catalog/repositories/ci-test -v report test-migration/endpoint-report migration/v10.0/migrate_endpoint.yaml

For databases where the UPDATE endpoint is different, it can be provided via -v update_endpoint URI in addition to the query endpoint provided via -v endpoint.

Do we want to include detailed documentation for the migration in the release notes of each major release, since users will not have access to our Confluence pages?

Copy link
Collaborator

@rjyounes rjyounes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see detailed instructions in a separate file, not in the release notes.

} UNION {
?oldClass ?defProp ?defObj
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have both action_rename_classes.rq and action_rename_classes_default.rq and similarly for properties?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do renaming in two steps:

  1. First rename everything that is in a named graph, retaining the modified triples in their original graphs, and then
  2. Rename all that is remaining, which would be in the default graph
    We do it in this manner because the default graph does not have a universally recognized reference URI, and in some triple stores it doesn't have a URI at all. Also, when processing local file data, there are no named graphs, so only the default queries (which don't specify a graph) are used.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A local .trig or N-quads file can have named graphs. I assume that works correctly for them as well.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jamie-SA There is no current support in onto-tool for .trig or .nq inputs. Do you view this as a deal breaker for this functionality? There is no way I can code and test this before Monday

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not a deal breaker for me, I had assumed the library you use to read/write would support all of the main formats. But if not, I wouldn't hold this up because of it.

I have switched almost entirely to .trig because of it's support for named graphs.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The toolkit (rdflib) does support it, but I have not enabled it - and there is an existing issue (semanticarts/ontology-toolkit#65), to which I will give some attention after this release.

@sa-bpelakh sa-bpelakh requested a review from rjyounes September 9, 2021 17:46
}
}
}
bind(<urn:deleted-entity-vaidation-report> as ?report)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor typo:

Suggested change
bind(<urn:deleted-entity-vaidation-report> as ?report)
bind(<urn:deleted-entity-validation-report> as ?report)

@sa-bpelakh
Copy link
Collaborator Author

@rjyounes Please read over my documentation. @bolerio Should be good to go as soon as @rjyounes approves the verbiage.

Copy link
Collaborator

@rjyounes rjyounes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great except for one or two suggested rewordings in the documentation. Use at your own discretion.

# Major Version Migration

When non-backward compatible changes are introduced into `gist` during major upgrades,
developers will create queries that attempt to migrate existing ontologies and data to
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of "developers will create queries that attempt to migrate..." how about "the release package will include queries to facilitate the migration of existing ontologies and date to conform to the new version of gist."

My reasoning here is that "try to" sounds like we're not sure. Where we have scripts that do the migration, we're sure that they work.

docs/MajorVersionMigration.md Outdated Show resolved Hide resolved
docs/MajorVersionMigration.md Show resolved Hide resolved
@bolerio
Copy link
Contributor

bolerio commented Sep 13, 2021

Thanks @sa-bpelakh and @rjyounes .

@rjyounes , waiting on your final approval to merge and seal the release.

@sa-bpelakh sa-bpelakh requested a review from rjyounes September 13, 2021 19:31
@sa-bpelakh
Copy link
Collaborator Author

@rjyounes Adjusted documentation as per your suggestions

@sa-bpelakh sa-bpelakh merged commit ee3f06e into release-10.0.0 Sep 14, 2021
@sa-bpelakh sa-bpelakh deleted the basic-migration branch September 14, 2021 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants