Extend native import/export plugin to include additional entities

[asmecher]

Such as (probably not canonical)...

• Not-yet-published files
• Review assignments
• Discussions/queries
• Log entries
• Journal configuration/settings (this might involve diminishing returns)
• ... etc

[asmecher]

(FYI, @jmacgreg)

[jmacgreg]

Oh, nice. CCing @mfelczak as well. Have we discussed the potential for using this as a PLN deposit option too? Should we ping Mark?

[asmecher]

The PLN plugin already uses the native import/export. This would extend its capabilities, at the risk of increasing storage requirements, and I could see us going either way (having the PLN plugin exclude the additional data and stick closer to its core mission, or allowing the additional data in to ease the "restoration" process). @mjordan, any preference?

[mjordan]

No objections here. If we don't want to load some of the new elements into an access version of a preserved journal, some preprocessing could strip them out. Or perhaps more sensibly, we preprocess the XML to be loaded to only include galleys, metadata, etc.

[diegoabadan]

See https://github.com/lepidus/fullJournalTransfer (for OJS 2.x)

[asmecher]

@diegoabadan, thanks, that's been a useful bit of work for OJS 2.x.

For 3.x, the XML import/export plugin uses the filter framework, which essentially allows each entity to have a matching conversion filter from XML, and to XML. This will facilitate modular code -- the OJS 2.x XML import/export plugin was overly monolithic.

Do you have intentions to adapt any of the full journal transfer plugin to OJS 3.x? If so, we should combine efforts.

[mjordan]

Will this change apply to OJS 3 only, or also OJS 2? I'm asking because, having consulted with PKP Legal, we don't want to store anything that was never published in the PN or even on the staging server. Which means that the filtering out of the published stuff needs to happen within the OJS plugin.

[asmecher]

Yes, this would be OJS3. Filtering would basically involve stripping out the extra entities, either via XSL or a bit of PHP. (Having my hands in a bit of XSL these days, I'll heartily recommend PHP.)

[mjordan]

@asmecher would it make sense for Dimitris to add an integration point for this now, before we release the 3.x plugin? The callback (or whatever) to do the actual filtering could then be added later (and tested as the changes to the export XML are being developed).

[asmecher]

I don't think so -- the plugin will get installed via the plugin gallery, and we can designate plugin version compatibility with each release, and updating the filter is essentially the same as updating the plugin (unless we're imagining hosting an .xsl e.g. on a PKP server, which would be a worse kind of dependency hell, I think).

[mjordan]

OK, thanks, I thought I'd ask in case it saved us some work and testing later. But just so I understand, the plugin will get installed in the Gallery initially, then included in the next release, correct? And it can be enabled either in the plugins list or directly in the Archiving tab. That probably doesn't matter but I just want to confirm that.

[asmecher]

I'd like to keep it in the Plugin Gallery throughout. We have 80 or so plugins already included in OJS and need to decrease that -- moving something to the plugin gallery is not an indication of lesser importance.

[diegoabadan]

Thanks for the info Alec!

It is not in our short term planning. We want to prioritize other plugins before.

[mjordan]

This thread in the forums is relevant. We probably should do the work to munge personal info before we release the 3.x version of the PN plugin.

[defstat]

@asmecher for reviewRounds/ReviewAssignments

PRs
pkp-lib: #3722
ojs: pkp/ojs#1978

On the same branches, coming up Queries/QueryParticipants/Notes

[KBodarwe]

Hello,
I was wondering what the status of the project is? As of 3.2 the feature does not seem to have made it into the release.
Thanks!

[asmecher]

@KBodarwe, this will be a bit of a long-term project, and we're hoping to achieve it by aligning the import/export tools with the API rather than trying to maintain parallel implementations. Is there a particular entity you're interested in supporting in the import/export toolset?

[SantiMaceri]

Hello @asmecher , i'm currently working on my thesis project, the main goal its to extend the native plugin so it can export (and import) more information than it does right now (like discussions, all the review stage, etc) for version 3.2.1-1.
I am analyzing the actual code and the xmls schemas so I can figurate out how to extend it.
Any documentation about how the plugin works (especially on filters) will be helpful.
Thanks.

[asmecher]

Hi @SantiMaceri! Thanks for your offer -- we'd like to take you up on it, but before you start, I'd like to make sure we're also not digging into technical debt where we can avoid it. Is there a specific problem you need to solve with the import/export tools that requires more entities to be added?

[SantiMaceri]

Thanks for the quick answer @asmecher .
I must clarify that im working along side @gonetil, he's my thesis professor.
We would like to give the chance to different journals to migrate a bigger part from a submission/article than they have right now.
To be more specific, we would like to add:

• The participants that work in the submission workflow

• All the discussions from all stages.

• The entire revision workflow

Currently, i'm making progress with the participants. Next, it'd be the discussions, then the revision workflow.
Again, if you have any type of documentation that speeds up de project, it'd be fantastic.
And, if in any case you want to contact us to talk about the project, its also welcome. This is a thesis project, but we also want to contribute.

[dennmuel]

Hi all!

Sorry to just bump this issue, but I stumbled upon it while looking for a way to transfer a journal from one instance to another, ideally migrating more than "just" users, issues and articles (particularly the three entities mentioned by @SantiMaceri). Has there been progress "behind the scenes" or is there any other news about this?

Best regards
Dennis

[SantiMaceri]

Hi @dennmuel !
Yes! Along side @gonetil, we managed to extend the native plugin to export the three points that i've mentioned. (It's a work in progress). In fact, we're writing the thesis about this work right now.
In the following months we'are going to test it with portals from our university, and if it's ready we're going to upload it to a public repository.

Cheers
Santiago

[dennmuel]

Thanks for getting back so quickly, @SantiMaceri! Nice to hear it, please keep me posted. Until then: good luck for your thesis! :)

[defstat]

Hi @SantiMaceri! Is there a PR that we could test/review regarding that?

[SantiMaceri]

@defstat So sorry for the late answer! Not yet but we´re actually in the final steps of the thesis! When ready, i will public the repo so you can use it as you want.
Please hang on, we are doing our best and we hope it helps!

[KBodarwe]

Hey, sorry for pulling this up again. I'm working on a similar project (exporting articles including additional data for long term storage) and wanted to ask if there's any news in this?

@SantiMaceri is there any news on your project?

if not, @asmecher do you (or anyone else for that matter) have any hints on how to register your own filters? I have found the configureObject() function in the FilterDAO class, but have no idea on how or when to use it.

Thanks!
Best regards,
Karl

[gonetil]

Hi @KBodarwe ,
yes, there are some news. We have completed the first step of the project, and even though we were unable to test it in a production environment, all tests we run in a controlled env. resulted quite ok.
It is important to note that both origin and destination OJS should be version 3.2 (we actually used ver. 3.2.1), and they must have the same configuration: sections, users and roles, review forms (it's all explain in the thesis)... Maybe someday the migration process includes all this settings, but so far we focused on submission mostly.

The thesis has been published in our institutional repository: http://sedici.unlp.edu.ar/handle/10915/120337 . It's in Spanish, I hope that's not a big issue for you.
Chapter 3 explains the solution we proposed, and chapter 4 shows all the tests we made. At the end of the document we also included the extensions we proposed to native.xsd to support new data (review rounds, review answers, review forms, discussions,...).
The code for the project is available under @SantiMaceri repostiory https://github.com/SantiMaceri/NativeXMLExtension

Let us know if this project is useful for you.
Best,
Gonzalo

[KBodarwe]

Hi @gonetil ,
that is actually very helpful, thanks. I'm not fluent in Spanish, but I can piece together the information from the code.

I see that you have extended the Native Import Export plugin. Did you add any new filters in the process? I'm trying to implement my project into a new plugin which needs new filters. Registering these into the database seems to be the problem.

Cheers,
Karl

[asmecher]

From PKP's side, we haven't made steps to improve the coverage of the XML, but our hope is that we can continue building out the REST API, adopting it from within the application as we do, then add some better command-line tools for interacting with the API outside the UI. The advantage of this is that we won't have to maintain conversion code for each entity to the import/export tools, and to the UI, and to the API -- we can ideally just do it once. That's not to say that we're deprecating the XML import/export toolset -- it'll be around for a long time -- but that we're not rushing to extend it to support more entities, as this generates a lot of brittle code the way it's currently implemented.

[KBodarwe]

Thanks @asmecher for the insight.
That seems like a long way still to go though.

If I may ask: I'm currently looking for a way to export on a per-article basis, including all submission files, comments, notes, additional files etc., into an XML format. What would be the best approach to this in your opinion? I've tried making a plugin similar to the native plugin, but can't find out when and how to register filters. I'm now trying a low-tech approach by going directly into the database (based on a shell script for example), but since submission data seems to be stored in a LOT of different tables, that sql statement could get real ugly real fast.
Any ideas for this problem? If this is the wrong place (which I'm semi-sure it is) for this question I'm happy to reformat into another place.

Thanks!
Best regards, Karl.

[asmecher]

Hi Karl -- I think you've already identified your options (if the API work is not going to be available quick enough, which seems likely):

• Work with the filter code and follow the existing practices; if you get stuck on e.g. installing filters, contact us on the forum (https://forum.pkp.sfu.ca) and we can give you a few suggestions. This would hopefully lead to your work being merged into the codebase and released with future releases. I know your list of desired entities above probably isn't complete, but you haven't listed much that isn't already at least partway there.
• Do something expedient outside of OJS -- but as you've noticed, this will grow into spaghetti pretty quickly. I think you'll need some application framing outside of pure SQL to make this workable.

I've long wondered whether there is a good, flexible, free database to XML and back again toolset, i.e. something where you could write up a descriptor to inform the tool about relationships (that aren't already described in SQL) and then turn it loose to do the work. However, I haven't been able to find anything.

[KBodarwe]

Hey @asmecher,
thanks very much for the info. I have started a thread in the PKP Forum:
https://forum.pkp.sfu.ca/t/ojs-3-3-registering-additional-filters-for-xml-export/70100

Cheers,
Karl.

[diegoabadan]

Do you have intentions to adapt any of the full journal transfer plugin to OJS 3.x? If so, we should combine efforts.

Hi @asmecher,

A Brazilian university asked us to implement this plugin for OJS 3.3, so we're going to work on it.

Do you want to offer us guidance to better combine efforts?

[asmecher]

@diegoabadan, our long-term plans are still what's described above; the problem with the current import/export and filter frameworks is that supporting additional entities generates a lot of new code, and it's not maintainable. I'd be open to a PR that adds the support, but it's not code we'd be likely to support in the future. (Note that there have been a lot of changes between 3.3.0 and 3.4.0, so forward-porting code from 3.3.0 to 3.4.0 to main will also be a lot of work.)

[diegoabadan]

our long-term plans are still what's #3261 (comment);

I agree that it is a more sustainable solution. Our problem is that the university has a more immediate need for imports, which is why a shorter term solution was studied.

Note: other universities have consulted us with the same need.

Note that there have been a lot of changes between 3.3.0 and 3.4.0, so forward-porting code from 3.3.0 to 3.4.0 to main will also be a lot of work.

Thanks! This was a concern, I had asked our team to evaluate it.

This university currently has an OJS with 20 journals, it is a typical user of LTS versions only.

[asmecher]

Relevant: https://forum.pkp.sfu.ca/t/native-xml-export-skipping-submission-files-in-ojs-3-4-0-x/82040

[diegoabadan]

We've released a preliminary package to start testing the OJS 3.3 plugin that extends what the native article/user import and export plugins do.

Development can be followed here:
https://github.com/lepidus/fullJournalTransfer

We're going to start more intensive testing now. Soon it should be ready for use in most cases.

[asmecher]

Thanks, @diegoabadan, that's likely to find a lot of interested users!

[diegoabadan]

The Federal University of São Paulo (Unifesp) confirmed that the OJS 3.3 plugin works after a journal migrated to its OJS. Our goal with this version has been achieved.

Additional features to the standard import and export of OJS users and articles:

Journal data
Navigation Menus
Plugins Configs
Sections
Review Forms
Review Assignments
Review Rounds
Review Files
Reviewer Files
Revision Files
Stage Assignments
Editor Decisions
Discussions
Metrics

[White-Horse-Press]

Sounds great, and just what we are looking for!