Migrate AUDIT datastream

[dannylamb]

Resolves Islandora/documentation#917

Here's two new migration configs to get AUDIT files and media wired together with the rest that get brought over using migrate_7x_claw. I had to update the AuthenticatedXml plugin to extend SimpleXML instead of XML because I need full XPath to get the audit trail. I also had to hack it a little to write out the full marked up xml as a string, instead of just string values. Migrate assumes you're trying to extract text, not full nested xml.

Ping @whikloj @mjordan

[mjordan]

@dannylamb sweet, I'll test.

[dannylamb]

@mjordan That'd be great. I'd like for this to get in and then I'll adapt it to work with #10, which has some issues I still need to address.

[mjordan]

Testing now.

[mjordan]

Getting an error I can't figure out. After I checked out your branch and updated my remote IP addresses, I ran drush -y fim islandora_migrate_7x_claw_feature.

Then when I go into the GUI to run the migrations, I get an error that leaves this in my apache error log:

[Wed Dec 05 17:52:39.980935 2018] [:error] [pid 1486] [client 10.0.2.2:53144] Uncaught PHP Exception Drupal\\migrate\\MigrateException: "Islandora source plugin requires a "content_model_field" and "content_model" be defined." at /var/www/html/drupal/web/modules/contrib/migrate_7x_claw/src/Plugin/migrate/source/Islandora.php line 140, referer: http://localhost:8000/admin/structure/migrate

Doing a grep on the YAML files shows:

migrate_plus.migration.islandora_basic_image_audit_media.yml:  content_model_field: RELS_EXT_hasModel_uri_ms
migrate_plus.migration.islandora_basic_image_audit_media.yml:  content_model: 'islandora:sp_basic_image'
migrate_plus.migration.islandora_basic_image_audit.yml:  content_model_field: RELS_EXT_hasModel_uri_ms
migrate_plus.migration.islandora_basic_image_audit.yml:  content_model: 'islandora:sp_basic_image'
migrate_plus.migration.islandora_basic_image_files.yml:  content_model_field: RELS_EXT_hasModel_uri_ms
migrate_plus.migration.islandora_basic_image_files.yml:  content_model: 'islandora:sp_basic_image'
migrate_plus.migration.islandora_basic_image_media.yml:  content_model_field: RELS_EXT_hasModel_uri_ms
migrate_plus.migration.islandora_basic_image_media.yml:  content_model: 'islandora:sp_basic_image'
migrate_plus.migration.islandora_basic_image.yml:  content_model_field: RELS_EXT_hasModel_uri_ms
migrate_plus.migration.islandora_basic_image.yml:  content_model: 'islandora:sp_basic_image'

So I'm not sure why those two config settings are not being picked up. Any suggestions?

[mjordan]

Caught a user error (mine) but tried again after reloading the feature with the same results. dd()ing the $configuration variable in the Islandora source plugin, I see that content_model and content_model_field are in fact present:

Array
(
    [plugin] => islandora
    [solr_base_url] => http://97.107.189.65:8080/solr
    [fedora_base_url] => http://97.107.189.65:8080/fedora
    [data_fetcher_plugin] => http
    [authentication] => Array
        (
            [plugin] => basic
            [username] => fedoraAdmin
            [password] => fedoraAdmin
        )

    [content_model_field] => RELS_EXT_hasModel_uri_ms
    [content_model] => islandora:sp_basic_image
    [data_parser_plugin] => authenticated_xml
    [item_selector] => /foxml:digitalObject

[...]

[dannylamb]

@mjordan I only ran the migration using drush (I'm a command line junkie when doing migrate work), so maybe there's differences between the two ways of running them?

Can you try running drush migrate:import islandora_basic_image_audit --update and then drush migrate:import islandora_basic_image_audit_media --update and see what happens?

[mjordan]

Thanks, that worked.... in that I got:

[notice] Processed 20 items (20 created, 0 updated, 0 failed, 0 ignored) - done with 'islandora_basic_image_audit'
 [notice] Processed 20 items (20 created, 0 updated, 0 failed, 0 ignored) - done with 'islandora_basic_image_audit_media'

but the other migrations didn't run, so there are no nodes. I'll run those separately and see what happens.

[mjordan]

Ran drush migrate:import islandora_basic_image (no nodes created) and got a bunch of [warning] Invalid argument supplied for foreach() AuthenticatedXml.php:40 warnings. The line named in the warning is foreach ($target_element->xpath($xpath) as $value) {.

[dannylamb]

@mjordan drush cr maybe? I changed a plugin, and you may have an old one cached.

[mjordan]

Thanks, didn't do that (usually do). Do I need to run the other migrations separately or will the media ones run those for me?

[dannylamb]

Sry, in my haste I see that's for the node migration... drush cr won't cut it. Let me try it out and see what if I can reproduce. I ran my node migration first, so you starting from scratch is gonna uncover all kinds of stuff.

[mjordan]

Cool, thanks. Signing off until tomorrow.

[dannylamb]

@mjordan If you run them as a group, e.g. with the --group parameter, in my experience it seems to figure out the order to run them in.

I'm going to make a pass and formally add in dependencies so they should get figured out if you run just one.

[dannylamb]

superceded by #10