Upgrade 5.4 symfony and separate EntityMapper

[whikloj]

GitHub Issue: Islandora/documentation#2220

Relies on and tests:

• Update dependencies Islandora/chullo#99
• Upgrade 5.4 symfony and pull out EntityMapper Islandora/Crayfish-Commons#63
• Upgrade symfony to 5.4 & separate EntityMapper Islandora/Crayfish#174
• Remove dependency on Crayfish-Commons Islandora/islandora#936

What does this Pull Request do?

Builds a box to test the above PRs

What's new?

Updates templated files to remove unnecessary parts and work with Symfony 5.4

• Does this change require documentation to be updated? no
• Does this change add any new dependencies? no
• Does this change require any other modifications to be made to the repository
  (i.e. Regeneration activity, etc.)? no
• Could this change impact execution of existing code? no

How should this be tested?

The base box needs to be rebuilt and then you need to build the vagrant box using the starter_dev profile to pull in the correct "starter site"
i.e.

> vagrant box remove islandora_base
> ISLANDORA_BUILD_BASE=true vagrant up 
> vagrant package --output islandora_base
> ISLANDORA_BUILD_BASE=true vagrant destroy
> ISLANDORA_INSTALL_PROFILE=starter_dev vagrant up

Then you should have a normal working site, if you vagrant ssh in you should see the changes from the various PRs listed above in the code.

Interested parties

Tag (@ mention) interested parties or, if unsure, @Islandora-Devops/committers

[seth-shaw-asu]

This appears to work as advertised. Objects and terms were indexed in Fedora and derivatives were created. The only concerning thing I saw was this message in the alpaca log:

RROR 20:37:19.509 [Camel (camel-1) thread #10 - seda://mediaIndex] (FcrepoIndexer) Error indexing resource in fcrepo: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500

org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
        at org.apache.camel.component.http.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:362)
        at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:246)
        at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
        at org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:195)
        at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:317)
        at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:180)
        at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.redeliver(RedeliveryErrorHandler.java:779)
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
        at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
        at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312)
        at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:274)
        at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:187)
        at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:130)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
        Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
                ... 17 more
                Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
                        ... 17 more
                        Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
                                ... 17 more
                                Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
                                        ... 17 more
                                        Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 404
                                                at org.apache.camel.component.http.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:362)
                                                at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:246)
                                                at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
                                                at org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:195)
                                                at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:317)
                                                at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:180)
                                                at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.doRun(RedeliveryErrorHandler.java:714)
                                                at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.run(RedeliveryErrorHandler.java:623)
                                                ... 10 more

ERROR 20:37:19.526 [Camel (camel-1) thread #10 - seda://mediaIndex] (DefaultErrorHandler) Failed delivery for (MessageId: C0B1158B01FC272-00000000000000EC on ExchangeId: C0B1158B01FC272-00000000000000EE). Exhausted after delivery attempt: 6 caught: org.apache.camel.http.common.HttpOperationFailedException: HTTP opera
tion failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500. Processed by failure processor: FatalFallbackErrorHandler[Channel[log12]]

Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[FcrepoIndexerMedia] [FcrepoIndexerMedia] [from[seda://mediaIndex]                                                       ] [      5860]

Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------

org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
        at org.apache.camel.component.http.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:362)
        at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:246)
        at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
        at org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:195)
        at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:317)
        at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:180)
        at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.redeliver(RedeliveryErrorHandler.java:779)
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
        at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
        at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312)
        at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:274)
        at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:187)
        at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:130)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
        Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
                ... 17 common frames omitted
                Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
                        ... 17 common frames omitted
                        Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://127.0.0.1:8000/milliner/media/field_media_file with statusCode: 500
...trimmed...

These messages appear to be connected to these milliner log messages:

[2023-04-26T16:37:00.138167-04:00] request.INFO: Matched route "media_save". {"route":"media_save","route_parameters":{"_route":"media_save","_controller":"App\\Islandora\\Milliner\\Controller\\MillinerController::saveMedia","source_field":"field_media_file"},"request_uri":"http://127.0.0.1:8000/milliner/media/field_media_file","method":"POST"} []
[2023-04-26T16:37:00.138987-04:00] security.DEBUG: Checking for authenticator support. {"firewall_name":"main","authenticators":1} []
[2023-04-26T16:37:00.139017-04:00] security.DEBUG: Checking support on authenticator. {"firewall_name":"main","authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:00.141094-04:00] security.INFO: Authenticator successful! {"token":{"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\Token\\JWTPostAuthenticationToken":"JWTPostAuthenticationToken(user=\"admin\", authenticated=true, roles=\"authenticated, fedoraadmin\")"},"authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:00.141173-04:00] security.DEBUG: Authenticator set no success response: request continues. {"authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:01.407295-04:00] app.DEBUG: FEDORA State Token: "E70A5E5DA8B77E1B3A77A25F44E11A40" [] []
[2023-04-26T16:37:01.475256-04:00] app.DEBUG: DRUPAL URL: http://localhost:8000/_flysystem/fedora/2023-04/as11-36-5390.tiff [] []
[2023-04-26T16:37:01.475310-04:00] app.DEBUG: FEDORA URL: http://127.0.0.1:8080/fcrepo/rest/2023-04/as11-36-5390.tiff/fcr:metadata [] []
[2023-04-26T16:37:01.475331-04:00] app.DEBUG: BEFORE: {"@graph":[{"@id":"http:\/\/localhost:8000\/_flysystem\/fedora\/2023-04\/as11-36-5390.tiff","@type":["http:\/\/pcdm.org\/models#File","http:\/\/pcdm.org\/use#OriginalFile"],"http:\/\/schema.org\/author":[{"@id":"http:\/\/localhost:8000\/user\/1"}],"http:\/\/purl.org\/dc\/terms\/title":[{"@value":"Test","@language":"en"}],"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#label":[{"@value":"Test","@language":"en"}],"http:\/\/schema.org\/dateCreated":[{"@value":"2023-04-26T20:36:45+00:00","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"}],"http:\/\/schema.org\/dateModified":[{"@value":"2023-04-26T20:36:57+00:00","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"}],"http:\/\/www.loc.gov\/premis\/rdf\/v1#hasSize":[{"@value":"17644656","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#int"}],"http:\/\/pcdm.org\/models#fileOf":[{"@id":"http:\/\/localhost:8000\/node\/2"}],"http:\/\/www.ebu.ch\/metadata\/ontologies\/ebucore\/ebucore#hasMimeType":[{"@value":"image\/tiff","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#string"}],"http:\/\/www.iana.org\/assignments\/relation\/describedby":[{"@id":"http:\/\/localhost:8000\/media\/1\/edit"}]},{"@id":"http:\/\/localhost:8000\/user\/1","@type":["http:\/\/schema.org\/Person"]},{"@id":"http:\/\/localhost:8000\/node\/2","@type":["http:\/\/pcdm.org\/models#Object"]}]} [] []
[2023-04-26T16:37:01.475375-04:00] app.DEBUG: AFTER: [{"@id":"http:\/\/127.0.0.1:8080\/fcrepo\/rest\/2023-04\/as11-36-5390.tiff\/fcr:metadata","@type":["http:\/\/pcdm.org\/models#File","http:\/\/pcdm.org\/use#OriginalFile"],"http:\/\/schema.org\/author":[{"@id":"http:\/\/localhost:8000\/user\/1"}],"http:\/\/purl.org\/dc\/terms\/title":[{"@value":"Test","@language":"en"}],"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#label":[{"@value":"Test","@language":"en"}],"http:\/\/schema.org\/dateCreated":[{"@value":"2023-04-26T20:36:45+00:00","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"}],"http:\/\/schema.org\/dateModified":[{"@value":"2023-04-26T20:36:57+00:00","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"}],"http:\/\/www.loc.gov\/premis\/rdf\/v1#hasSize":[{"@value":"17644656","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#int"}],"http:\/\/pcdm.org\/models#fileOf":[{"@id":"http:\/\/localhost:8000\/node\/2"}],"http:\/\/www.ebu.ch\/metadata\/ontologies\/ebucore\/ebucore#hasMimeType":[{"@value":"image\/tiff","@type":"http:\/\/www.w3.org\/2001\/XMLSchema#string"}],"http:\/\/www.iana.org\/assignments\/relation\/describedby":[{"@id":"http:\/\/localhost:8000\/media\/1\/edit"}]}] [] []
[2023-04-26T16:37:02.580549-04:00] request.INFO: Matched route "external_save". {"route":"external_save","route_parameters":{"_route":"external_save","_controller":"App\\Islandora\\Milliner\\Controller\\MillinerController::saveExternal","uuid":"fe0adfb2-ca2f-48ab-831e-c3b5826986bf"},"request_uri":"http://127.0.0.1:8000/milliner/external/fe0adfb2-ca2f-48ab-831e-c3b5826986bf","method":"POST"} []
[2023-04-26T16:37:02.581052-04:00] security.DEBUG: Checking for authenticator support. {"firewall_name":"main","authenticators":1} []
[2023-04-26T16:37:02.581093-04:00] security.DEBUG: Checking support on authenticator. {"firewall_name":"main","authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:02.583162-04:00] security.INFO: Authenticator successful! {"token":{"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\Token\\JWTPostAuthenticationToken":"JWTPostAuthenticationToken(user=\"admin\", authenticated=true, roles=\"authenticated, fedoraadmin\")"},"authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:02.583251-04:00] security.DEBUG: Authenticator set no success response: request continues. {"authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:02.727141-04:00] request.INFO: Matched route "media_save". {"route":"media_save","route_parameters":{"_route":"media_save","_controller":"App\\Islandora\\Milliner\\Controller\\MillinerController::saveMedia","source_field":"field_media_image"},"request_uri":"http://127.0.0.1:8000/milliner/media/field_media_image","method":"POST"} []
[2023-04-26T16:37:02.727564-04:00] security.DEBUG: Checking for authenticator support. {"firewall_name":"main","authenticators":1} []
[2023-04-26T16:37:02.727594-04:00] security.DEBUG: Checking support on authenticator. {"firewall_name":"main","authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:02.733459-04:00] security.INFO: Authenticator successful! {"token":{"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\Token\\JWTPostAuthenticationToken":"JWTPostAuthenticationToken(user=\"admin\", authenticated=true, roles=\"authenticated, fedoraadmin\")"},"authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:02.733564-04:00] security.DEBUG: Authenticator set no success response: request continues. {"authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"} []
[2023-04-26T16:37:02.814634-04:00] app.ERROR: Caught exception saving media resource. {"Exception":"[object] (GuzzleHttp\\Exception\\ClientException(code: 404): Client error: `GET http://localhost:8000/media/2/edit?_format=json` resulted in a `404 Not Found` response:\n{\"message\":\"The \\u0022media\\u0022 parameter was not converted for the path \\u0022\\/media\\/{media}\\/edit\\u0022 (route nam (truncated...)\n at /var/www/html/Crayfish/Milliner/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)"} []
[2023-04-26T16:37:03.834265-04:00] request.INFO: Matched route "media_save". {"route":"media_save","route_parameters":{"_route":"media_save","_controller":"App\\Islandora\\Milliner\\Controller\\MillinerController::saveMedia","source_field":"field_media_image"},"request_uri":"http://127.0.0.1:8000/milliner/media/field_media_image","method":"POST"} []

I seem to recall seeing these errors before, so I don't know that this was introduced by these PRs, but I thought I would mention it here anyway.

[whikloj]

Yeah, those media pre-save events need some review to see if we can filter them in the Drupal side. I have seen these before and they always worry me. Then 10 days later I realize that I've been down this road and it was nothing.

Narrator: It wasn't "nothing".

[ruebot]

I updated our playbook with basically the same updates @whikloj has here, and everything seemed to work smoothly.

[rosiel]

This runs good and i have jpg, tiff, and audio derivatives being created, as well as OCR and FITS.