Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chop off charset from mimetype for external content header #131

Merged
merged 1 commit into from
Oct 22, 2021

Conversation

whikloj
Copy link
Member

@whikloj whikloj commented Oct 14, 2021

GitHub Issue: Islandora/documentation#1918

What does this Pull Request do?

Checks for a ; in the Content-type header and if it exists, then splits and uses the first part.

Additionally we we're referencing $headers in createMediaVersion and it was not defined.

Also fixed a bad @covers statement

How should this be tested?

Create a new object, add some media. Check the Milliner logs for the error message like

[2021-10-14 15:41:45] app.ERROR: {"Exception":"[object] (RuntimeException(code: 400): Client error: PUT http://127.0.0.1:8080/fcrepo/rest/87/3a/d3/68/873ad368-b833-4a95-b1d5-79bdaf05d239 resulted in a 400 Bad Request response: External content link header url is malformed at /home/vagrant/all_claw/Crayfish/Milliner/src/Service/MillinerService.php:499)"} []

Also there is no record for the media in Fedora.

Pull in this PR and try again.

Interested parties

@Islandora/8-x-committers

@ruebot
Copy link
Member

ruebot commented Oct 15, 2021

Tested and can confirm the original 400 errors I was getting are gone. But, I have two other errors now.

[2021-10-15 07:49:50] app.ERROR:  {"Exception":"[object] (GuzzleHttp\\Exception\\ClientException(code: 404): Client error: `HEAD http://10.0.0.16:8080/fcrepo/rest/ca/94/71/41/ca947141-a447-4a43-a70c-aaf11eb86b48` resulted in a `404 Not Found` response at /var/www/html/Crayfish/Milliner/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)"} []
[2021-10-15 07:49:50] app.ERROR:  {"Exception":"[object] (RuntimeException(code: 400): Client error: `PUT http://10.0.0.16:8080/fcrepo/rest/64/90/9d/80/64909d80-ea3d-43c7-9ebf-abb89b474ee5` resulted in a `400 Bad Request` response: External content is disallowed by the server at /var/www/html/Crayfish/Milliner/src/Service/MillinerService.php:501)"} []

@ruebot
Copy link
Member

ruebot commented Oct 15, 2021

If it helps, here's what's happening in Karaf:

2021-10-15 08:51:23,602 | ERROR | ng-fcrepo-media] | DefaultErrorHandler              | 57 - org.apache.camel.camel-core - 2.20.4 | Failed delivery for (MessageId: queue_islandora-indexing-fcrepo-media_ID_beta-38173-1634168442786-3_1447_-1_1_2 on ExchangeId: ID-beta-1634168561770-2-23470). Exhausted after delivery attempt: 11 caught: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://10.0.0.16/milliner/media/field_media_image with statusCode: 404. Processed by failure processor: FatalFallbackErrorHandler[Channel[Log(ca.islandora.alpaca.indexing.fcrepo.FcrepoIndexer)[Error indexing resource in fcrepo: ${exception.message}

${exception.stacktrace}]]]

Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[FcrepoIndexerMedia] [FcrepoIndexerMedia] [broker://queue:islandora-indexing-fcrepo-media                                ] [     10903]
[FcrepoIndexerMedia] [unmarshal11       ] [unmarshal[org.apache.camel.model.dataformat.JsonDataFormat@34037cef]          ] [         0]
[FcrepoIndexerMedia] [setProperty38     ] [setProperty[event]                                                            ] [         0]
[FcrepoIndexerMedia] [setProperty39     ] [setProperty[sourceField]                                                      ] [         1]
[FcrepoIndexerMedia] [setProperty40     ] [setProperty[jsonUrl]                                                          ] [         0]
[FcrepoIndexerMedia] [setProperty41     ] [setProperty[fedoraBaseUrl]                                                    ] [         0]
[FcrepoIndexerMedia] [log20             ] [log                                                                           ] [         0]
[FcrepoIndexerMedia] [removeHeaders11   ] [removeHeaders[*]                                                              ] [         0]
[FcrepoIndexerMedia] [setHeader30       ] [setHeader[CamelHttpMethod]                                                    ] [         0]
[FcrepoIndexerMedia] [setHeader31       ] [setHeader[Content-Location]                                                   ] [         0]
[FcrepoIndexerMedia] [setHeader32       ] [setHeader[X-Islandora-Fedora-Endpoint]                                        ] [         0]
[FcrepoIndexerMedia] [setBody11         ] [setBody[simple{Simple: ${null}}]                                              ] [         0]
[FcrepoIndexerMedia] [toD16             ] [                                                                              ] [     10902]
[FcrepoIndexerMedia] [log16             ] [log                                                                           ] [         2]

Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://10.0.0.16/milliner/media/field_media_image with statusCode: 404
	at org.apache.camel.component.http4.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:311)[116:org.apache.camel.camel-http4:2.20.4]
	at org.apache.camel.component.http4.HttpProducer.process(HttpProducer.java:206)[116:org.apache.camel.camel-http4:2.20.4]
	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:112)[58:org.apache.camel.camel-jms:2.20.4]
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:721)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:681)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:313)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:251)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1164)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1156)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1053)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_292]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_292]
	at java.lang.Thread.run(Thread.java:748)[:1.8.0_292]
2021-10-15 08:51:23,603 | WARN  | ng-fcrepo-media] | EndpointMessageListener          | 57 - org.apache.camel.camel-core - 2.20.4 | Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException - org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://10.0.0.16/milliner/media/field_media_image with statusCode: 404]
org.apache.camel.RuntimeCamelException: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://10.0.0.16/milliner/media/field_media_image with statusCode: 404
	at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1831)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:195)[58:org.apache.camel.camel-jms:2.20.4]
	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:116)[58:org.apache.camel.camel-jms:2.20.4]
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:721)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:681)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:313)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:251)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1164)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1156)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1053)[76:org.apache.servicemix.bundles.spring-jms:4.2.8.RELEASE_1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_292]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_292]
	at java.lang.Thread.run(Thread.java:748)[:1.8.0_292]
Caused by: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://10.0.0.16/milliner/media/field_media_image with statusCode: 404
	at org.apache.camel.component.http4.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:311)[116:org.apache.camel.camel-http4:2.20.4]
	at org.apache.camel.component.http4.HttpProducer.process(HttpProducer.java:206)[116:org.apache.camel.camel-http4:2.20.4]
	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)[57:org.apache.camel.camel-core:2.20.4]
	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:112)[58:org.apache.camel.camel-jms:2.20.4]
	... 11 more

I don't see anything in fcrepo though, and I'm not 100% if this is all related to the original issue you created @whikloj.

@whikloj
Copy link
Member Author

whikloj commented Oct 15, 2021

@ruebot Could you check your External Content setup. I'm guessing now that the external requests are working your Fedora is not allowing the hostname.

@ruebot
Copy link
Member

ruebot commented Oct 15, 2021

@whikloj ah, reading that page again made me catch something;

This example allows external content references to local files at any path within /allowed/path/ and /another/allowed/path/, as well as the specific file /exact/match/file.txt.

I didn't have trailing slashes on any of my urls in allowed-external-content.txt.

All good now. Thank you taking care of this PR, and helping me out 🙏

@elizoller
Copy link
Member

@whikloj since @ruebot already approved this, can i merge? or is it still waiting for something?

@elizoller
Copy link
Member

i'm not entirely sure why the tests didn't run?

@whikloj
Copy link
Member Author

whikloj commented Oct 21, 2021

@elizoller tests didn't run because GH Actions are still looking at dev, @rosiel made the necessary changes in this PR. So it needs to be reviewed and merged.

@whikloj
Copy link
Member Author

whikloj commented Oct 21, 2021

...and the PR is #130 🤦

@elizoller
Copy link
Member

gotcha, i will test #130 then

@elizoller
Copy link
Member

@whikloj i don't see a way to give this a poke to make the tests run, do you?

@codecov
Copy link

codecov bot commented Oct 22, 2021

Codecov Report

Merging #131 (6882984) into 2.x (ee9ea25) will decrease coverage by 0.80%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##                2.x     #131      +/-   ##
============================================
- Coverage     77.55%   76.74%   -0.81%     
- Complexity      159      161       +2     
============================================
  Files             6        6              
  Lines           655      658       +3     
============================================
- Hits            508      505       -3     
- Misses          147      153       +6     
Impacted Files Coverage Δ
...dir/Milliner/src/Controller/MillinerController.php 95.40% <0.00%> (-4.60%) ⬇️
build_dir/Milliner/src/Service/MillinerService.php 70.58% <0.00%> (-0.42%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee9ea25...6882984. Read the comment docs.

Copy link
Member

@elizoller elizoller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the coverage didn't change on the patch for this PR, its only complaining about the coverage on the project as a whole so i think this is still ok to merge. i also gave it a test.

@elizoller elizoller merged commit 09a62a4 into Islandora:2.x Oct 22, 2021
@whikloj whikloj deleted the issue-1918 branch October 22, 2021 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants