ActiveMQ scaler does not work with CORS enabled

[xphyr]

Report

While using Keda and the ActiveMQ scaler, I found that I am unable to connect to ActiveMQ version 5.17.0 out of the box. The error that I get is ERROR activeMQ_scaler Unable to access activeMQ management endpoint {"managementEndpoint": "pobox.xphyrlab.net:8161", "error": "ActiveMQ management endpoint response error code : 200 403"} github.com/kedacore/keda/v2/pkg/scalers.(*activeMQScaler).IsActive I have also tested version 5.16.4 of ActiveMQ and get the same results. This appears to obe related to the "CORS" configuration of Jolokia blocking the Keda application due to an Origin header that is "NULL".

The work-around is to edit the <apache-activemq-root>/webapps/api/WEB-INF/classes/jolokia-access.xml file and remove the following lines:

  <!-- Enforce that an Origin/Referer header is present to prevent CSRF -->
  <cors>
    <strict-checking/>
  </cors>

The problem goes away.

Expected Behavior

I expected that the ActiveMQ scaler would work out of the box, without needing to re-configure or remove default ActiveMQ configuration.

Actual Behavior

Keda scaller is unable to connect to ActiveMQ to monitor queue depth and thus is unable to scale target workloads.

Steps to Reproduce the Problem

1. Install ActiveMQ from https://activemq.apache.org/components/classic/download/ using defaults
2. Create a Keda ScaledObject that points to the ActiveMQ instance
3. Check the logs of the "keda-operator" and "keda-metrics-apiserver" for the error logs
4. edit <apache-activemq-root>/webapps/api/WEB-INF/classes/jolokia-access.xml and remove the CORS configuration and restart ActiveMQ
5. Review the Keda logs and see that the connection is now working

Logs from KEDA operator

keda-operator logs

.6492533349499571e+09 ERROR activeMQ_scaler Unable to access activeMQ management endpoint {"managementEndpoint": "pobox.xphyrlab.net:8161", "error": "ActiveMQ management endpoint response error code : 200 403"}
github.com/kedacore/keda/v2/pkg/scalers.(*activeMQScaler).IsActive
/workspace/pkg/scalers/activemq_scaler.go:149
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
/workspace/pkg/scaling/cache/scalers_cache.go:87
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:149
1.649253334966924e+09 ERROR activeMQ_scaler Unable to access activeMQ management endpoint {"managementEndpoint": "pobox.xphyrlab.net:8161", "error": "ActiveMQ management endpoint response error code : 200 403"}
github.com/kedacore/keda/v2/pkg/scalers.(*activeMQScaler).IsActive
/workspace/pkg/scalers/activemq_scaler.go:149
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
/workspace/pkg/scaling/cache/scalers_cache.go:92
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:149
1.6492533349671416e+09 ERROR scalehandler Error getting scale decision {"scaledobject.Name": "activemq-scaledobject", "scaledObject.Namespace": "scaller", "scaleTarget.Name": "amqpexample-consumer", "error": "ActiveMQ management endpoint response error code : 200 403"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:278

keda-metrics-apiserver

E0406 13:55:24.511506 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:55:39.988464 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:55:39.988649 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:55:55.124037 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:55:55.124135 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:56:10.413379 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:56:10.413524 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:56:25.689172 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:56:25.689410 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test

KEDA Version

2.6.1

Kubernetes Version

1.23

Platform

Red Hat OpenShift

Scaler Details

ActiveMQ

Anything else?

Using tcpdump I captured the traffic from the Keda application to ActiveMQ. The error that is returned from ActiveMQ is:

HTTP/1.1 200 OK
Date: Wed, 06 Apr 2022 13:03:02 GMT
X-FRAME-OPTIONS: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/plain;charset=utf-8
Cache-Control: no-cache
Pragma: no-cache
Expires: Wed, 06 Apr 2022 12:03:02 GMT
Transfer-Encoding: chunked

7F
{"error_type":"java.lang.Exception","error":"java.lang.Exception : Origin null is not allowed to call this agent","status":403}

I think I know a long term fix for this, and will create a followup PR that hopefully fixes this issue.

[xphyr]

So I would appreciate some feedback on this one ... as I am not a CORS expert. But from what I can tell the "Origin" field of the HTTP header needs to be set to something, even though it really doesn't apply. Looking at https://w3c.github.io/webappsec-cors-for-developers/#avoid-returning-access-control-allow-origin-null it seems to say that just using the word "NULL" is not valid, so I am actually thinking of using "http://localhost:8080", but from the testing I have done so far, it doesnt make a difference. As long as there is something in that field ActiveMQ is happy and will process the request. Open to suggestions.

[zroubalik]

CC @melisatanrverdi @arschles

[melisatanrverdi]

Hi @xphyr, ActiveMQ scaler uses Jolokia to access management endpoint of AciveMQ. It is disabled by default in jolokia-access.xml file. So yes, you should remove the CORS field to enabled the Jolokia. With the new versions of ActiveMQ remote access also disabled by default in jetty.xml file. So, you need to configure these files to use the scaler. ActiveMQ Artemis scaler also has the same situation. You can look at ConfigMap YAML of ActiveMQ scaler from its E2E test to understand how to configure the files (https://github.com/kedacore/keda/blob/main/tests/scalers/activemq.test.ts).

[xphyr]

@melisatanrverdi OK cool. While I would say that its not really disabled as much as obfuscated since you need only the anything in a "Origin" header and it works ... (at least based on the official binary package that I downloaded). It might be good to update the docs for this scaler and call out the need to update the proper ActiveMQ config files to enable this Keda scaler. If you would be ok with an update to the docs, I would be happy to write that up and submit it as a PR instead. Please advise. Thanks.

[zroubalik]

Would it make a sense to add optional parameter to tweak this setting? Does it make sense for anyone? I am ActiveMQ noob, so just asking.

[xphyr]

The more I think about this, the more I am wondering if the Origin field should be the IP address of the pod that Keda is running from. This would allow you to take advantage of the CORS configuration to limit what apps can call into Jolokia to get this data. Thoughts?

[zroubalik]

@xphyr does this work even if KEDA doesn't have a public IP address?

[xphyr]

@zroubalik yes it does. As I continue to investigate this, and dig in more ... I feel like the right "final" solution is to send the container/pod hostname as the origin. If the code does this, it fills in the Origin tag as required by jolokia, it fills it in with a hostname that could then be matched (or denied) using the CORS configuration in ActiveMQ as documented here: https://activemq.apache.org/rest#rest-management and section 4.1.5 of Jolokia https://jolokia.org/reference/html/security.html So out of the box, the Keda ActiveMQ app will work, and if you wanted to lock it down more, you could update the CORS config to only match on the host name "keda-operator-*" as the keda app will pass "keda-operator-" as the hostname.

The other option is just to document that in order for this scaler to work, you need to remove the CORS config ...

[xphyr]

@zroubalik and @melisatanrverdi sorry for a lot of back and forth on this. I think I have figured out the best (and final from my standpoint) solution to this. The artemis scaler already solved this problem with the origin header, so I have taken the same logic used in the artemis scaler and moved it into the activemq scaler. This can be seen in my fork here: https://github.com/xphyr/keda/commit/3ef2ad16b68790e339ed378f229c8cbc0ba59ae9 if this is something that would be acceptable, I can create a PR request for it. Thanks for all the help and guidance.

[zroubalik]

@xphyr thanks for digging into this. It seems like the best approach for me. Let's see what @melisatanrverdi says about this.

Also @melisatanrverdi seems like the corsHeader field in Artemis Scaler that @xphyr referenced to is missing from the docs? 🤔
docs: https://keda.sh/docs/2.6/scalers/artemis/
code:

keda/pkg/scalers/artemis_scaler.go

Lines 118 to 122 in 99619e3

	if val, ok := config.TriggerMetadata["corsHeader"]; ok && val != "" {
	meta.corsHeader = config.TriggerMetadata["corsHeader"]
	} else {
	meta.corsHeader = fmt.Sprintf(defaultCorsHeader, meta.managementEndpoint)
	}

[xphyr]

If @melisatanrverdi is OK with this change, I will update the docs for both Artemis and ActiveMQ to reference the CorsHeader and create a PR for that change as well. Thanks.

[zroubalik]

Awesome, thanks! Also, if there's some part of the code, that could be shared between those 2 scalers, please try to unify them. We have some common places for example for Azure related scalers, GCP, etc.

[xphyr]

I have created a keda-docs fork and created small updates to the docs to cover the corsHeader for both Artemis and ActiveMQ on the 2.7 docs ... https://github.com/xphyr/keda-docs/commit/00b2f1a475d96d9f66b417a01240afff8392d8ff and will create a PR for this if the proposed changes to ActiveMQ are OK.

[melisatanrverdi]

Thank you @xphyr it seems okay for me too