Add config option to select a different azure cloud env in the azure-eventhub input and azure module

[narph]

What does this PR do?

Adds override_resource_manager_endpoint config option to select a different azure cloud env in the azure-eventhub input and azure module.
Users should enter the resource manager endpoint in this case.
Ex:
https://management.chinacloudapi.cn/: azure.ChinaCloud,
https://management.microsoftazure.de/: azure.GermanCloud,
https://management.azure.com/: azure.PublicCloud,
https://management.usgovcloudapi.net/: azure.USGovernmentCloud,

Why is it important?

To support other azure cloud env besides the public one

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes Add support for different Azure Cloud environments in event hub input #17649

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[blakerouse]

Whats the reasoning for the prefix of override_? Couldn't it just be resource_manager_endpoint?

[kaiyan-sheng]

I have the same question as Blake. Also do we need to add this for Metricbeat?

[jsoriano]

Thanks for fixing this so quickly! Following with the comments by Blake and Kaiyan I have some suggestions about the settings and its docs.

[jsoriano]

Agree with the comments by Blake and Kaiyan, don't consider this an override, just a normal setting. Add also something about possible values. And possibly something about hybrid clouds support.

Suggested change

	==== `override_resource_manager_endpoint`
	Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.
	==== `resource_manager_endpoint`
	Optional, resource manager endpoint to use, defaults to the public environment, possible values are... . If a URL is provided, the environment is obtained from the URL, this is useful in Hybrid Cloud deployments.

Maybe we could add an explicit option for environment URLs, instead of accepting names and urls in the same option. And maybe use something like environment instead of resource_manager_endpoint, in case we use the same settings for features that need access to other endpoints.

Suggested change

	==== `override_resource_manager_endpoint`
	Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.
	==== `environment.name`
	Optional, environment to use, defaults to the public environment, possible values are... .
	==== `environment.url`
	Optional, loads the environment from a URL, this is useful in the Hybrid Cloud model. If this setting is used, `environment.name` is ignored.

[narph]

@jsoriano , the reason I went for the override- is because I do not want users to necessary specify this setting, this should be only used for other private env. To look for the azure resource manager endpoint might take some effort for the users and I want to avoid that in most of the cases where the are using the public cloud.
One workaround I would go for is to rename this option to resource_manager_endpoint but not add it in the config files so users will not be tempted to enter it if unnecessary.
Not sure what you mean by " instead of accepting names and urls in the same option" , we only accept azure resource manager endpoints, the azure-eventhub sdk can retrieve the specific environment based on that.
Also adding environment.name and environment.url might cause more confusion as users will have a hard time figuring those out, also there is no environment.url , we can talk about management_portal_URL or resource_manager_endpoint or service_endpoint but with the resource_manager_endpoint the SDK can help us identify the right env.

[jsoriano]

@jsoriano , the reason I went for the override- is because I do not want users to necessary specify this setting, this should be only used for other private env.

Other optional settings behave the same way, and don't have an override prefix, would it be different to other optional settings?

To look for the azure resource manager endpoint might take some effort for the users and I want to avoid that in most of the cases where the are using the public cloud. One workaround I would go for is to rename this option to resource_manager_endpoint but not add it in the config files so users will not be tempted to enter it if unnecessary.

Yes, I find perfectly fine to have this option in the reference docs, but not in the config files.

Not sure what you mean by " instead of accepting names and urls in the same option" , we only accept azure resource manager endpoints, the azure-eventhub sdk can retrieve the specific environment based on that.

Oh, ok, my mistake then, I saw that we accept URLs, and I thought we also accepted some kind of identifier there. One more reason to add some example values in the docs 🙂
I have still a question about that, the URLs accepted by EnvironmentFromURL, are of the same kind as the ones of the resource manager endpoints?

Also adding environment.name and environment.url might cause more confusion as users will have a hard time figuring those out, also there is no environment.url , we can talk about management_portal_URL or resource_manager_endpoint or service_endpoint but with the resource_manager_endpoint the SDK can help us identify the right env.

I was suggesting that to make configuration and its documentation easier. A user wanting to use the US Gov cloud would use environment.name: usgov, or something similar, accepted values could be in the documentation, so they don't need to look for specific endpoints in other places. A user that is using a hybrid cloud would use environment.url: http://.... with an URL that can be accepted by EnvironmentFromURL.
Something like this would be specially helpful if we need to use at some point other endpoints, e.g: the Service Manager Endpoint, so we don't have to add a setting like service_manager_endpoint.

But I don't know much about these endpoints and URLs, so as you prefer 🙂

[jsoriano]

I answer myself after reading EnvironmentFromURL source. It seems that the URL accepted by this method is actually the resource manager endpoints and from that it obtains the other endpoints, so this solves my concerns about having different kind of values for hybrid clouds.

So I think that the only pending things to consider would be:

• Rename of the setting to resource_manager_endpoint.
• Document possible values for this setting, and how it works for hybrid clouds.

[jsoriano]

Someone from the community has tried this change and it worked for them 🙂 https://discuss.elastic.co/t/azure-module-error/227210/13?u=jsoriano

[jsoriano]

👍

[jsoriano]

Nit. Should we try without the / path? So it works both for https://management.azure.com/ and https://management.azure.com.