Log Analytics DataSource lacks proper definitions

[magodo]

Currently, different data sources' property are represented as plain JSON. It is non-trivial for the developer to digest the corresponding SDK, i.e., developer has to figure out the correct JSON format and define the model on their own.
It would be great if the swagger could define the "discriminator" for log analytics data sources.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AzmonLogA.

[yossi-y]

Thanks for your comment.
We are aware in the gap in Data Sources and inability to export templates for modifications. This is in our backlog, but we don’t have commitment nor ETA for Data Sources support in templates currently.
With that said, we have several examples for Data Sources in Log Analytics workspace Azure Resource Manager template that can be used as reference.

[magodo]

@yossi-y I'm wondering if the definition has been supported in some latest swagger spec version?

[yossi-y]

This wasn't added and in review state. As I mentioned, it's not trivial to add swaggers for datasources considering its structure and we are evaluating our options to support common datasources. We don't have ETA at this point. we have several examples for datasources that can be utilized for templates. like the link above and this one and additional examples

[magodo]

So then can we keep this issue open as it hasn't been implemented?

[yossi-y]

We have made record of the missing functionality and will consider it in near planning and therefore closed the case here. Please note that if needed, you can reopen this issue at any time.

[magodo]

I have no right to reopen this issue, would you please keep it open so that we can use it to track the progress?

[LukeCarrier]

It may be interesting for Microsoft to know that ~40 people have 👍'd a ticket for getting this into the ARM Terraform provider, which is blocked by API incompleteness: hashicorp/terraform-provider-azurerm#3182

[yossi-y]

Luke, this incompleteness evolves from the way the current Data Sources are structured and the requirement to restructure some components to get this "swaggered" properly. I don't have ETA for this work at current time.

[tombuildsstuff]

@yossi-y can we get this issue re-opened to track this bug then?

[yossi-y]

Thank you for your responses and votes. We are aware of the missing swaggers and proper ability to export Data Sources templates from Azure environments, which makes their use in templates difficult.

Although not addressing the pain, I would like to point out that the service team put effort into API consolidation in a recent stable version we released, which comprises all operationalinsight APIs (https://azure.microsoft.com/updates/azure-monitor-logs-log-analytics-rest-apis-general-availability) – we added swaggers for several APIs as part of it and updated documentations and samples (https://docs.microsoft.com/azure/azure-monitor/samples/resource-manager-workspace) accordingly. We considered the same for Data Sources, but due to the way this API were originally built where the body of each data source is different couldn’t be documented in swaggers unless separated to different APIs.

Azure Monitor has introduced a new way to collect data using data collection rules (https://docs.microsoft.com/azure/azure-monitor/platform/data-collection-rule-overview) and once GA, will be the recommended way to collect data from agents. Separating Data Sources API would have created another disruptive change and confusion given our upcoming transition to DCR and we wanted to avoid that. With the new Data Source configuration method (DCR), you can define data collection with proper swagger documentation that can be leveraged easily in templates.

Thanks,
Yossi

[yossi-y]

I suggest to close this case since progression is expected in data collection rules (https://docs.microsoft.com/azure/azure-monitor/platform/data-collection-rule-overview) and not in the APIs reported in this case.

Data collection rules is the new way to collect logs, transform it and send to selected destinations.

[tombuildsstuff]

@yossi-y that there's a new method available is great and all, but what about users of the existing functionality? To be honest it feels like this functionality isn't intended to be used outside of the Portal - whilst I appreciate it'd take time to document this functionality - if it's not documented how are users supposed to use it?

[yossi-y]

In my reply earlier, I explained that we aren't adding swaggers for data sources due to constraint and the complication involved. I also mentioned the availability of examples that can help you get started with common data sources. Although this isn't a solution, it could use as mitigation for common cases until the new data collection rules (https://docs.microsoft.com/azure/azure-monitor/platform/data-collection-rule-overview) is adopted instead, providing proper swagger coverage.

Since we aren't planning to invest in data sources and moving to the new and enhances method, I suggest to close this case at this point.

[tombuildsstuff]

@yossi-y so if there's no plans to support this going forward, is there a deprecation notice for these Data Sources/this Service?

cc @JeffreyRichter for info - this is a great example for the ARM Team

[yossi-y]

No, There is no retirement of the data sources APIs or the support for these. The only gap is the swaggers. Adding swaggers for these require major investment considering their structure and with the upcoming new data collection method, we were forced to prioritize unfortunately and this work isn't planned currently.

[tombuildsstuff]

So if these aren’t deprecated and will continue to be supported going forward, can we get this issue re-opened to keep track of this issue? Whilst I appreciate it’s work to maintain the Swagger files, this is a usability issue, if a service isn’t usable (via native types) in the Azure SDK’s - it’s not really intended to be used? As such this functionality should either be documented (which I can appreciate may take some time) - or deprecated.

[zioalex]

Any progress here? This is blocking the hashicorp/terraform-provider-azurerm#3182