Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Factory: repeated Breaking Changes to an API version from 2018 #28380

Open
tombuildsstuff opened this issue Mar 21, 2024 · 6 comments
Open
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Data Factory Mgmt This issue is related to a management-plane library. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@tombuildsstuff
Copy link
Contributor

👋

There’s currently two versions of the Data Factory API:

Now firstly, I enjoyed 2018 as much as anyone else, but it was also 6 years ago at this point.. frankly I find it surprising that a service that’s still in active development is still making changes, let alone breaking changes to a six year old API version on a regular basis.

Per the ARM Specification and Breaking Change processes - breaking changes are supposed to be introduced in a new API version, but perusing the Git history for this API version, regularly make it into this existing API Version, here’s a selection from the past year:

This happens so regularly that in order to populate this list, I haven’t even clicked onto the second page of Git history!

Each of these breaking changes has been approved, meaning that it must have gone through the breaking change process - yet in the PR description there’s no explanation of the reasoning for the change.

In addition to that the Data Factory API is documented using custom format values, for example:

"type": {
  "type": "string",
  "description": "The read setting type."
},
"maxConcurrentConnections": {
  "type": "object",
  "x-ms-format": "dfe-int",
  "description": "The maximum concurrent connection count for the source data store. Type: integer (or Expression with resultType integer)."
},
"disableMetricsCollection": {
  "type": "object",
  "x-ms-format": "dfe-bool",
  "description": "If true, disable data store metrics collection. Default is false. Type: boolean (or Expression with resultType boolean)."
}

As mentioned above, fields get accidentally removed which is missed at review time despite the PR containing 10 lines of changes - and then in a follow up PR to reintroduce these a complaint is raised about the type and discarded.

This PR updates a model so that it’s no longer marked as a Discriminated Type (and all implementations) and then later reintroduces them because it turns out people were using these types.

I can’t speak to each of these Swagger PRs because unfortunately they don’t contain the full context - but my point here is that for a Stable API version from 2018 there sure are a lot of changes, particularly Breaking Changes happening.

Questions

I understand the unique nature of this API is adding new Entity Types etc, so I understand adding (and potentially deprecating) Discriminator Values. However, at the same time, I can’t help but notice that the Data Factory API seems to regularly get breaking changes approved.

Whilst some older API versions do get a free pass with regards to fixing linting issues and similar, I suspect that because there hasn’t been another API version in so long that the Data Factory API is intentionally running below the radar here.

As such:

  1. Can the ARM Team confirm what’s happening here? Why there’s been so many breaking changes to this API version?
  2. Can the Service Team confirm when a new API version will be introduced? As much as I appreciate reminiscing about 2018, it was 6 years ago at this point, so it feels like there should be a 2024 version ;)
  3. Why is this Service using custom format values? There’s a perfectly good type field in the Swagger/OpenAPI specification for this purpose, and outputting booleans as object is.. unique.
  4. Are there plans to fix this Swagger definition? In particular most of the Discriminated Types don’t define a x-ms-discriminator-value value.

Thanks!

@microsoft-github-policy-service microsoft-github-policy-service bot added question The issue doesn't require a change to the product in order to be resolved. Most issues start as that customer-reported Issues that are reported by GitHub users external to the Azure organization. labels Mar 21, 2024
@lirenhe lirenhe added Data Factory Service Attention Workflow: This issue is responsible by Azure service team. labels Mar 22, 2024
@lirenhe
Copy link
Member

lirenhe commented Mar 22, 2024

cc @mikekistler and @JeffreyRichter for awareness.

@lirenhe lirenhe added the Mgmt This issue is related to a management-plane library. label Mar 22, 2024
@konrad-jamrozik
Copy link

@tombuildsstuff re this:

Each of these breaking changes has been approved, meaning that it must have gone through the breaking change process - yet in the PR description there’s no explanation of the reasoning for the change.

We recently switched to a model where the approval labels provide general reason in them why given approval happened:

@tombuildsstuff
Copy link
Contributor Author

tombuildsstuff commented Apr 18, 2024

Another breaking change: #28538

Note the PR Description:

Screenshot 2024-04-18 at 08 31 57

@konrad-jamrozik
Copy link

@JeffreyRichter
Copy link
Member

Mike & I are talking to the Data Factory team about all these breaks to see how we can improve the customer experience.
@tombuildsstuff: Please keep reporting these and it gives the team evidence of the problems and customer pain to raise the importance of fixing their versioning strategy.

@tombuildsstuff
Copy link
Contributor Author

tombuildsstuff commented May 16, 2024

Another breaking change here: #28294 - with no details as to the need for this breaking change in the PR description/thread (despite being asked to provide that in a comment):

Screenshot 2024-05-16 at 10 30 11

But apparently this has been approved by the ARM Breaking Changes team?

Screenshot 2024-05-16 at 10 32 42

This is exposed as a hard field rename:

Screenshot 2024-05-16 at 10 31 21

Which shows for us here:

https://github.com/hashicorp/pandora/pull/4138/files#diff-a46695a73172fc26f0a3fb681b1298f7c4b002386c5f80aa4459fd6d943d568dR23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Data Factory Mgmt This issue is related to a management-plane library. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

4 participants