Regression: support for the 2016/2017 API's has been removed

[tombuildsstuff]

Which version of the SDK was used?

0.3.0

Which platform are you using? (ex: Windows, Linux, Debian)

n/a

What problem was encountered?

Prior to 0.3.0 the Azure Storage SDK for Go supported allowing the user to decide which version of the API should be used. This option is no longer available and users are required to use the 2018 API's which are not available in all regions - which unfortunately this approach means it's not possible to adopt version 0.3.0 of the Azure Storage SDK for Go.

The Azure SDK for Go has adopted a versioning strategy based on discussion with the community where multiple versions of the API are available, since not all versions of the Storage API's are available in all Azure Regions/Sovereign Clouds (incl. Azure Stack) - which allows users to pick the API versions which are most appropriate for their needs.

How can we reproduce the problem in the simplest way?

Navigate to https://github.com/Azure/azure-storage-blob-go and notice that the following paths are no longer present:

• /2016-05-31/azblob
• /2017-07-29/azblob

Whilst shipping only a single API version allows for simpler maintenance of this SDK (and some simpler uses of this SDK) - unfortunately this means it's not possible to adopt it. For reference the Azure SDK for Go consulted on the API versioning strategy - which is where the support for multiple versions came from; in order to make the day 1 support easier - the Go SDK supports a 'latest' profile which points to the latest API version - which allows both use cases to be supported.

On a separate note - whilst perhaps I'd missed the Github issue for this - was this major functionality removal discussed with the community/users of this SDK prior to being made?

Have you found a mitigation/solution?

Continuing to use the Storage SDK in the Azure SDK for Go - which is deprecated.

[zezha-msft]

Hi @tombuildsstuff, thanks for reaching out! I see that there's some misunderstanding concerning our intention behind deleting the 2016 and 2017 folders.

To start, I would like to clarify that our plan is still to maintain multiple service versions simultaneously, but we will be doing it in a slightly different but more efficient way. Please allow me to explain.

Before Go.11, we had to maintain separate directories for each service version so that users can side-load different versions. The main purpose was to allow them to gradually upgrade their dependency on the SDK, and more importantly be able to target different environments like Azure Stack. However, since module support was released in Go 1.11(to be finalized in Go 1.12), it is no longer a requirement to have separate directories in the repo for this purpose; users can easily load multiple (major) versions of the SDK simultaneously. And since each major version will correspond to a particular service version(please refer to the section New Storage SDK versions in this article), once the preview period ends, therefore we can instead have a single azblob package in the repo. This change has many advantages:

• The versioning story is much simpler: we can use a single semantic version to refer to a particular version of the SDK, since each major version will directly correspond to a different service version.
• We'll still be able to maintain the older versions, and it might be much more efficient since we can leverage git to cherry-pick commits between version branches instead of comparing files manually.
• It's a much better experience for the users, as having many directories named with dates is very unusual and confusing.
• Also, there's no need to maintain "profiles", which have many caveats.

Another reason for why we removed the 2016-05-31 and 2017-07-29 folders is because they were never brought up to the same quality as the 2018-03-28 version, since we were going through massive changes all the time due to being in preview, it was untenable to maintain 3 versions when we were still tackling technical issues. In addition, the 2018-03-28 version had the most up-to-date and accurate Swagger spec, so it's natural to go GA with it. In fact, all the new versions of the Storage SDKs will GA with version 10, supporting the 2018-03-28 service version.

I certainly understand your concern that not all Azure environments support the 2018-03-28 version yet. But soon they will, and in the mean time, the users can still access the other preview versions(please refer to the version table). Alternatively, they could use the latest version, and override the x-ms-version header with a custom policy(example).

We already started communicating the new versioning strategy of the Storage SDKs, and we intent to document it in more places. Please don't hesitate if you have any question or concern.

FYI when I discussed versioning strategies with @marstr, my understanding is that azure-sdk-for-go and the Storage SDKs are not exactly in the same situation. Instead of shipping Autorest-generated code directly, we have a convenience layer wrapping the protocol layer. Doing so shields the users from the updates of the generator; and we also have extensive testing to make sure the APIs are functioning as expected, so there's less occasions where the older versions need to be updated(besides serious bugs). Therefore I do not think that their versioning strategy automatically maps to all Go projects in Azure.

[paultyng]

@zezha-msft when you say:

I certainly understand your concern that not all Azure environments support the 2018-03-28 version yet. But soon they will

Are you including Azure Stack? what would be your recommendation for handling the potential different installs that could be in the wild that code could run against? The version table doesn't seem to show any additional major versions yet, so we wouldn't be able to do any loading of multiple versions (0.2 and 0.3) side by side.

[zezha-msft]

Hi @paultyng, you are right, this SDK is still in preview so there's no other major version yet. The first official version of this new SDK will be 10.0.0, and supporting the 2018-03-28 service version. This will be consistent across all Storage SDKs.

As for users who target other environments, our suggestion is:

Alternatively, they could use the latest version, and override the x-ms-version header with a custom policy(example).

In general, newer Storage service versions are additive, so it should be possible to simply replace the x-ms-version of the newest SDK to target an older version of the Storage service. However, the user is responsible for testing whether there's any unexpected behavior.

Please let me know if you have any question or concern.

[tombuildsstuff]

@zezha-msft

Hi @paultyng, you are right, this SDK is still in preview so there's no other major version yet. The first official version of this new SDK will be 10.0.0, and supporting the 2018-03-28 service version. This will be consistent across all Storage SDKs.

TBH going from 0.3.0 -> 10.0.0 seems a little extreme - I'm not sure I understand the desire to keep all versions of the SDK at the same version since they'll obviously deviate over time (for example does a version bump in the .net storage SDK to fix a null-reference cause the Golang SDK version to be bumped?) - but whatever :)

---

We already started communicating the new versioning strategy of the Storage SDKs, and we intent to document it in more places. Please don't hesitate if you have any question or concern.

Cool - glad to hear it; unfortunately I believe I missed this consultation - would it be possible to provide some links to Github issues where this was discussed in advance of it happening?

---

To start, I would like to clarify that our plan is still to maintain multiple service versions simultaneously, but we will be doing it in a slightly different but more efficient way. Please allow me to explain.

Unfortunately neither of these options are really viable solutions - let me go through both in turn.

I certainly understand your concern that not all Azure environments support the 2018-03-28 version yet. But soon they will, and in the mean time, the users can still access the other preview versions(please refer to the version table).

Using an older version of this SDK could possibly work if it weren't for all the Panic's in versions prior to 0.3.0 (however that means we'd still end up with different codepaths in the future for different versions of this SDK and it's dependencies, which is undesirable since this pushes this logic into consumers of this SDK).

Alternatively, they could use the latest version, and override the x-ms-version header with a custom policy(example).

Again, this could work - but it's undesirable insofar as it's pushing the complexity/testing requirements outside of this SDK and into consumers of the SDK. As you've mentioned there's also no guarantee all fields are supported for all versions of the SDK - and as such we'd end up having to test this in client applications (and keep track of these new fields).

As such, whilst I understand it's more work in the SDK to support this in the short-term (since it requires some refactoring) - I believe it's necessary for the SDK to be architected in such a way that it supports multiple versions of the Storage API's within a single version of the SDK. I believe this is due to the way that API rollouts happen across all the Azure clouds (Public/Government/Germany/China/Stack - which roughly get deployed in that order) - and whilst the 2018 API's may be rolled out soon, the next version may not appear on Azure Stack for another year; which pushes this complexity into client applications, rather than in this SDK.

Unfortunately we're not the only consumers of this SDK who rely on the older API versions - such that I don't believe dropping support for this API version is possible at this time (since it's not available on Azure Stack instances).

Thanks!

[zezha-msft]

Hi @tombuildsstuff, thanks for providing feedbacks!

TBH going from 0.3.0 -> 10.0.0 seems a little extreme - I'm not sure I understand the desire to keep all versions of the SDK at the same version since they'll obviously deviate over time (for example does a version bump in the .net storage SDK to fix a null-reference cause the Golang SDK version to be bumped?) - but whatever :)

0.3.0 -> 10.0.0 is indeed extreme, but the purpose is to clearly indicate that we are talking about the newly designed SDKs. And to clarify, only the major version of the SDKs in different languages will correlate, as they all use the same service version, but the minor and patch versions won't. In other words, breaking changes in the SDKs will only be introduced with a new service version, which is also a new major version, but minor changes and bug fixes can be released at any time. I believe the new strategy is well defined, and while not perfect, provides numerous benefits such as easier communication and troubleshooting; in addition, it also sets clear expectations for when breaking changes can be introduced. Please let me know if you still have concerns.

Cool - glad to hear it; unfortunately I believe I missed this consultation - would it be possible to provide some links to Github issues where this was discussed in advance of it happening?

I certainly understand your frustration, and I apologize for our lack of consideration. I'll definitely make sure that we do that in the future once we GA.

Using an older version of this SDK could possibly work if it weren't for all the Panic's in versions prior to 0.3.0 (however that means we'd still end up with different codepaths in the future for different versions of this SDK and it's dependencies, which is undesirable since this pushes this logic into consumers of this SDK).

I'm a bit confused here. Once the 2018-03-28 version is rolled out across every environment, users won't need different codepaths.

To clarify, with the previous strategy, the import paths are like so:

import (
  azblob16  "github.com/Azure/azure-storage-blob-go/2016-05-31/azblob"
  azblob17  "github.com/Azure/azure-storage-blob-go/2017-07-29/azblob"
)

In the future, they would be:

import (
  azblob10  "github.com/Azure/azure-storage-blob-go/v10/azblob"
  azblob11  "github.com/Azure/azure-storage-blob-go/v11/azblob"
)

Unfortunately we're not the only consumers of this SDK who rely on the older API versions - such that I don't believe dropping support for this API version is possible at this time (since it's not available on Azure Stack instances).

I understand your concern about the SDK dropping support for the 2016 and 2017 versions. I'll discuss this internally and get back to you.

[landro]

Any updates on this, @zezha-msft ?

[zezha-msft]

@landro thanks for following up on this issue.

We have decided not to pour more efforts into 2016 and 2017 packages. The SDK will go GA with 2018-03-28.