Proposal: Versioning of provider artifacts and provider resources

[asilverman]

TL;DR

• We propose to relax the constraint on provider versions so that they are not required to conform to semantic versioning.

• We propose the provider version to be defined as any valid OCI tag descriptor

• We propose that the underlying type versions for the resource types are a concern of the provider author

• We believe that doing the above will improve the user experience by making it self-evident for the user what types are loaded from a provider declaration syntax.

Overview

The current implementation of Bicep describes the ability to import type namespaces and Bicep extensibility providers using the following syntax per the Bicep documentation: Import Bicep namespaces

import '[email protected]'
import '[email protected]'
import '[email protected]'

The syntax object used to parse the statements above constrains the version to the right of the '@' character to be a semantic version as enforced by the code below.

bicep/src/Bicep.Core/Syntax/ImportSpecification.cs

Lines 23 to 30 in ac299a3

	// Regex copied from https://semver.org/.
	// language=regex
	private const string SemanticVersionPattern = @"(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?";
	private static readonly Regex SpecificationPattern = new(
	@$"^(?<name>{NamePattern})@(?<version>{SemanticVersionPattern})$",
	RegexOptions.ECMAScript | RegexOptions.Compiled);

bicep/src/Bicep.Core/Syntax/ImportSpecification.cs

Lines 65 to 78 in ac299a3

	private static (string Name, string Version, bool IsValid) Parse(string value)
	{
	var match = SpecificationPattern.Match(value);
	if (!match.Success)
	{
	return (LanguageConstants.ErrorName, LanguageConstants.ErrorName, false);
	}
	var name = match.Groups["name"].Value;
	var version = match.Groups["version"].Value;
	return (name, version, true);
	}

We propose to relax the enforcement of the import version and delegate the concern of versioning to provider developers, in the sections below we will provide the motivation for doing this and how this change results in a better overall user experience for resource type provider authors and Bicep file creators alike.

Motivation

In the following sections we will describe the limitations with the current design using examples.

Example 1: The Azure provider

In the import statement currently used to load the azure types provider (import '[email protected]') the version ('1.0.0') is a placeholder. The versioning of the types is reflected in the NuGet package reference version (from Bicep.Core.csproj), as shown below:

bicep/src/Bicep.Core/Bicep.Core.csproj

Line 43 in ac299a3

<PackageReference Include="Azure.Bicep.Types.Az" Version="0.2.552" />

The definitions are generated via a console application (azure/bicep-types-az ) that is triggered manually for each Bicep release. The NuGet package version is then updated to reflect that the new types are a result of that generation event.

Inspection of the underlying types reveals that they follow the Azure REST API versioning specification, this specification defines a version as a date(e.g. 2023-11-01-preview, 2023-01-10).

This creates a challenge for our customers since any semantic version of the azure types provider doesn't hint about what versions will be supported in the types package.

Proposed Solution

In the case of the azure types provider, we propose to use the date when the package was generated as a version (matching the Azure Rest Specs pattern), this way the costumer will know that the versions available in that package are all the versions defined in the azure-rest-api-specs repository at the time of generation.

Such a versioning scheme is supported by the OCI distribution model and can be represented in the form of an OCI Tag. For example:

mcr.microsoft.com/bicep/providers/az:2023-01-10-preview

import 'az@2023-08-22' as az

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-07-01' = {}

Example 2: The Kubernetes Provider

In the import statement currently used to load the Kubernetes types provider (import '[email protected]') the version ('1.0.0') is a placeholder. The versioning of the types is reflected in the NuGet package reference version (from Bicep.Core.csproj), as shown below:

bicep/src/Bicep.Core/Bicep.Core.csproj

Line 44 in ac299a3

<PackageReference Include="Azure.Bicep.Types.K8s" Version="0.1.504" />

The actual definitions are generated are derived from a Kubernetes OpenApi spec (see here). These type definitions for Kubernetes are updated to v1.25.3 of Kubernetes and are not updated in between releases of Bicep.

These constraints customers to declare manifests that are compatible with Kubernetes clusters with versions v1.23 to v1.28 and there is no compatibility guarantee once Kubernetes versions evolve.

The customer has no hint about this potential problem since the version in the provider ('1.0.0') is a placeholder.

We propose that the Kubernetes provider is versioned using the versioning scheme of Kubernetes (e.g. '1.25.3', '1.28.0', etc...)

The types would match the OpenAPI specification for that release of Kubernetes and the underlying resource types would be versioned according to the Kubernetes API Server scheme (v1, v1beta1, v1alpha1, etc..). Its worth noting that such a version would also be OCI compliant, for example

mcr.microsoft.com/bicep/providers/kubernetes:1.25.3

@secure()
param kubeConfig string

import '[email protected]' with {
  namespace: 'default'
  kubeConfig: kubeConfig
} as k8s

resource t 'flowcontrol.apiserver.k8s.io/FlowSchema@v1beta1' = {}

resource tt 'flowcontrol.apiserver.k8s.io/FlowSchema@v1beta2' = {}

Example 3: The Microsoft Graph Provider

WIP

[majastrz]

I think versioning az using dates to match how the types within it are versioned makes sense as an option. Dates are a lot easier to read by users than Git commit SHAs but can convey similar concepts and are naturally sequential (unlike the SHAs). We will need a deterministic rule how commits are chosen for a specific date. For example, generating a provider for 2023-09-15 at different times of the day should produce a provider with the same contents.

During our past discussions, one of our goals was to bring the idea of semantic versioning to the az provider to make it more obvious to the user that they should expect breaking changes or major feature additions. Unfortunately, I don't think we have any practical options for converting Swagger commit IDs into a semver version without also having to solve the problem of classifying each diff as breaking, non-breaking, major, minor feature, patch, etc.

I think this option side steps the problem nicely. Are there any other option for us to consider or do you feel this is the only viable one?

[majastrz]

Regarding the Kubernetes part, I don't have an objection to matching the k8s versioning conventions. However, the 1.0.0 placeholder is due to the provider being baked into Bicep. Are we planning to switch that provider to pull it dynamically as well?

[majastrz]

Would be good to also get @shenglol's thoughts since he's working on updates to the import syntax.

[shenglol]

Yeah, we had offline discussions.

In the case of the azure types provider, we propose to use the date when the package was generated as a version (matching the Azure Rest Specs pattern), this way the costumer will know that the versions available in that package are all the versions defined in the azure-rest-api-specs repository at the time of generation.

This makes sense. However, I just realized this has one drawback. In the future, the Azure provider version may also define what az functions are available. We may use it to deprecate functions or make breaking changes. If we take that into consideration, semantic versioning makes more sense.

[anthony-c-martin]

This proposal does attempt to address the problem of being able to quickly understand which Az resource types are available in a given Az package, but I have the following concerns:

• In the long run, I feel that consistency is extremely important. If we imagine a world where we have 100s of providers available, each with their own versioning scheme and rules, I think the experience would be very confusing for users.
• This is a heavily Azure-centric solution - there's no guarantee the same discoverability benefit would apply to other providers.
• There are issues with practicality (for example, what happens if a provider misses publishing an API version 2022-01-01, and our 2022-01-01 package misses this type, but it is then available in 2022-02-01? - or vice versa with API deprecation).
• In practice, wouldn't people generally just want to be on "latest" for Azure? Is being able to be accurate with version targeting really as valuable as we think it is?
• The point Shenglong raised about being able to reason about breaking changes using semver.

In general, I'm not in favor of this proposal over using semver. Consistency is really my most important concern, but I also feel like we're over-optimizing on solving a particular (very Azure-centric) problem without fully understanding the details - for example, I would want to ask the question "Why do we feel we can't solve this problem with tooling or documentation?".

That said, I fully agree version/type discoverability is an important problem to solve, and appreciate the thought you've put into it here.

[majastrz]

Assuming we're not talking about a degenerate form of semver ("every version is a major version"), I agree that it communicates the types of changes you have from version to version very well. How are we going to implement it in a scalable way?

For example, let's say we're publishing types daily and the head of https://github.com/Azure/azure-rest-api-specs/commits/main changed from Azure/azure-rest-api-specs@ad6484f to Azure/azure-rest-api-specs@b646a42. What is the version increment we calculate? And more importantly how?

[shenglol]

Here's my thought on how to use semantic versioning for the az provider:

• PATCH version can be calculated based on the git commits of the https://github.com/azure/bicep-types-az repo instead of https://github.com/Azure/azure-rest-api-specs. PATCH version should be updated if:
  • New API versions are added*
  • Existing API versions are updated*
• MINOR version should be bumped if:
  • New resource types are added*
  • New az template functions are added
• MAJOR version should be bumped if there are resource types
  • Resource types or API versions are deleted*
  • There are breaking changes to az template functions

*: Can be detected with an automated CI pipeline

[asilverman]

Here's my thought on how to use semantic versioning for the az provider:

• PATCH version can be calculated based on the git commits of the https://github.com/azure/bicep-types-az repo instead of https://github.com/Azure/azure-rest-api-specs. PATCH version should be updated if:

  • New API versions are added*
  • Existing API versions are updated*

• MINOR version should be bumped if:

  • New resource types are added*
  • New az template functions are added

• MAJOR version should be bumped if there are resource types

  • Resource types or API versions are deleted*
  • There are breaking changes to az template functions

*: Can be detected with an automated CI pipeline

I think that although it seems technically feasible to do this, I worry that its not self evident what are the consequences of updating a provider version are and also what api-versions belong to a given package.

Also, although technically feasible, from a time/resource investment standpoint it seems like a big lift to implement this detection and also to maintain it. Does it make sense for us to incur in this cost in consideration to its benefits compared to using dates?

[asilverman]

Yeah, we had offline discussions.

In the case of the azure types provider, we propose to use the date when the package was generated as a version (matching the Azure Rest Specs pattern), this way the costumer will know that the versions available in that package are all the versions defined in the azure-rest-api-specs repository at the time of generation.

This makes sense. However, I just realized this has one drawback. In the future, the Azure provider version may also define what az functions are available. We may use it to deprecate functions or make breaking changes. If we take that into consideration, semantic versioning makes more sense.

With regards to this potential future problem, I think that we should not optimize for this yet. I think eventually upgrading those az specific functions could be handled by decoupling further the az provider into smaller packages where az would have the base azure provider types and function definitions and the resource provider specific type definitions are handled in their own provider.

For example

import '[email protected]' as az
import 'Microsoft.Storage@2020-08-01-preview' as ms-storage

var location = az.resourceGroup().location
resource t 'ms-storage/storageAccounts' = {
  location: location
...
}

The main thing here is that revving up azure functions that currently are hardcoded into the compiler should be discussed in a separate topic IMO

[asilverman]

Are we planning to switch that provider to pull it dynamically as well?

Yes, we should do this regardless of wether we are using semVer or dates. I will create an issue to follow-up in this topic

• The Bicep Language Server should provide completions for import 'az@ by inspection of the OCI registry #11610

[asilverman]

I think this option side steps the problem nicely. Are there any other option for us to consider or do you feel this is the only viable one?

I think that its natural for us to follow the versioning scheme that the resource providers use instead of introducing our own versioning on top of that for the purpose of consistency at the cost of user experience, users think of versions for these resources in the form of api-version so I think its most helpful for them to use the same structure when they are creating the Bicep files.

If we use the dates, we could also simplify the declaration of resources as described in #11593 (comment). Since there would be no constraint regarding import of multiple versions it would be acceptable to have something like the below:

The advantages are:

• Less stuff to type for authors
• Less stuff to download on provider restore (because we would download only relevant types)
• Faster compilation times (as a consequence of the above)
• Governance of provider packages can be delegated to the resource provider teams

import '[email protected]' as az
import 'Microsoft.Storage@2020-08-01-preview' as ms-storage.
import 'Microsoft.Storage@2023-10-10' as ms-storage2

var location = az.resourceGroup().location
resource t 'ms-storage/storageAccounts' = {
  location: location
...
}

resource tt 'ms-storage2/storageAccounts' = {
  location: location
...
}

[shenglol]

Yeah, we had offline discussions.

In the case of the azure types provider, we propose to use the date when the package was generated as a version (matching the Azure Rest Specs pattern), this way the costumer will know that the versions available in that package are all the versions defined in the azure-rest-api-specs repository at the time of generation.

This makes sense. However, I just realized this has one drawback. In the future, the Azure provider version may also define what az functions are available. We may use it to deprecate functions or make breaking changes. If we take that into consideration, semantic versioning makes more sense.

With regards to this potential future problem, I think that we should not optimize for this yet. I think eventually upgrading those az specific functions could be handled by decoupling further the az provider into smaller packages where az would have the base azure provider types and function definitions and the resource provider specific type definitions are handled in their own provider.

For example

import '[email protected]' as az
import 'Microsoft.Storage@2020-08-01-preview' as ms-storage

var location = az.resourceGroup().location
resource t 'ms-storage/storageAccounts' = {
  location: location
...
}

The main thing here is that revving up azure functions that currently are hardcoded into the compiler should be discussed in a separate topic IMO

I'm not sure if it's just an optimization for the future. We already have this problem today where we cannot make breaking changes to fix some of the existing ARM template functions. Besides, since we are designing a feature for the future, we should consider make it as future-proof as possible.

[shenglol]

I think this option side steps the problem nicely. Are there any other option for us to consider or do you feel this is the only viable one?

I think that its natural for us to follow the versioning scheme that the resource providers use instead of introducing our own versioning on top of that for the purpose of consistency at the cost of user experience, users think of versions for these resources in the form of api-version so I think its most helpful for them to use the same structure when they are creating the Bicep files.

If we use the dates, we could also simplify the declaration of resources as described in #11593 (comment). Since there would be no constraint regarding import of multiple versions it would be acceptable to have something like the below:

The advantages are:

• Less stuff to type for authors
• Less stuff to download on provider restore (because we would download only relevant types)
• Faster compilation times (as a consequence of the above)
• Governance of provider packages can be delegated to the resource provider teams

import '[email protected]' as az
import 'Microsoft.Storage@2020-08-01-preview' as ms-storage.
import 'Microsoft.Storage@2023-10-10' as ms-storage2

var location = az.resourceGroup().location
resource t 'ms-storage/storageAccounts' = {
  location: location
...
}

resource tt 'ms-storage2/storageAccounts' = {
  location: location
...
}

IMHO we should avoid the merging of importing control plane / API providers and importing Azure resource providers. These two concepts have distinct purposes, with the latter primarily focused on simplifying resource declarations.

• Less stuff to type for authors: While reducing the code burden for authors makes sense, there are alternative strategies / syntaxes to achieve this goal. I think there is an existing issue that covers this, and we should discuss it within the relevant issue (Do *something* to make it easier to deal with resource types & api versions #622).
• Less stuff to download on provider restore: The benefits of download optimization and faster compilation times are valuable, but these could be seen as implementation details rather than inherent advantages of the resource provider import syntax. For instance, we might consider making type downloads lazy for something like import '[email protected]' if it aligns better with user preferences. Additionally, it's worth evaluating whether lazy download provides a superior user experience compared to a one-time download.
• Governance of provider packages can be delegated to the resource provider teams: This may present practical challenges. Honestly, I think this will never happen...

Another important question to answer here is, will users still be able to use resource types without importing an Azure resource provider and version at the beginning? If so, where do the resource types come from?

Nevertheless, I believe it's best to carry on this discussion within the appropriate issue thread.

[asilverman]

This proposal does attempt to address the problem of being able to quickly understand which Az resource types are available in a given Az package, but I have the following concerns:

• In the long run, I feel that consistency is extremely important. If we imagine a world where we have 100s of providers available, each with their own versioning scheme and rules, I think the experience would be very confusing for users.

In principle I agree with consistency being a good practice. However the current state also breaks consistency. For instance, an az provider with a semantic version of X.0.0 is exposing resources that are versioned using dates. The kubernetes provider exposes resources using the KubeAPI group versions (beta, v1alpha1, v1). So there are really two kinds of consistencies 1) consistency of versions for provider type bundles and 2) consistency between the bundle version and its underlying resources.

I feel like (2) is more important to ensure.

With that said, can you elaborate how do you perceive this confuses users?

In my opinion its unlikely for a single bicep module to have 100s of provider import statements.

Rather, in the context of a single bicep file I would expect authors to collect azure resource deployment in modules that are separate from other kind of resources. So when using best practices I would expect it to be a best practice for the files describing Azure resource types to not have kubernetes types intermingled amongst them.

So in the context of the bicep file that declares azure resources the versioning that uses dates would make sense. And in the contet of a bicep file declaring kubernetes resources semVer would make sense and maintain consistency of type (2)

Do you expect authors to mix&match these resource types in a single file in a production setting?

This is a heavily Azure-centric solution - there's no guarantee the same discoverability benefit would apply to other providers.

To be clear I am not advocating for the use of only dates as a version but rather relax the constraints so that each provider can use their own versioning scheme. In the case of azure the versioning scheme would be dates but in the case of kubernetes it would be semVer.

• There are issues with practicality (for example, what happens if a provider misses publishing an API version 2022-01-01, and our 2022-01-01 package misses this type, but it is then available in 2022-02-01? - or vice versa with API deprecation).

I am not sure how this could happen, isn't the version date for an API version set for the date the API is published? Maybe I am missing something here... I'm not an expert on the release process for new API versions.

• In practice, wouldn't people generally just want to be on "latest" for Azure? Is being able to be accurate with version targeting really as valuable as we think it is?

I think if people likely do want to be on "latest" for Azure, but as I understand it the API Versions are not in practice immutable causing problems so being able to be explicit is helpful, perhaps I am overestimating how many "surprises" of this kind truly exist.

Regarding the question:

Why do we feel we can't solve this problem with tooling or documentation?

I don't really see how we could solve this problem with tooling and documentation but I am open to ideas so happy to continue to think about this and evaluate any alternatives, I just can't think of one at this time.

[majastrz]

Here's my thought on how to use semantic versioning for the az provider:

• PATCH version can be calculated based on the git commits of the https://github.com/azure/bicep-types-az repo instead of https://github.com/Azure/azure-rest-api-specs. PATCH version should be updated if:

  • New API versions are added*
  • Existing API versions are updated*

• MINOR version should be bumped if:

  • New resource types are added*
  • New az template functions are added

• MAJOR version should be bumped if there are resource types

  • Resource types or API versions are deleted*
  • There are breaking changes to az template functions

*: Can be detected with an automated CI pipeline

I think semver creates an expectation that we'd flag any breaking change within resource bodies as well.

[shenglol]

I think it's still possible to identify breaking changes within resource bodies for the same API (e.g., by running a diff algorithm). However, I'm not sure if it's worth doing it, given that resource body breaking changes are extremely rare (actually, I don't know if such breaking changes would even be allowed to be accepted for merging today). In most instances, an azure-spec-rest-api PR updating an existing API version that is marked with "BreakingChangeReviewRequired" typically pertains to additions of optional properties, property annotations, and similar modifications that, in practice, do not constitute breaking changes for Bicep or ARM template users.

[anthony-c-martin]

@asilverman I think probably easiest to discuss offline - I can talk you through my objections in more detail.

To give a simpler counterpoint:

Assumptions:

1. Each provider package MUST be immutable.
2. Providers can and will make mistakes or omissions when publishing API versions, and often go back to make corrections when issues are reported. This is VERY common.

The benefit of this proposal is that a user can purely use the version of the package to determine which resource types are supported. However, if we accept assumptions 1 & 2, this is not true - there will be cases where they need to arbitrarily increase the version, or snap to "latest" to deal with inaccuracies or omissions. At this point I really can't see any benefit over semver - the version is just an opaque string.

I am not sure how this could happen, isn't the version date for an API version set for the date the API is published? Maybe I am missing something here... I'm not an expert on the release process for new API versions.

I'll fill you in on this - background on the release process is crucial for this proposal.

[jeskew]

I think it's still possible to identify breaking changes within resource bodies for the same API (e.g., by running a diff algorithm). However, I'm not sure if it's worth doing it, given that resource body breaking changes are extremely rare (actually, I don't know if such breaking changes would even be allowed to be accepted for merging today). In most instances, an azure-spec-rest-api PR updating an existing API version that is marked with "BreakingChangeReviewRequired" typically pertains to additions of optional properties, property annotations, and similar modifications that, in practice, do not constitute breaking changes for Bicep or ARM template users.

I agree that we should be able to automatically detect resource body breaking changes. Cribbing from @shenglol's suggestion, I would propose the following algorithm for detecting the type of change to make:

• PATCH version should be updated if:
  • Documentation strings are changed
• MINOR version should be bumped if:
  • New API versions are added*
  • New resource types are added*
  • New resource properties are added*
  • New az template functions are added
• MAJOR version should be bumped if:
  • Resource types or API versions are deleted*
  • az template functions are removed
  • There are breaking changes to az template functions

*: Can be detected with an automated CI pipeline

Semver requires that any new feature result in a minor version bump, and rolling new resource types or API versions in patch releases breaks this expectation. Following the above would mean that almost all updates would be a new minor version, with some not infrequent major version bumps when RP teams remove preview API versions. Patch versions would be theoretically possible but would never happen in practice.

That said, I would argue that semver is the wrong approach to take here because it introduces cognitive overhead without providing any of the affordances we see from semver with code dependencies. We have no plans to introduce features that would make semantic versioning useful, so tracking the semantic version of a provider increases complexity without increasing utility.

The main reason for this is that the current syntax requires that template authors pin a specific version of a provider. Semver matching notation (e.g., import 'az@^1.0.0') is not supported, meaning that reporting which releases contain breaking changes in the version number doesn't enable any additional automation. And given how often RPs retire preview API versions, I imagine we would train humans to ignore the major version number pretty quickly.

Secondly, there's no actual runtime effect of pinning a version and ignoring updates. Users could check their templates against an older version of the AZ provider, but they can't actually use it at runtime. Both RP contracts and az template functions will always use the latest version of the AZ provider (i.e., whatever the currently deployed state of ARM and ARM RPs might be). The primary consideration for a user is therefore, how close is my version of the provider to the current state of the world? Phrased different, how current is my version of the provider? You can't tell that from a semantic version, but you can tell it from a date.

I would argue that we should also support a special sentinel version value (e.g., import 'az@latest') so that users don't have to constantly tweak their templates in order to take advantage of separate vended provider types. This would be safe to do because users already have to pin their templates to specific API versions at the resource level, and that is how Azure RPs communicate which changes are incremental (performed within an API version) and which are breaking (performed by introducing a new API version). Using '@latest' is arguably safer than pinning a specific version of the provider because it reduces the drift between the Bicep compiler's view of the world and the state of the world that will be enforced when the template is deployed.

For non-Az providers, is the plan to have multiple versions of the server side code running? If so, semver would make sense, as the import communicated in the template would be respected at deploy time. This capability seems particularly important for providers that talk to different instances of something (e.g., k8s), where each instance may be running a different version of the thing ARM extensibility is talking to. It seems less important for providers where ARM will communicate with a centralized control plane (e.g., az or graph), where deployment requests will always be handled by the latest version of the API.