Inspecting bindings for spec 3.0

[jonaslagoni]

Reason/Context

We have no overview of how the spec 3.0 changes affect the bindings. Or if they even affect them at all. We also need to figure out if we need to make spec 3.0 specific bindings, how to visualize them.

Preliminary possible binding changes:

1. Operator changes, might affect HTTP bindings
2. Request reply, affect all bindings that has relevant properties for this.

This issue are meant as a placeholder until the changes in the spec are finalized where code owners will be pinged for feedback.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[jonaslagoni]

As the spec changes are nearly there, it's time to inspect the bindings. Tagging code owners.

@rcoppen do you have the bandwidth to inspect ibmmq?
@lbroudoux @dalelane do you folks have the bandwidth to inspect kafka?
@whitlockjc do you have the bandwidth to inspectgooglepubsub?
@damaru-inc @CameronRushton do you folks have the bandwidth to inspectsolace?
@VisualBean do you have the bandwidth to inspectpulsar?

[jonaslagoni]

Generally, for all bindings the following applies:

• all operation binding objects, need to change because we no longer have Applies To with Publish and Subscribe values.
• Change examples structure where AsyncAPI v2 contains

Further more, we do not add features here, it's only about adapting to the new 3.0 structure.

Here is the inspection of the other bindings:

amqp

• Operation binding object
  • mandatory, is message specific, but part of operation, not sure if this should change?
  • replyTo, with the request/reply change, this could probably be removed? BTW, send what response?

amqp1

• None

anypointmq

• Server Object
  • With the change to URL, the constraints change as well.
• Message Binding Object
  • Linking to schema object for headers, which should now be the custom header type.

http

• Operation Binding Object
  • type probably need to be removed, because of the new request/reply feature
  • query probably need to change because of new schema for queries?
• Message Binding Object
  • headers probably need to change because of new schema

jms

• None

mercure

• None

mqtt

• None

mqtt5

• Server Binding Object
  • sessionExpiryInterval might need to change because of usage of schema object.

nats

• None

redis

• None

sns

• None

sqs

• None

stomp

• None

websockets

• Channel Binding Object
  • query probably need to change because of new schema
  • headers probably need to change because of new schema

[jonaslagoni]

I am gonna create individual PRs for each protocol above for it to make sense on a review point of view.

[whitlockjc]

I will gladly take care of the googlepubsub bindings. I'm not sure what, if anything, needs to be done right now.

[jonaslagoni]

I will gladly take care of the googlepubsub bindings. I'm not sure what, if anything, needs to be done right now.

@whitlockjc just had a quick look:

• Examples needs to change
• Message Binding Object - schema might change based on feat: allow defining schema format other than default spec#910

But that's about it I think 🤔

[jonaslagoni]

I created 6 PRs based on #182 (comment)

• feat: adapt AMQP bindings to v3 #202
• feat: adapt anypointmq bindings to v3 #203
• feat: adapt HTTP bindings to v3 #204
• feat: adapt MQTT bindings to v3 #205
• feat: adapt MQTT5 bindings to v3 #206
• feat: adapt websockets bindings to v3 #207

[VisualBean]

Is there some existing documentation on an 'upgrade path' for V3? Not for bindings, but for the V3 spec.
having a decent overview of that would help 'migrate' bindings. Although they would all have to be backwards compatible i guess.

[jonaslagoni]

@VisualBean I am on it, finishing the release notes today that you can use as guidelines.

[jonaslagoni]

I have updated the release notes: Read it here: https://deploy-preview-659--asyncapi-website.netlify.app/blog/release-notes-3.0.0 (asyncapi/website#659)

The migration guide has not been finalized, but it does contain some information, might be useful as well: https://deploy-preview-660--asyncapi-website.netlify.app/docs/migration/migrating-to-v3 (asyncapi/website#660)

You can also find the most recent spec document here: https://www.asyncapi.com/docs/reference/specification/v3.0.0-next-major-spec.12

cc @VisualBean

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[derberg]

Kindly pinging

@rcoppen please inspect ibmmq binding
@lbroudoux @dalelane please inspect kafka bining
@damaru-inc @CameronRushton please inspect solace binding
@VisualBean please inspect pulsar binding

[VisualBean]

Kindly pinging

@rcoppen please inspect ibmmq binding @lbroudoux @dalelane please inspect kafka bining @damaru-inc @CameronRushton please inspect solace binding @VisualBean please inspect pulsar binding

I think the pulsar bindings are fine as is.
They basically only have channel bindings, and those seem to fit with the v3 narrative still

[lbroudoux]

Hey there,

I think that Kafka bindings are globally fine, but examples have to be adapted because we have bindings at the operation and the message levels.

That said, I have a general concern regarding the versioning of bindings and the matching with the spec versions. Today, we have bindings with 0.x.y versions that match the AsyncAPI spec in version 2.x. In the proposed PR by @jonaslagoni, the second x digit has been incremented but with sometimes no change on the binding schema, just the way it's included in an AsyncAPI document, and it applies to new 'send/receive' semantics.

This is a bit confusing to me because of:

• the upgrade of a digit without any structural changes,
• the dealignment it would bring regarding different bindings.Ex: mqtt-0.1.0 is for AsyncAPI 2.x but mqtt-0.2.0 is for AsyncAPI 3.x. kafka-0.4.0 is for AsyncAPI 2.x and kafka-0.5.0 will be for AsyncAPI 3.x. You'll have to know for each binding when the transition from 2.x to 3.x occurs

I would have thought of other options for versioning In the case of no binding schema update:

• I would have kept the same binding version and just enriched the example sections to illustrate how to include it in an AsyncAPI 2.x or an AsyncAPI 3.x document. Ex: 0.4.0 version of Kafka binding can be included in both AsyncAPI 2.x or 3.x
• OR I would have forced the artificial upgrade of a version digit to ease the transition. Ex: all bindings defined 0.x are for AsyncAPI 2.x, and all bindings defined 1.x are for AsyncAPI 3.x (well maybe 1.x is not the better choice but ... I guess you see what I mean).

What do you think? Am I the only one to be a bit confused on this?

[smoya]

What do you think? Am I the only one to be a bit confused on this?

No, you are not the only one for sure. I also think it might look very confusing, and forcing the users to carry on with that complexity is not desired IMHO.

• OR I would have forced the artificial upgrade of a version digit to ease the transition. Ex: all bindings defined 0.x are for AsyncAPI 2.x, and all bindings defined 1.x are for AsyncAPI 3.x (well maybe 1.x is not the better choice but ... I guess you see what I mean).

I think this option helps reducing that complexity. However, I don't think we can force or ensure all bindings versions will follow such versioning strategy. For example, what if a binding needs to introduce a breaking change? Will it mean its version can't increase in one major because no new major spec version is out?

I would keep the idea of releasing a major for each major spec version that is supported, but keeping a table in the markdown that shows a compatibility matrix, mapping binding version with supported AsyncAPI major versions.

[lbroudoux]

Thanks @dalelane and @smoya for the comments!

I also think that transitioning from AsyncAPI 2.x to 3.x will take some time and I am afraid that linear versioning will introduce confusion...

Imagine I'm still using AsyncAPI 2.x and I'd like to propose a change to a binding - starting with the following situation:

Version

Current version for AsyncAPI v3.x documents is 0.2.0.
Current version for AsyncAPI v2.x documents is 0.1.0.

In case of a non-breaking change:

• Should we produce 0.3.0 and 0.4.0(report on 3.x) using the second digit like we did before?
• Should we produce 0.2.1 and 0.3.1 (report on 3.x) now using the patch digit?
• Should we forbid the change, "forcing" people to upgrade to AsyncAPI 3/x before the binding update to 0.3.0?

In case of a breaking change:

• Should we produce 1.0.0 and 2.0.0(report on 3.x) using only the first digit as semver requires it?
• Should we produce 1.1.0 and 1.2.0(report on 3.x) using the first digit as semver requires it but keeping the second one to track the history line?
• Should we forbid the change, "forcing" people to upgrade to AsyncAPI 3/x before the binding update to 1.0.0?

I'm scratching my head over this... and I am sorry if it has already been discussed elsewhere 😖

Shall we adopt another versioning scheme clearly stating the major spec number? Ex: v2-0.1.0, v3-0.2.0. Even if it's not conventional, it makes things clear and avoid relying on compatibility matrix that are always hard to keep up to date.

[jonaslagoni]

Love this discussion, keep going you are definitely on the right track 👏

Today, we have bindings with 0.x.y versions that match the AsyncAPI spec in version 2.x. In the proposed PR by @jonaslagoni, the second x digit has been incremented but with sometimes no change on the binding schema, just the way it's included in an AsyncAPI document, and it applies to new 'send/receive' semantics.

Just for the record, I have no preference for a solution, I just used the simplest approach for bumping the binding versions, because, for AsyncAPI v2, I would still like to be able to use the old bindings with the proper definitions and descriptions (i.e. not AsyncAPI v3). That's why it was bumped even though only the description/constraint changed 🙂 But yea, could also just as well have added v3-specific information along side v2, just didn't come up 😄

Somewhat related to asyncapi/spec#590

[dalelane]

I think that Kafka bindings are globally fine, but examples have to be adapted because we have bindings at the operation and the message levels.

+1 on this

[jonaslagoni]

@dalelane @lbroudoux any of you have the bandwidth to update the examples, or possibly create a good first issue for it?

[lbroudoux]

I should be able to do it at the end of the week.

[lbroudoux]

As I'll probably work on the PR tomorrow, do you think about the versioning convention above?

@dalelane: What do you think of having this v3-0.4.0 version for the Kafka binding?
@jonaslagoni: Is it still time to introduce this, or do you think consistency with other bindings is much more important and we should stick with bumping? (and maybe never get the opportunity to align on all bindings...)

[jonaslagoni]

@jonaslagoni: Is it still time to introduce this, or do you think consistency with other bindings is much more important and we should stick with bumping? (and maybe never get the opportunity to align on all bindings...)

Technically it's all yours and @dalelane decision @lbroudoux 👍 But from my side, I do not see anything wrong with trying out different version strategies across different bindings, as long as it's well-documented (compatibility matrix would probably be a good idea) 👍

The strategy that works the best will win over time 😄 So I think at some point it will most likely be aligned 👍

[dalelane]

@lbroudoux I notice something went wrong with the last merge

https://github.com/asyncapi/bindings/blob/master/kafka/README.md?plain=1#L148

https://github.com/asyncapi/bindings/blob/master/kafka/README.md?plain=1#L155

Do you want to do this as part of the PR for this? Or shall we fix it separately?

[lbroudoux]

Will integrate it in the PR. I will then produce 2 versions of the binding integrating this fix:

• v2-0.4.0 for AsyncAPI v2
• v3-0.4.0 for AsyncAPI v3

I propose to split the README file into 2 different files (keeping the main one to introduce both versions and include a compatibility matrix) and add subfolders to /json_schemas so that we can also have different schemas referencing the different #/definitions/specificationExtension for both versions.

I think I will not have the bandwidth to finalize today but probably during the weekend.

[jonaslagoni]

add subfolders to /json_schemas so that we can also have different schemas referencing the different #/definitions/specificationExtension for both versions.

@lbroudoux Remember to make the JSON Schema changes in the spec-json-schema repo 😄

They will be removed soon from this repository.

[lbroudoux]

Oops! Forgot about that. Thanks @jonaslagoni for the reminder!

[lbroudoux]

Following our previous discussions, I have created:

• feat: adapt Kafka bindings to v3 #221
• feat: adapt Kafka bindings to v3 spec-json-schemas#446

Feel free to review and comment 😉

[derberg]

Hey, do not wanna be a party pooper but just had a look at

• feat: adapt Kafka bindings to v3 spec-json-schemas#446
• feat: adapt Kafka bindings to v3 #221

and I'm very confused, especially "v3-0.4.0", "0.4.0" custom version format puts me in super confusion mode 😄

bindings should have defaults aligned with spec. And in spec strategy is simple, no features in old majors, this is why in spec repo in master you only have latest spec version. So when we release v3 and somebody using v2 wants a new feature, they need to migrate to v3 and add feature to v3. The same must be with bindings.

Imagine I'm still using AsyncAPI 2.x and I'd like to propose a change to a binding

You won't be able to do it as nobody want to deal with old versions and remember both ways. I for example plan to forget v2 world as soon as possible 😄
So if you are using v2 you need to use v2-related Kafka binding. If you need something new in Kafka binding, add it to v3-related Kafka binding and migrate to v3 🤷🏼

In the proposed PR by @jonaslagoni, the second x digit has been incremented but with sometimes no change on the binding schema

I'm not sure what PR and changes you mean exactly, but binding schema is not part of binding specification. Versioning is related to basically a README.md that is the specification, for example of kafka binding. binding schema (that I understand you mean JSON Schema that describes binding spec) is a tool, not part of spec - README.md is sournce of truth.

So if in README nothing changes, maybe just examples, then version should not change

examples:

• in feat: adapt anypointmq bindings to v3 #203 we needed version bump as structure of binding changed in README
• in feat: adapt MQTT bindings to v3 #205 we did not have to bump version as I see no changes in the binding that change it's behaviour. But not a big problem as later in binding JSON Schema we can modify enum "enum": ["0.2.0"] to "enum": ["0.1.0", "0.2.0"] if older binding also works with v3

does it somehow help?

tl;dr spec vs bindings, whatever is in spec master is what bindings master needs to support
for previous versions, we probably need to introduce a proper tag in bindings just like we have in spec

[derberg]

of course happy to hop on a call if anybody needs

[lbroudoux]

Hey Lukasz,

Thanks for the comments.

So when we release v3 and somebody using v2 wants a new feature, they need to migrate to v3 and add a feature to v3. The same must be with bindings.

I asked this question in the past, and the answer wasn't that assertive.

So if you are using v2 you need to use v2-related Kafka binding. If you need something new in Kafka binding, add it to v3-related Kafka binding and migrate to v3 🤷🏼

I think that organizations that already invest in AsyncAPI and manage several specs may take some time to migrate to v3 and adapt their toolchain as well. I think it's usually great to have flexible migration paths and be able to dissociate migrating the abstract level (the spec) and the protocol one (the bindings).

The other concern I had with the single-digit bump (or no bump if not required) was about the compatibility matrix. As versioning among the bindings is not homogeneous (some are 0.4.0, others still 0.1.0), it's hard to know which versions can be applied to v2 and which ones must be applied to v3. Thus this notation proposal allows not to rely on a - probably out-of-sync - compatibility matrix documentation.

The above concerns found some echo, but if we're happy going the way of breaking changes, constrained upgrades and compatibility matrix, that's also fine for me. We'll just have to revert some stuff. The most important thing here is to be clear that - as you said: "When we release v3 and somebody using v2 wants a new feature, they need to migrate to v3 and add a feature to v3"

Cheers,

[derberg]

It is a complicated topic, as on one hand bindings are there, maintained by different people to assure flexibility, but on the other hand when I look at it as spec repo maintainer, it is clear for me we need to be consistent. Especially versioning, the strategy must be the same because we know people use different protocols, never stick to one, so from DevX and tooling support perspective I cannot imagine that user ends up in a world where every binding has its own release strategy and release conventions.

the good thing is that in JSON Schemas that describe the spec, for 2.6 and prior, we have zero constraints -> https://github.com/asyncapi/spec-json-schemas/blob/next-major-spec/definitions/2.6.0/bindingsObject.json. So people can put whatever they want (and they actually do it already, and for example in docs generation we take whatever is there and do not fail AsyncAPI document validation). This means that if they really want to add something to v2 instead of migrating to v3 - they can add it, but not through official bindings-contribution process. And it is up to the tooling, if something is supported or not.

if we wanna change the approach, we need a dedicated discussion where we make sure all owners are notified and follow.

The above concerns found some echo, but if we're happy going the way of breaking changes, constrained upgrades and compatibility matrix, that's also fine for me.

I completely agree but this can be consistently solved with GitHub tags. So whenever we release main specification, we tag bindings repo as well, and point links from spec to a specific tag in github. So whatever is there in master is draft with editorial changes, and for older check tags. This is what we have in spec repo:

The latest draft specification can be found at [spec/asyncapi.md](https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md) which tracks the latest commit to the master branch in this repository. The human-readable markdown file is the source of truth for the specification.

[Version 2.6.0](https://github.com/asyncapi/spec/blob/v2.6.0/spec/asyncapi.md) (latest)
[Version 2.5.0](https://github.com/asyncapi/spec/blob/v2.5.0/spec/asyncapi.md)
[Version 2.4.0](https://github.com/asyncapi/spec/blob/v2.4.0/spec/asyncapi.md)
[Version 2.3.0](https://github.com/asyncapi/spec/blob/v2.3.0/spec/asyncapi.md)
[Version 2.2.0](https://github.com/asyncapi/spec/blob/v2.2.0/spec/asyncapi.md)
[Version 2.1.0](https://github.com/asyncapi/spec/blob/v2.1.0/spec/asyncapi.md)
[Version 2.0.0](https://github.com/asyncapi/spec/blob/2.0.0/versions/2.0.0/asyncapi.md)
[Version 1.2.0](https://github.com/asyncapi/spec/blob/1.2.0/README.md) (deprecated)
[Version 1.1.0](https://github.com/asyncapi/spec/blob/1.1.0/README.md) (deprecated)
[Version 1.0.0](https://github.com/asyncapi/spec/blob/1.0.0/README.md) (deprecated)

The new constraint would be that if you add something new to Kafka binding, it needs to wait for main spec release to be supported. This is not a problem though as we anyway release spec regularly 4 times a year.

Scenario:

kafka binding is on version 0.5.0 now
spec is on 3.0.0

###
we release spec 3.1.0
no changes in kafka binding in this release

in both, spec and bindings repos, 3.1.0 tag is created
if kafka was on version 0.5.0, it is tagged with 3.1.0 and considered latest to use with 3.1.0 just like it was for 3.0.0

###
we release spec 3.2
kafka at the same time was also extended and is on version 0.6.0

in both, spec and bindings repos, 3.2.0 tag is created
kafka binding 0.6.0 is considered to be latest version best to use with 3.2.0. From version number you see version is not a breaking change, you can just switch your binding from 0.5 to 0.6 without any migration

###
we release spec 3.3
kafka at the same time was also extended but with breaking changes 1.0.0

in both, spec and bindings repos, 3.3.0 tag is created
kafka binding 1.0.0 is considered to be latest version working with 3.3.0. From version number you see it is major so breaking, and since work was aligned with 3.3.0, better to really use 1.0.0 binding only with 3.3.0 (especially that in case of spec minor releases migration is as simple as version number change in asyncapi document

it is just a rough idea 🤔 basically wanna visualize what options we have other than doing custom-per-binding approach. Thoughts?

---

so my suggestion:

• adjust whatever needs an adjustment in kafka bindings just like others did
• let us tag bindings repo with v2 right before the release (or now, if we do not see any problems with that - freezing master basically) and on release date of spec 3.0.0 let us tag bindings repo with v3.0.0 and immediately kick off a discussion on how the versioning should work. As tags will be aligned, maybe bindings should go inside spec
• in v3 release notes we make a clear statement what is supported and what not. That whatever is in v3.0.0 tag is the latest and greatest that should be used with spec

@smoya @char0n @fmvilas @dalelane your view so we can somehow move forward?

[smoya]

I'm interested in how common this use case or need @lbroudoux mentioned:

I think it's usually great to have flexible migration paths and be able to dissociate migrating the abstract level (the spec) and the protocol one (the bindings).

It is just a feeling or it is really a need users are asking for? I don't know about numbers, metrics or provided company use cases that speak about this, but that doesn't mean this is not common.
I'm really interested in this matter because if that's a real need, we then need definitely to switch to a per-binding release strategy, including hosting most likely their JSON Schemas near their "spec", and treat them like if they would be separated repositories (but in this case would be a monorepo), plus some other constraints we already mentioned here in this issue.

The answer to this question is, IMHO, the critical decision maker here.

Any thoughts?

[lbroudoux]

Hey friends!

Thanks for your detailed thoughts on this.

I think the most important part of it that I agree with is:

if we wanna change the approach, we need a dedicated discussion where we make sure all owners are notified and follow.

So as a conclusion, I propose to edit a new PR for Kafka bindings just adapting the 0.4.0 README so that it embeds new examples on how to use it with a v3 spec. Please be sure to tag the current 0.4.0 README (with examples on v2). to v2 before the final merge and I think we'll have everything necessary to still allow both usages.

Also, before issuing this new PR, I'll merge #220 so that it will be quickly available.

Finally, I propose to be collectively vigilant on this point I mentioned above:

to have flexible migration paths and be able to dissociate migrating the abstract level (the spec) and the protocol one (the bindings).

I agree that this is currently a feeling on a requirement and not a user's feedback. But I have experience with large organizations where those 2 concerns (channel & message definitions vs bindings choices) would be handled by different personas, at different times within an API lifecycle. Don't know if AsyncAPI has adopters yet in that kind of organizations but better watch out.

[jonaslagoni]

Hey folks, as this issue was solely focused on getting the bindings up to date with v3, I am gonna close down the issue. Thanks for all the help with updating and inspecting bindings @lbroudoux @whitlockjc @VisualBean @smoya @derberg @mboss37 🙏

I suspect it's not the final discussion we are gonna have on versioning bindings either, if we need further discussion please create an issue 💪

[derberg]

yes, I encourage anyone to open up a followup issue with details on use case whenever if your communities you hear people complaining about how bindings work with asyncapi spec, and stuff like that, migrations and other topics