Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for multiple Kafka clusters within a spec #465

Closed
dalelane opened this issue Nov 25, 2020 · 21 comments · Fixed by #842
Closed

Support for multiple Kafka clusters within a spec #465

dalelane opened this issue Nov 25, 2020 · 21 comments · Fixed by #842
Labels
💭 Strawman (RFC 0) RFC Stage 0 (See CONTRIBUTING.md)

Comments

@dalelane
Copy link
Collaborator

dalelane commented Nov 25, 2020

Is your feature request related to a problem? Please describe.

A Kafka cluster is typically made of a group of Kafka brokers. The brokers act as peers, so a client is able to make an initial connection to any broker in the cluster, and a metadata exchange takes place to inform the client which broker it should connect to.

To enable the cluster to be highly-available to connections from clients, Kafka clients are typically configured with the address of every broker in the Kafka cluster - so that it can try each broker in turn in the event that the first broker it attempts to connect to is unavailable.

Currently, this is being modelled in AsyncAPI by identifying each broker as a separate server in the list of servers. For the purposes of code generation, an assumption is made that all the Kafka servers listed in the servers section of the spec belong to the same cluster, and so all server URLs can be combined to provide a single bootstrap servers list.

Describe the solution you'd like

I'd like a way to identify multiple separate Kafka clusters within a single AsyncAPI spec.

For example, imagine I have a development/test Kafka cluster composed of three brokers A, B, C
and a production Kafka cluster composed of three brokers D, E, F

It's not safe to put all six broker addresses as six separate server objects in the spec, as I don't have a way to identify the grouping, and current code-generation would treat it as a single Kafka cluster with six brokers in.

Additional context

@github-actions
Copy link

Welcome to AsyncAPI. Thanks a lot for reporting your first issue.

Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

@derberg
Copy link
Member

derberg commented Nov 26, 2020

Copying over discussion from Slack as we don't know when it will dissapear cause of the free plan


Dale Lane Yesterday at 10:51 AM
It's customary in Kafka to provide client applications with a comma-separated list of brokers in the cluster (so the client is able to attempt connection to any of the broker addresses in that list)
Are there any issues with putting a comma-separated list in the url field of a Server object for this purpose?
https://www.asyncapi.com/docs/specifications/2.0.0#fixed-fields-4
(or are there better ways I should represent this? like having each broker as a separate server object?) (edited) 




9 replies

Lukasz Gornicki  22 hours ago
Hey Dale, at the moment url is just for one url, and you should have multiple Server objects. This is how it is for example supported in the java template -> https://github.com/asyncapi/java-spring-template/blob/master/template/src/main/resources/application.yml#L61-L64
I do remember though that we had discussion about it once in the past (but we have free slack and it is lost) and I’m not sure…. @Semen do you remember if we ended up creating an issue to discuss further?
The problem is that having those bootstrap servers, every single one in separate server object is in conflict with our assumption how servers should be use, that they should represent environments (this is how generator supports it) (edited) 

Lukasz Gornicki  22 hours ago
The problem is that having those bootstrap servers, every single one in separate server object is in conflict with our assumption how servers should be use, that they should represent environments (this is how generator supports it)
This is something I’m discussion at the moment with @Semen here -> https://github.com/asyncapi/java-spring-template/pull/55 (edited) 

Semen  22 hours ago
Yes, because of this limitation in spec, spring-Java-template uses all server from API marked with protocol: kafka

Dale Lane  22 hours ago
that's really useful - thanks, both

Fran Méndez  22 hours ago
I wonder if it would be a good thing to add to the spec itself. Something like alternativeUrls . Or maybe a good use case for the Kafka Server Binding, which still doesn't exist. It could be called altenativeHosts, since Kafka usually requires hosts instead of urls.

Dale Lane  21 hours ago
That would be helpful for the case where I have two clusters (e.g. a production cluster and a development cluster) each made up of three brokers. Listing all six brokers in the servers section would get confused.
That may be an edge case though

Fran Méndez  21 hours ago
I don't think it is. It might be an edge case for development but people often have production clusters and staging/test clusters or even a separate cluster for partners.

Fran Méndez  21 hours ago
Feel free to leave your opinion on an issue at github.com/asyncapi/asyncapi. I think this is something we should consider having on the spec, either on the core spec or on the kafka bindings.

Dale Lane  21 hours ago
will do, thanks

@dalelane
Copy link
Collaborator Author

Strawman suggestion for an approach - we could provide a cluster identifier for each server

asyncapi: '2.0.0'
servers:  
  prod-broker0:
    url: dale-prod-broker-0:9092
    protocol: kafka
    bindings:
      kafka:
        cluster: production
  prod-broker1:
    url: dale-prod-broker-1:9092
    protocol: kafka
    bindings:
      kafka:
        cluster: production
  prod-broker2:
    url: dale-prod-broker-2:9092
    protocol: kafka
    bindings:
      kafka:
        cluster: production
  dev-broker0:
    url: dale-dev-broker-0:9092
    protocol: kafka
    bindings:
      kafka:
        cluster: development
  dev-broker1:
    url: dale-dev-broker-1:9092
    protocol: kafka
    bindings:
      kafka:
        cluster: development
  dev-broker2:
    url: dale-dev-broker-2:9092
    protocol: kafka
    bindings:
      kafka:
        cluster: development

@fnobilia
Copy link

fnobilia commented Dec 5, 2020

I'm wondering if we really need the multiple clusters feature. For instance, if you think about how you would deploy an application generated with this AsyncApiSpec in K8s, the multiple cluster feature would not add any benefit. Usually, you have one Async spec for dev and one for prod because they will evolve at a different speed. Am I missing anything here? 🤔

Having said that, I like the feature of passing multiple brokers for one cluster. It's very in line with almost every streaming platform 🙂

@dalelane
Copy link
Collaborator Author

dalelane commented Dec 5, 2020

I think I'm just used to being able to include both because that's how I use OpenAPI (e.g. https://spec.openapis.org/oas/v3.0.3#server-object-example )

@fnobilia
Copy link

fnobilia commented Dec 5, 2020

Usually, I use /doc, or I have a separate service serving just the spec. This setup plus my CI/CD forces me to have only one spec per environment.
How do you expose your OpenAPI spec? Can you help me understand your setup?

@buyukim
Copy link

buyukim commented Jan 13, 2021

+1 for this feature request. When developing a new version of the spec, we will typically create a new branch with a new spec version. We do not programmatically change the contract when promoting the spec from dev/test to prod, it is just approved and promoted.

Our Kafka clients need a cluster server list in prod that is separate and independent from the list for the dev/test environment

@fmvilas fmvilas added this to the AsyncAPI specification 2.1.0 milestone Jan 31, 2021
@smoya
Copy link
Member

smoya commented Mar 19, 2021

Could this be considered a duplicate of #244? Not sure. That's why I'm asking.

@dalelane
Copy link
Collaborator Author

Could this be considered a duplicate of #244? Not sure. That's why I'm asking.

Yes, I think so - I hadn't seen that issue before

@buyukim
Copy link

buyukim commented Mar 21, 2021

Isn't that issue more related to using multiple sets of brokers, each with different topics? Seems like a more complicated implementation since there is still only one url per server

@fmvilas fmvilas removed this from the Next specification version milestone May 12, 2021
@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity 😴
It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation.
Thank you for your contributions ❤️

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity 😴
It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation.
Thank you for your contributions ❤️

@github-actions github-actions bot added the stale label Sep 11, 2021
@derberg derberg removed the stale label Sep 13, 2021
@fmvilas fmvilas added this to the 3.0.0 Release milestone Sep 14, 2021
@smoya
Copy link
Member

smoya commented Oct 5, 2021

Yes, I think so - I hadn't seen that issue before

It is not. That one was pointing to another direction.

This one is still valid and I think this should become an actual strawman RFC0.

@smoya
Copy link
Member

smoya commented Oct 5, 2021

I think the question we might need to answer here is: Is it worth to be added in the core spec or rather as a Kafka server binding?

Does the "cluster" concept applies to other protocols (It doesn't have to apply to all but most)?

@jonaslagoni
Copy link
Member

jonaslagoni commented Dec 17, 2021

This is not really native to Kafka, NATS have the same "problem". Usually, in code you just pass an array or ; separated string of URLs to connect to. I wonder if it would make sense to allow multiple connection URL to be defined?

asyncapi: '2.0.0'
servers:  
  prod-broker:
    url: 
      - dale-prod-broker-0:9092
      - dale-prod-broker-1:9092
      - dale-prod-broker-2:9092
    protocol: kafka
  dev-broker:
    url: 
      - dale-dev-broker-0:9092
      - dale-dev-broker-1:9092
      - dale-dev-broker-2:9092
    protocol: kafka

To me, this is the most simple and uncomplicated approach to solve this 🤔

@jonaslagoni
Copy link
Member

@dalelane do you want to champion this? 🙂 Or can we consider this issue as needs champion? 🤔

@dalelane
Copy link
Collaborator Author

@jonaslagoni sure - I'd be happy to pick this up again

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added stale and removed stale labels Jul 28, 2022
@dalelane
Copy link
Collaborator Author

support for tags in servers being added in #465 gives us a mechanism for describing this

I'll leave the issue open for now so I can add an example that demonstrates this

@smoya
Copy link
Member

smoya commented Sep 22, 2022

support for tags in servers being added in #465 gives us a mechanism for describing this

I think the link is not correct. Maybe you wanted to refer to #809

@dalelane
Copy link
Collaborator Author

yes, that's right... thanks!

(sorry - that'll teach me to try and multi-task! 🤦‍♂️)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💭 Strawman (RFC 0) RFC Stage 0 (See CONTRIBUTING.md)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants