-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use a specific value for nodePort when the type of Service is NodePort #8840
Conversation
@iocanel I am not sure that the Decorator implementation I added is the most appropriate... @Ladicek @maxandersen I would like to hear your thougths as well. |
@maxandersen brought up the point that this will fail if more than one replicas are deployed. Max also raised the point that these kind of settings should be applied when the user is in "development mode", which unfortunately for the time being doesn't tied in to the Quarkus Dev Mode, since the Kubernetes extension requires the prod jar to be built. My idea with this PR is to make the deployment / update experience for a user using Mininkube as friction-less as possible. Thoughts? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am a bit skeptical about using a fixed node port.
What will happen if for any reason the port is already taken?
What will happen if I am using the kubernetes
extension in multiple applications or modules?
So, I would like to do some small research for alternatives.
If no other alternative can be found, then I would add the following behavior:
- Feature flag to turn this on and off (default value is debatable).
- I would hash the service name and namespace and get the default node port (to prevent conflicts if no node port has been explicitly specified).
- Deployer should display the node port to the user.
|
||
public AddNodePortDecorator(String name, int nodePort) { | ||
super(name); | ||
if (nodePort < MIN_VALUE || nodePort > MAX_VALUE) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This https://kubernetes.io/docs/concepts/services-networking/service/#nodeport says that the default range is 30000-32767, and apparently it's configurable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, nice find! I need to remove these checks then.
/** | ||
* The nodePort to set when serviceType is set to nodePort | ||
*/ | ||
@ConfigItem(defaultValue = "31987") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a way to not specify the concrete nodePort and let Kubernetes assign a port automatically? Perhaps allow the 0 value, which typically means "port should be auto-assigned"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When you don't set it, it gets assigned automatically. The problem is that it gets re-assigned to another value when you update the application and that's what I am trying to avoid.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. What I'm trying to say is: is there a way how to get back to the default behavior?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that essentially boils down to what @iocanel is asking for :)
Why would you need a feature flag? Isn't the use of
Yes, this is pretty much what I suggested above.
Good idea |
I'd say there's too much assumptions in this, so there should be a way out. Other than that, I don't have an opinion, because I have no idea what NodePort services are good for. |
The way I see it, is that this thing is opt-in. You need to set the service-type to
AFAIK, they are only good for "local development" - meaning they are the easiest way to expose a service to the outside world. |
Not really, the user should be able to select 1 replica, nodePort without using a fixed port.
I am guilty of posting without reading :-D
|
If we get the deployer to display the node port to the user, we could make it somewhat smarter and not reapply the service if it hasn't actually changed (in the hope of the using the same port), so that the user doesn't have to change ports all the time. |
So if I add feature flag and nice logging messages, do we agree on this approach? |
This seems more complicated to get right to me. |
The docs on nodeport gives the reasons what nodeport is useful for:
We are kinda doing the latter here. I still think the default behavior is best kept at "let kubernetes deal with it" but make it easy to activate this "dev-deployment" mode. |
It looks like having more than one replica, does not cause a problem with |
That seems to be the consencus here - if I have read all the comments correctly that is. |
are you sure ? I read the docs stating this: "you can specify a value in the nodePort field. The control plane will either allocate you that port or report that the API transaction failed. This means that you need to take care of possible port collisions yourself. You also have to use a valid port number, one that’s inside the range configured for NodePort use." |
Yes, I just tried it. There was no failure, all pods came up and the service was exposed, but only one of the pods was receiving traffic. |
Which makes sense cause you are not load balancing. |
Here is my latest idea: We don't use a flag per se, but instead have an enum configuration value named
The default should be What do you think? |
IMHO, it doesn't feel quite right. For starters, we shouldn't I also don't like the
Now, since we can't query the environment for minikube / minishift cluster status etc, I am wondering if we could introduce |
I don't see why we shouldn't do this. It's purely a dev thing, so why not make developers lives easier?
Letting Kubernetes pick the port however disrupts the developers workflow by changing the URL that is used to access the application. That's what I am really trying to avoid here.
Sure, but why do this if we can infer the environment? |
Having a really awesome and easy "kick the tires" experience for new users to quarkus and kubernetes running on their local developer laptop (especially for windows, mac users) is something that can really help quarkus and kick us ahead. Just think what Docker Desktop does for user experience and how to install and use it. Minikube is reasonable easy to install - and having a great blog, and video how to do the code.quarkus.io -> zip -> docker mutlistage build -> kubectl apply -f -> runs on kubernetes with 4mb of memory usage. And you would need the NodePort to call your deployed kubernetes. And btw would also be good if quarkus could log on startup whether it runs in JVM or Native mode. |
If we're after good developer experience, what's the point of messing with NodePorts when you can |
Because |
That I find hard to agree with. EDIT: I mean, we actually should point people towards using an Ingress, as that's how traffic income in Kubernetes should be done. To the best of my knowledge. |
In any case, if we make nodePort configurable (I don't disagree with that!), and if we make the default value some specific number (I don't disagree with that either, if it's documented), then there needs to be a way how to get back to the default behavior (automatic assignment), because you just can't assume that NodePorts will only be used for the scenario you're thinking of right now. |
IMHO resource generation should be |
That understanding is correct.
So currently if the user adds the I personally think this behavior is reasonable, but it definitely needs to be properly documented. |
That was my initial thought as well. We could certainly add that as an enhancement later on. |
IIUC, this is perfectly aligned with what we currently have. So, this pull request just introduces a new deployment target, that the user can optionally enable. I like it! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM,
I would like though a minor additon: A testcase that shows that minikube resource is only added when explicitly specified.
...etes/vanilla/deployment/src/main/java/io/quarkus/kubernetes/deployment/KubernetesConfig.java
Show resolved
Hide resolved
In
which does test that when there is no Did you have something more in mind @iocanel ? |
this is kinda what the original PR was doing which would make things slow and you would always have a remote lookup - which could fail if the cluster not available etc. Thus I like this issue much better where if you don't say anything it will pick the extension that are avialable to generate - but if any ambiguity you have to state which to use - either on command line or in application.properties to use as default. |
I think the myself, Ioannis and Max agree on the current approach, so unless @Ladicek objects very strongly, I believe the way forward is to merge this. |
You took it out of context. I only proposed discovering what kind of cluster is there when stuff is being deployed to it. Not when YAML files are generated. (If that would be possible, we could get rid of Sorry, I currently don't have time to respond properly. |
The current approach adds more inconsistency (new |
The idea is to create a new I don't think it's adding any inconsistency, it's just adding more options. But like I said, we need to move forward one way or another, so if you don't like it (which is perfectly understandable and acceptable), then we really need a proposal for an alternative. |
More options that are not orthogonal = inconsistency. (My proposal? Determine which YAML files to generate solely based on which At this point, I cave in. I officially retract everything I wrote in this issue, please treat it as if I never commented. Sorry, I don't have the required time these days. |
@Ladicek thanks for the input. The problem I see with your proposal is that for the example the At this point I'll leave it up to the man in change to make the call, @maxandersen :) |
I know you said you want to ignore but I think you made some interesting points @Ladicek I'm going to ask in hope to better understand your concerns. "not orthogonal = inconsistency" - is that the issue that there are (currently) no minishift.* options to tweak ? if yes, then this approach does not exclude this from happening/getting added if we get some usecases where relying on the kubernetes config is not enough for minikube variation. About ignoring deployment target for generating files - maybe I'm being naive here but what would that give you if only the specific files will be used when deploying ? I'm missing some usecase where you think its valuable to have this behavior. |
I think what I'm trying to say is that this is getting progressively harder to understand. |
I rebased the PR onto master because I wanted to make sure that nothing from #8894 would interfere with this one. |
Essentially now the first user supplied value is used to deploy. When no user supplied value exists, then priorities are used to determine which target should be deployed
I'm a bit puzzled by your comment here - minikube, kubernetes and openshift looks much a like since they are alike.
for I feel the configuration and behaviors we allow to set here are as close to reality as we can get.
I agree we should generalize where we can - I'm struggling to see where we can generalize more in this case. if you have some ideas i'm all ears on that! on this issue I say we should LGTM as it fits into the current approach of deployment targets thus adding it does not "hurt" and if we find ways to generalize more it should be doable with 4 deployment target options as it would be with the current 3. |
I still need to follow up with some documentation |
Follows up on quarkusio#8840 Co-authored-by: Guillaume Smet <guillaume.smet@gmail.com>
Follows up on quarkusio#8840 Co-authored-by: Guillaume Smet <guillaume.smet@gmail.com>
The idea behind this is that subsequent updates / pushes to a cluster
using a NodePort Service (most likely to be used with Minikube)
will result in the same external URL being used.
This way users don't have to find the new URL each time they update / push.