-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cluster bootstrapping with Istio #209
Comments
We have not looked into Istio so far in this context; if you could look into it and share your findings that would be very useful |
The current conclusion is that Istio doesn't allow pod to pod communication and there is no way to disable this restriction. Meaning that we can't form an Akka cluster using this method on an Istio cluster. Any other ideas to form a cluster? |
It's works by using pod dns names see.
|
Interesting, thanks for sharing! |
@thomschke Thanks for sharing, I'll try to give it a go in the upcoming days and let you know how it goes |
I forgot to expect that my approach has two side effects:
|
I am waiting for #217 to be merged and released to evaluate your solution |
@thomschke your solution works perfectly. One question though, is there any way to set readiness probe? It is very important because the services take time to boot and I don't like them to get traffic. |
If you are using
Or you write your own checks (see) and have to register it (see) |
Yes I am using |
An important point to be mentioned: Under Istio the ingress traffic is passed through the sidecar-proxy. For this reason all application sockets have to bind to
If you using
|
Already configured everything regarding the binding and hostname, I use Helm charts for my deployments based on |
Sorry, I have to correct myself: There is an issue #236 and it's fixed :-) |
Hello, |
The mutual auth is I think is being caused because the bootstrap is attempting to call itself which doesn't match to the local egress cluster that enables tls but instead matches the ingress cluster which is expecting tls and drops the connection. This seems to be a flaw in the definition of the clusters and rules in istio, but one thats not easy to resolve. This only happens when one of the TLS methods are enabled. |
I don't think bootstrap should need to probe its self. Raised #376 |
* Update to latest version of akka-management * Include workarounds to make cluster bootstrap work with istio as documented in akka/akka-management#209 (comment)
So there is a work around to this, if you disable TLS only on the read only management port then the bootstrapping will work correctly. This can be done with a permissive policy
|
There is another significant pitfall to this configuration - nodes rarely if ever leave the cluster gracefully, eg when scaling down or doing a rolling upgrade. The reason for this is that when the pod is stopped, both the Istio sidecar and the Akka cluster node are stopped simultaneously. The Istio sidecar stops almost immediately, which means the remoting traffic immediately stops, and that means that the Akka node cannot attempt to leave the cluster. That failed attempt will actually delay stopping the pod until it times out, meanwhile from the point of view of all the other nodes in the cluster, it will become unreachable, and will stay in that status until a downing strategy kicks in to down it. This will make rolling upgrades in particular quite clunky. |
Hmm, that is a pretty serious limitation. Is there any way to postpone the sidecar termination until after the node has successfully left? |
We have to wait for k8s container dependencies (kubernetes/enhancements#753). In the meantime you can use a prestop hook - but this is a hack and prevent us to use automatic sidecar injection. BTW: This is a good example of having a good downing strategy - especially during rolling updates because there is only a small window of time to react :-) |
That container dependency feature looks like it will solve the issue perfectly for us. By the way, thanks @thomschke for the instructions posted here, I don't know how long it would have taken me to get a cluster bootstrapping in Istio without your instructions here. I'm hoping we (@lightbend) will be able to publish some documentation on doing this soon, and we're also talking to IBM, RedHat and Google to see whether we can work with them to get better first class support for the Akka use case (and any other technologies that need pod to pod communication) in Istio. |
I have done some work on this recently.
So with the above configured, and assuming we can fix the kubernetes-api discovery so that it just looks up IP addresses rather than requiring the port to be configured, we will be able to run an Akka cluster easily in Istio, with only those two annotations added, nothing else would be different from a normal Akka cluster development, and the traffic wouldn't go through envoy, which means the order of container shutdown wouldn't matter so nodes can exit the cluster gracefully, and also ready checks can be used. On the minus side, that means the remoting can't take advantage of transparent mTLS, but that's probably fine, it should use another mechanism for configuring mTLS, cluster traffic is not service mesh traffic and routing it through envoy is likely to adversely impact performance. |
I've made the following pull requests:
With these two patches applied, I have been able to establish an Akka cluster in Istio, using the standard kubernetes API cluster bootstrap mechanism just by adding the following annotations to my pod template: annotations:
traffic.sidecar.istio.io/excludeInboundPorts: "2552,8558"
traffic.sidecar.istio.io/excludeOutboundPorts: "2552,8558" And not declaring the management port. Readiness checks are implemented and work. An example working project (though needs to be upgraded to new Akka management once the PR is merged) can be found here: https://github.com/jroper/akka-grpc-ddata-shopping-cart/tree/istio-future |
Hi all, Akka management 1.0.1 has been released with the changes, and the Istio PR has been merged into master. Istio 1.2 is scheduled to be released on June 19 I believe, so if anyone on this thread would be able to test it before then, that would be really helpful. There are instructions on here for installing the latest Istio nightly build: https://github.com/istio/installer Alternatively, for a very basic setup, simply run the following on a fresh Kubernetes cluster:
This will install my own build of Istio master, published to docker hub. |
Hi @jroper, I tested my lagom services under |
Thanks @thomschke! To complete the circle, I've added documentation here. |
The documentation is now published at https://doc.akka.io/docs/akka-management/current/bootstrap/istio.html |
Hi @jroper, I tested eclipse ditto 0.9.0 with istio 1.2.2 and akka management 1.0.1 and your described K8s annotations. Works without any problems 👍 |
Hi @jroper Looks like its no longer working for akka management 1.0.8 and Istio 1.6 I have the following config in my pod
And I have added the annotations on my pod
I am getting probe failures
If I try to access the probe directly, its returning 200 OK
I have created a bug on Istio |
Istio blocks direct connection between pods, is there a way to overcome this to allow the cluster bootstrapping?
This is the error I get right now:
The text was updated successfully, but these errors were encountered: