Turn Server #12
Turn Server #12
Conversation
There was a problem hiding this comment.
Good shit!
The big problem right now is we don't actually have any way of routing UDP traffic into the cluster. For HTTP traffic, take a look at https://www.ocf.berkeley.edu/docs/staff/backend/kubernetes/#h2_getting-traffic-into-the-cluster. (This is slightly out of date, we use HAProxy instead of NGINX now.) HAProxy does not support UDP, so we need to come up with some sort of different reverse proxy for UDP.
We could directly try to create a NodePort that exposes the UDP service directly on the masters at the lb IP (169.229.226.79). This might be as simple as creating a NodePort with externalIPs [169.229.226.79], but idk if that would work since that is a virtual IP. (For reference, here is the NodePort used by the normal HTTP flow: https://github.com/ocf/puppet/blob/master/modules/ocf_kubernetes/templates/ingress/ingress_expose.yaml.erb#L13-L36.)
This might be a good topic for discussion at the Kubernetes meeting on Wednesday! Thanks for spurring the conversation!
| targetPort: 3487 | ||
| selector: | ||
| app: coturn | ||
| type: LoadBalancer |
There was a problem hiding this comment.
I don't think LoadBalancer will do anything in our cluster.
| apiVersion: v1 | ||
| data: | ||
| # see https://github.com/matrix-org/synapse/blob/master/docs/turn-howto.md#configuration | ||
| turnserver.conf: | |
There was a problem hiding this comment.
imo we should build this into the container, like I mentioned in #11
There was a problem hiding this comment.
Hmm... it's not fully static because it needs the INTERNAL_IP variable. I'll need to look into why it needs that in the first place.
kubernetes/coturn.yml.erb
Outdated
| data: | ||
| # `pwgen -s 64 1 | base64 -w0` to generate | ||
| # identical to secret in homeserver.yaml | ||
| auth-secret: U2pKbFNwRjU4d016TW9FM0piVzFxZUFXVTk1Nzhta0JlbzgxSGxwMk9jNFpsbnRZVVI0MXR4VkJpcVJ2a1I1RQo= |
There was a problem hiding this comment.
obviously we need to move these out of here before merging. homeserver.yaml will probably be moved to use ocf/utils#146, here we can just use a standard templated secret (like https://github.com/ocf/grafana/blob/c5e6a1c9c429dc1c0e962cb90842b01b94b5a887/kubernetes/grafana.yml.erb#L107)
| name: coturn | ||
| namespace: matrix | ||
| spec: | ||
| externalTrafficPolicy: Cluster |
There was a problem hiding this comment.
not sure this does anything outside of AWS
| valueFrom: | ||
| fieldRef: | ||
| apiVersion: v1 | ||
| fieldPath: status.hostIP |
There was a problem hiding this comment.
seems like we would want the kubernetes pod IP, not the host IP (aka the IP of the worker). that would be status.podIP. that said, I don't really know what this is used for so that might be wrong
This is already being worked on this repo in #7, look how it's done there. Alternatively, look at ocfweb for a more stable repo that uses multiple Dockerfiles/images.
We need docs for this... but basically this is changing pretty frequently. Currently, there are two paths: hostPath mounts (example), or templated secrets injected during CI (example). Also, ocf/utils#146 is on the horizon which will allow for easier usage of templated injected secrets. All the secrets themselves are stored in the puppet private share on lightning.
idk if they need to be in the same namespace.
imo build it into the dockerfile. We want to put config in git as much as possible, we should definitely avoid mounting it from some external FS or something when possible. So, from best to worst: |
|
If it's too hard to figure out how to get UDP into the cluster, you could opt to not use Kubernetes for this and instead make a VM. |
closes #10
Still WIP, I need to figure out the following...
matrix? I don't even know what that does, but it sounds like a good idea to put matrix and coturn in the same namespace.BTW: Most of it was taken from here https://github.com/ananace/matrix-synapse/tree/master/kubernetes - which has updated almost immediately after each new matrix/riot update for a while now. Seems pretty reliable to me.