[TESTING][GCP][GKE] Chart on GCP #1
Comments
|
|
|
Hey @taktakpeops I was thinking we should maybe continue this discussion on your jitsi issue because the jitsi team is super helpful and they could help if we run into problems. I would love for them to see the architecture you have created here, looks impressive.. |
|
OK |
|
One thing to add is that this architecture works only if you have multiple nodes in your cluster and each JVB instance has its own node because JVB will bind the public IP of the node. So some affinity rules are needed for the deployment of multiple JVB |
|
Sorry I have been AWOL for a bit. Also I am creating the jibri yaml file with container post-start and pre-stop scripts to correctly change the /home/jibri/.asoundrc file in each of the containers. I will probably use hostpath to read and write the same file within all the containers on a node so they all know which sound cards are available and which aren't. As I mentioned earlier the Jibri in the architecture can possibly be a group like JVB is correct? |
|
Yes, indeed it can. Even if I think that it’s strongly tied to Jicofo + Prosody and I would see all of it in one pod. About the multiple JVB, it used some pod anti-affinity rules to ensure the dispatch of the pods across nodes. Since each JVB deployment deploys every instance to its own pod with a port unique to the deployment it works with many Jitsi servers. About Jibri, one issue remain: not all cluster can access the sound device of the VM. I am starting to explore how to use a dummy sound driver that can be used as a loop back for recording sound from a pod without needing the physical device. I will most likely develop a K8S plugin device for that. Let me know if you have more questions and feel free to push a PR with your scripts for GKE if you are using the chart :) |
|
Sure I will start looking into this from tomorrow again. Will raise a PR if there are any changes required.. |
|
Hey @taktakpeops I noticed that you are using - name: DOCKER_HOST_ADDRESS
valueFrom:
fieldRef:
fieldPath: status.hostIP
|
|
Hi @ChrisTomAlx, Looking into the documentation here, it's available since v1.7. Checking here, it seems that GCP is currently supporting 1.15.9, 1.16.5, 1.17 and higher for GKE. In AWS, I am using the latest version supported by EKS which is v1.16.8 (you can find all supported versions of K8S for EKS here: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). Have you tried deploying and noticed that the IP wasn't exported in the environment? |
|
Hi @taktakpeops Thanks for your quick response. You just saved me a whole lot of cluster reconfiguration |
|
Hey @taktakpeops .. I have a couple of questions. If time permits please do get back to me.
|
|
Hello, For question 1: yes, you need to be able to remotely access your nodes which means that they need a public IP. In AWS, I did so by enabling the SSH access in my node group. The public IP are being detected using STUN servers. Question 2, yes, you are right but for that I am writing the logic in helm which will generate the domains for each Jicofo / prosody couple (so work in progress). Question 3, the config are there for testing. I want to start using octo and enable some modules in posody. |
|
Awesome. Thanks for the quick response. There seems to be some issue on my end when using NodePort. The JVB is not accessible. Tried with LoadBalancer and it works. kubectl describe pod {jvb pod} gives me :-Name: jitsi-meet-jvb-0-7c5cdc6f54-slfn5
Namespace: default
Priority: 0
Node: gke-neumeet-pool-1-4768d01b-xd50/10.128.0.26
Start Time: Fri, 22 May 2020 17:15:34 +0530
Labels: app.kubernetes.io/instance=neumeet
app.kubernetes.io/name=jitsi-meet-jvb-0
pod-template-hash=7c5cdc6f54
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container jitsi-meet-jvb
Status: Running
IP: 10.56.4.199
IPs: <none>
Controlled By: ReplicaSet/jitsi-meet-jvb-0-7c5cdc6f54
Containers:
jitsi-meet-jvb:
Container ID: docker://7819c812939676fe72a614256ec57ce8f83a11ab0aa71c2121f9501878a90daf
Image: asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20
Image ID: docker-pullable://asia.gcr.io/webrtcvideo-274408/neutrinos/jvb@sha256:a2e405eb74e33accc0e62ec1a798d918874b505fe58cd4f0e77bc0ac719fff43
Port: 30300/UDP
Host Port: 0/UDP
State: Running
Started: Fri, 22 May 2020 17:15:37 +0530
Ready: True
Restart Count: 0
Requests:
cpu: 100m
Environment:
XMPP_SERVER: jitsi-meet-prosody
PROSODY_INSTANCE: 0
JICOFO_AUTH_USER: focus
JICOFO_AUTH_PASSWORD: <set to the key 'JICOFO_AUTH_PASSWORD' in secret 'jitsi-meet-jicofo-config'> Optional: false
JVB_AUTH_USER: jvb
JVB_AUTH_PASSWORD: <set to the key 'JVB_AUTH_PASSWORD' in secret 'jitsi-meet-jvb-config'> Optional: false
JICOFO_COMPONENT_SECRET: <set to the key 'JICOFO_COMPONENT_SECRET' in secret 'jitsi-meet-jicofo-config'> Optional: false
JVB_PORT: 30300
JVB_STUN_SERVERS: stun.l.google.com:19302,stun1.l.google.com:19302,stun2.l.google.com:19302
JVB_TCP_HARVESTER_DISABLED: true
DOCKER_HOST_ADDRESS: (v1:status.hostIP)
JVB_OPTS: --apis=xmpp,rest
ENABLE_STATISTICS: true
XMPP_DOMAIN: testmeet.neutrinos.co
XMPP_AUTH_DOMAIN: auth.jitsi-meet-prosody.default.svc
XMPP_INTERNAL_MUC_DOMAIN: internal-muc.jitsi-meet-prosody.default.svc
XMPP_MUC_DOMAIN: muc.testmeet.neutrinos.co
XMPP_GUEST_DOMAIN: guest.jitsi-meet-prosody.default.svc
JVB_BREWERY_MUC: jvbbrewery
TZ: Europe/Amsterdam
Mounts:
/defaults from config (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from neumeet-jitsi-meet-token-tclkm (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: jitsi-meet-jvb-config-cm
Optional: false
neumeet-jitsi-meet-token-tclkm:
Type: Secret (a volume populated by a Secret)
SecretName: neumeet-jitsi-meet-token-tclkm
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30m default-scheduler Successfully assigned default/jitsi-meet-jvb-0-7c5cdc6f54-slfn5 to gke-neumeet-pool-1-4768d01b-xd50
Normal Pulling 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Pulling image "asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20"
Normal Pulled 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Successfully pulled image "asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20"
Normal Created 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Created container jitsi-meet-jvb
Normal Started 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Started container jitsi-meet-jvbAnyhow just dropped this here in case you find something off. I will look into stun servers I think that must be why I am having issues. Because I am able to ping the external ip of the node from an external network but not with UDP which is strange. So from a first look stun servers should solve this issue if I am not wrong. |
|
@ChrisTomAlx, for the stun + public IP detection, you should see that in the logs of JVB. Did you make sure that the security group for your nodes accept traffic on port 30300? |
|
@taktakpeops Yea will look through the logs for some hints |
|
@ChrisTomAlx : it’s not sure that the nodes will have the port open. In EKS, for example, the public network for the nodes has its own security group which has to be modified accordingly after deploying Jitsi to allow UDP traffic on the node. |
|
Hey @taktakpeops you were right. There was a firewall blocking UDP traffic, trying to get around it. |
|
Hey @ChrisTomAlx - thanks for the suggestion. After playing around with Alsa (used currently in Jitsi) - I ended up realizing that it requires a modification of the node OS, therfore specialized nodes for running Jibri. I saw quite a few other people also modifying the config of Jibri for using PulseAudio. After going through issues / pull-requests and the forum, I found out that PulseAudio was originally used in Jibri. The Jitsi team dropped it because of stability issues, while recording calls with a large amount of participants they had drops / packet losts. They replaced it by Alsa. I want to keep the Helm chart with the official containers. For now, I am looking into how the AMIs for EKS are configured and how to enable some specific kernel modules as the one missing currently: Once The custom AMI will be then used for all the nodes. If X recordings at a time aren't possible, we can still run one Jibri per nodes. I thought also about another (weird and terrible) thing: running Jibri inside a VM running inside a container running Qemu 🤣 |
This is intriguing, but I am even scared to ask about it after reading up a bit on Qemu. Although it could work right? It will be one heavy pod but I think anything is better than making node level changes
Nice catch, would have spend a lot of time going down that rabbit hole otherwise.
Agree. I would prefer the official ones as well. Then we can simply change container images and stay up to date with new features..
For GCP I could change the node images to ubuntu then ssh into each of the VMs and change them manually as suggested by the Jitsi Team and this pr. Although this won't work in the case of node auto scaling. Hence my plan is to create a daemonset that run on all nodes and adjusts them to run jibri. Just a side note remember one jibri pod can only do one recording at a time. This video gave me a lot of insight into jibri. Leaving it here in case you want to watch it as well. |
|
Also thanks a ton for the firewall tip.. Got JVB to run as expected.. Testing scaling next |
Yes - it could work but it would also be a hell to debug - it's a high level of inception :D Agreed - it's pretty tricky to gather info about jitsi as it's all over the place For GCP, we might have the same issue as with AWS which is that the VM don't have a sound device. I am looking further into dummy-snd but it remains a kernel module and not included in Ubuntu on AWS EC2. Will keep you posted once I find a workaround for that |
|
I just followed the below steps as shown in the pr for each node on GCP and I was able to get jibri to run. For each node in the cluster do the following:Install generic kernel imageChange grub to start node VM with generic kernelReplace #GRUB_DEFAULT=0 with Reboot VMSetup virtual sound device in the nodeinstall the moduleconfigure 5 capture/playback interfacessetup autoload the moduleRebootcheck that the module is loaded |
|
Hey @ChrisTomAlx, thank you for sharing. I don’t see in your steps anything related to setting up the snd-dummy component, is it present in your nodes? |
|
The Setup virtual sound device in the node section handles that I assume. You can also refer this. The jitsi official documentation mentions how to set it up on AWS. Although I guess they meant it for VM's. In GKE the the kubernetes nodes are also VM's. I am not sure if its the same in AWS For GCP if node auto scaling and node auto repair are on these changes won't stick. So I had to turn these off as well. |
|
Here is my jibri yaml file. You might want to change it according to your needs slightly.. |
|
If you run « aplay -L » in your terminal, what does it print? Also, master and nodes are VM. Inside of the node VM, kubelet must be running to subscribe to the master. Then, pod scheduling happens in this VM. |
It just says
Yes, makes sense, similar to GKE. |
|
Ah oki, i installed alsa on the VM to test my OS setup. Got it to work on AWS. Now, for Jibri, how do you mount the device for the pods? |
|
For the EC2 instance, I had to use a different Could you share the logs from Jibri to see how alsa gets init and so on? |
|
Sorry - checked everything, looks good. Regarding the logs, I was just wondering if alsa is being loaded correctly or not. Because there are no health checks for Jibri, it can be in crashed inside of the pod but still considered as healthy as the process keeps running (in all the Jitsi Docker containers, the software runs as a daemon). I am going to script the setup for the EKS nodes. One issue remain: currently, only 1 jibri per nodes can run. 2 instances on one node would create a conflict for accessing the device |
|
no worries 😄
If you look at the
I actually did run two jibri pods in the same node. with different interfaces. One as Loopback and the other as Loopback_1. I think the best way to do this, since it has to happen dynamically per pod is to have a file at hostpath which keeps a list of interfaces already being used and then adjust the file as per the postStart and preStop lifecycle hooks of the pods. So all the pods can access the same file and edit it as required. What do you think about this approach ?
You are right, health checks are an issue as well I guess... |
|
I think the best approach remains having a device plugin for managing the sound card. Still work in progress on my side 😄 |
|
Hi guys! Awesome work on this so far, thank you both so much. I'm just coming here to point out that on GKE, it seems that you don't need all the trickery with GRUB and a non-GCP boot image. If you are running Ubuntu, you can simply |
|
Hey @VengefulAncient Although I will still have to do the following I assumeconfigure 5 capture/playback interfacessetup autoload the moduleRebootMaybe autoload and reboot isn't required but configuring playback must be correct.? |
|
@VengefulAncient : thank you for the tip ! I assume that using the following for AWS does the same: https://packages.ubuntu.com/bionic/linux-modules-extra-aws Testing it now and will update the documentation accordingly. |
|
@ChrisTomAlx Unfortunately, it looks like the other steps are still required. The module is only installed for the new GCP image, which is not in use until the node is rebooted. I haven't gotten around to the DaemonSet yet, but perhaps its script could check if the |
|
@VengefulAncient Agreed. I am a bit new too. Daemonsets should be able to handle this as per my understanding. |
|
@VengefulAncient, the script in the docker container takes care of it: https://github.com/jitsi/docker-jitsi-meet/blob/master/jibri/rootfs/etc/cont-init.d/10-config#L25 |
|
@taktakpeops Sadly, that only seems to exit if |
|
I see - but I don't think it needs to be a modification of the node by kubernetes. Instead, wouldn't it be easier to customize the script starting the VM? At the end, we need the update of the VM + reboot before launching the kubelet |
Here is the Daemonset I am working with.kind: DaemonSet
apiVersion: apps/v1
metadata:
name: daejibri
spec:
selector:
matchLabels:
name: daejibri # Label selector that determines which Pods belong to the DaemonSet
template:
metadata:
labels:
name: daejibri # Pod template's label selector
spec:
hostPID: true
# nodeSelector:
# type: jibri
containers:
- name: daejibri
image: asia.gcr.io/google-containers/startup-script:v1
securityContext:
privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
env:
- name: STARTUP_SCRIPT
value: |
! /bin/bash
mkdir -p /chris/hey/
# apt-get install linux-modules-extra-gcp
# sudo reboot
# echo done The asia.gcr.io/google-containers/startup-script:v1 image already exists in GCP by default. I think something similar might be there for AWS. I ran the the daemonset and noticed node level changes after the daemonset ran (meaning I did One thing I would add is Node selectors for both jibri and daemonset pods. So these nodes will have only jibri running on them |
@taktakpeops I am not sure that is possible on GKE. But I recall reading that is possible on AWS. This is from memory so I could be wrong |
|
It seems possible using a gcloud config file (https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance#using_cloud-init_with_the_cloud_config_format) All the setup of the node is managed through system.d (https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#system_initialization) - so the script checking the setup and exiting in case of issues would then be a system.d service. For installing the dependencies and doing the reboot (when needed), the flag |
|
It's to be tested, I don't have a GKE cluster available currently to confirm |
|
I will look into it. But I am not sure this will work. Because GKE VM's and Compute engine VM's don't always follow the same rules. I ran into a similar issue with firewall, because I could not add firewall rules to VM's within kubernetes so I had to go around it by editing an already existing rule. In https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--metadata, It specifically mentions the startup-script is reserved for use, which is understandable since GKE sets most of it up. The master worker setup and all the kubernetes components Additionally, the following keys are reserved for use by Kubernetes Engine:
cluster-location
cluster-name
cluster-uid
configure-sh
enable-os-login
gci-update-strategy
gci-ensure-gke-docker
instance-template
kube-env
startup-script
user-dataStill thanks for the in-depth research and all the links. Very helpful. Let me check if its doable. |
|
Let me know how it goes. On another hand, a custom AMI or base Image seems the best pattern to follow: kubernetes/kops#387 |
|
Yes I agree. Unforunately I don't think GKE VM's support custom base images while GCP's compute engine VM's do. |
|
@ChrisTomAlx I don't think it's necessarily such a bad thing. Frankly, I'd prefer it if the configuration was done by the DaemonSet - this approach lends itself much better to IaC (Infrastructure as Code). Doing additional configuration fetching scripts that have to be stored somewhere that will differ between providers is less maintainable. Ideally, all I'd want to do is add a node pool with label/taint that will prevent other pods from being scheduled there, deploy the chart with a few customized values, and have everything configure itself. The privileged DaemonSet configuring the nodes for us plays into that nicely - if it can do that. |
|
@VengefulAncient I am also leaning towards the daemonset currently, mostly because I know it can be done. Only two problems I see. |
|
I have compiled all my work over the past month here. Hope this helps people who stumble on this. Some points to note.
My FInished deployment files. Please replace the two docker images with your ownPlease change the deployment images ( Jibri and getRecording )as you require. # DAEMONSET FOR STARTUP SCRIPTS
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: daejibri
spec:
selector:
matchLabels:
name: daejibri # Label selector that determines which Pods belong to the DaemonSet
template:
metadata:
labels:
name: daejibri # Pod template's label selector
spec:
hostPID: true
nodeSelector:
type: jibri
containers:
- name: daejibri
image: asia.gcr.io/google-containers/startup-script:v1
securityContext:
privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
env:
- name: STARTUP_SCRIPT
value: |
! /bin/bash
mkdir -p /yourcompany;
if [ -z "$(lsmod | grep -om1 snd_aloop)" ];
then
sudo apt update --yes && sudo apt-get install linux-modules-extra-gcp --yes && sudo echo "options snd-aloop enable=1 index=0" > /etc/modprobe.d/alsa-loopback.conf && sudo echo "snd-aloop" >> /etc/modules && sudo reboot
fi
---
# DEPLOYMENT FOR JIBRI MAIN
apiVersion: apps/v1
kind: Deployment
metadata:
name: yourcompany-jitsi-meet-jibri
spec:
replicas: 1
selector:
matchLabels:
app: yourcompany-jitsi-meet-jibri
template:
metadata:
labels:
app: yourcompany-jitsi-meet-jibri
spec:
nodeSelector:
type: jibri
serviceAccountName: yourcompany-jitsi-meet
securityContext:
fsGroup: 999
volumes:
- name: dev-snd
hostPath:
path: "/dev/snd"
type: Directory
- name: dev-shm
hostPath:
path: "/dev/shm"
type: Directory
- name: test-volume-claim
persistentVolumeClaim:
claimName: test-volume-claim
containers:
- name: yourcompany-jitsi-meet-jibri
image: 11111<===replace with your jibri image or the latest on dockerhub===>11111
imagePullPolicy: Always
resources:
requests:
memory: ".8Gi"
cpu: "2.0"
limits:
memory: "1.0Gi"
cpu: "2.1"
volumeMounts:
- mountPath: "/dev/snd"
name: dev-snd
- mountPath: "/dev/shm"
name: dev-shm
- mountPath: "/data/recordings"
name: test-volume-claim
securityContext:
privileged: true
capabilities:
add:
- SYS_ADMIN
lifecycle:
postStart:
exec:
command:
- "sh"
- "-c"
- >
mkdir -p "/config/recordings";
mkdir -p "/data/recordings";
echo "mv -f /config/recordings/* /data/recordings" > /config/finalize.sh;
envFrom:
- configMapRef:
name: yourcompany-jitsi-meet-jicofo
- configMapRef:
name: yourcompany-prosody-common
- configMapRef:
name: yourcompany-jitsi-meet-web
env:
- name: DISPLAY
value: ':0'
- name: JIBRI_FINALIZE_RECORDING_SCRIPT_PATH
value: /config/finalize.sh
- name: JIBRI_STRIP_DOMAIN_JID
value: muc
- name: JIBRI_LOGS_DIR
value: /config/logs
- name: JIBRI_RECORDING_DIR
value: /config/recordings
---
# DEPLOYMENT FOR JIBRI GET-RECORDING API
apiVersion: apps/v1
kind: Deployment
metadata:
name: yourcompany-jitsi-meet-getrecording
spec:
replicas: 3
selector:
matchLabels:
app: yourcompany-jitsi-meet-getrecording
template:
metadata:
labels:
app: yourcompany-jitsi-meet-getrecording
spec:
volumes:
- name: test-volume-claim
persistentVolumeClaim:
claimName: test-volume-claim
containers:
- name: yourcompany-jitsi-meet-getrecording
image: 11111<===Your get recording docker image===>11111
imagePullPolicy: Always
volumeMounts:
- mountPath: "/data/recordings"
name: test-volume-claim
lifecycle:
postStart:
exec:
command:
- "sh"
- "-c"
- >
mkdir -p "/data/recordings";
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-volume-claim
spec:
storageClassName: "nfs"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
---
# SERVICE TO GET RECORDING
apiVersion: v1
kind: Service
metadata:
name: yourcompany-meet-get-recording
labels:
app: yourcompany-jitsi-meet-getrecording
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: yourcompany-jitsi-meet-getrecording
---
# HORIZONTAL POD SCALER FOR JIBRI POD
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: yourcompany-jitsi-meet-jibri-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: yourcompany-jitsi-meet-jibri
minReplicas: 3
maxReplicas: 6
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 3
---
# HORIZONTAL POD SCALER FOR JVB POD
# apiVersion: autoscaling/v2beta1
# kind: HorizontalPodAutoscaler
# metadata:
# name: yourcompany-jitsi-meet-jibri-hpa
# spec:
# scaleTargetRef:
# apiVersion: apps/v1
# kind: Deployment
# name: yourcompany-jitsi-meet-jibri
# minReplicas: 3
# maxReplicas: 6
# metrics:
# - type: Resource
# resource:
# name: cpu
# targetAverageUtilization: 3My get recording source code - NodeJSIts not perfect but it mostly works. Make sure to change yourJitsiMeetFullDomainName. Use this to dockerize it const http = require('http');
const fs = require('fs');
const os = require('os');
var glob = require("glob");
const path = require("path");
// var contentDisposition = require('content-disposition')
const server = http.createServer((function (req, res) {
if (req.url != '/favicon.ico') {
try {
let url = req.url;
let filePattern = url.split("/")[url.split("/").length - 1].trim();
filePattern = filePattern.toLowerCase();
// let recordingPath= "D:" + path.sep + "Work" + path.sep + "yourcompany" + path.sep + "Experiments" + path.sep + "jitsi-helm" + path.sep + "data" + path.sep + "recordings";
let recordingPath = path.sep + "data" + path.sep + "recordings";
let pathSplitter = os.type() == "Windows_NT" ? '/' : path.sep;
let files = glob.sync(recordingPath + "/**/*.mp4", {});
console.log(files);
let fileNames=[];
let validFilePaths=[];
// Get only valid file paths and file names for the corresponding conference
for (let index = 0; index < files.length; index++) {
let fullFilePath = files[index];
let fileName = fullFilePath.split(pathSplitter)[fullFilePath.split(pathSplitter).length - 1];
if(fileName.includes(filePattern) && filePattern !== "") {
fileNames.push(fileName);
validFilePaths.push(fullFilePath);
}
console.log("fileName",fileName);
}
console.log("fileNames");
console.log(fileNames);
console.log("validFilePaths");
console.log(validFilePaths);
// Show no conference found message if not found else get the array of files and serve the latest
if(fileNames.length == 0) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Sorry no such conference found! Please recheck the conference name and open yourJitsiMeetFullDomainName/api/getrecording/<conference_name> in your browser');
} else {
// Get the array of dates
let dates = [];
for (let index = 0; index < fileNames.length; index++) {
let fullFileName = fileNames[index];
let fileHalves = fullFileName.split("_");
let timeAndExtHalf = fileHalves[fileHalves.length - 1];
let timeHalf = timeAndExtHalf.split(".")[0];
let extHalf = timeAndExtHalf.split(".")[1];
let timeSplit = timeHalf.split("-");
dates.push(new Date(timeSplit[0]+'/'+timeSplit[1]+'/'+timeSplit[2]+' '+timeSplit[3]+':'+timeSplit[4]+':'+timeSplit[5]));
console.log(timeHalf);
console.log(extHalf);
}
console.log(dates);
let max = dates.reduce(function (a, b) { return a > b ? a : b; });
let maxDateString = max.toISOString().replace(/T/, '-').replace(/\..+/, '').replace(/:/g, '-');
console.log(maxDateString);
let finalfileName = fileNames.reduce(function (a, b) { return a.includes(maxDateString) ? a : b; });
let finalFilePath = validFilePaths.reduce(function (a, b) { return a.includes(finalfileName) ? a : b; });
// Download the file
var stream = fs.createReadStream(finalFilePath);
res.writeHead(200, { 'Content-disposition': 'attachment; filename='+finalfileName+'.mp4', 'Content-Type' : 'video/mp4' });
stream.pipe(res); // also you can set content-type
}
} catch (error) {
console.log(error);
}
}
}));
server.listen(80, () => {
console.log('server started');
}); |
|
Hi @ChrisTomAlx - sorry for the late reply I was a but busy on another project ! Thank you for your feedback, will look carefully into it today ! As it seems that the chart works, let's start preparing for pushing that to the central Helm repo? :D |
|
Incidentally, same situation as @taktakpeops for me, we're just getting back to Jitsi after some other stuff that took priority away from it. I have a few questions for both of you that I hope you would be able to help me with:
@ everyone: Each JVB pod needs its own UDP port. This brings up two problems.
As always, thank you both for continuing to look into this, your efforts are highly appreciated. |
|
@taktakpeops Sure.. Although I am not entirely certain helm is still accepting charts into their repo. @VengefulAncient Here are my views on these questions :-
Hope this helps!! |
|
@ChrisTomAlx : you are right. It’s not about pushing to their repo but making the chart available by following these guidelines: https://github.com/helm/hub/blob/master/Repositories.md Regarding the HPA, it applies only to the web element and the JVB element which can be scaled. Jibri + Jicofo could be moved from a statefulset to a daemonset, would be more logical I think. The PVC such as the broadcaster are optional, if you don’t want them you can disable it in the value file used to deploy your chart. For the recording part, not sure Alsa is the best solution as we discussed earlier with @ChrisTomAlx - more investigations are on going on my side. Regarding the HPA for JVB, I want to get it to work in sync with Octo. About Jibri, an HPA doesn’t make much sense I think as you would require also vertical scaling (more sound devices). |
@ everyone
Also, a new question came up: in my deployment, Jibri sometimes becomes ready before Prosody, so it (predictably) can't authenticate and just fails (which is where a k8s-native application would just kill the pod and restart, but alas, no such thing with Jibri). Normally I'd solve that by attaching an init container that would poll Prosody service, but since none of these components have proper healthcheck endpoints, this isn't going to work - Prosody only returns XMPP, not HTTP status codes I could use for Kubernetes health checks. Have either of you run into this issue? If so, how do you handle it? The official Jitsi TIA and sorry for another wall of text 😢 |
But how do you scale it? You need an HPA right to check and scale based on cpu utilisation? and HPA's don't work on daemonsets as far as I know. Or atleast they are not designed for that purpose.
Exactly but I had to use podAntiAffinity and that isn't infinitely scaleable so I went with the one pod per node concept. Also I was experiencing random crashes which could be solved by adding more cpu maybe. But I stopped going down that rabbit hole after the podAntiAffinity issue.
So what I was planning is to have a HPA for JVB. Again with one JVB per Node. firewall rule will only open one particular UDP port in all the nodes of the JVB nodepool. This is again vertical scaling so I would not suggest this setup for on prem deployments.
I had that issue as well.. For now I am deploying jibri only once prosody is up.. But there has to be a better way. If prosody restarts for whatever reason all jibri pods will go down with no restart. So that is an issue.. If you do find a workaround do post it here. |
|
+1 |
|
Hello @ChrisTomAlx, Sorry (again) for the late reply. In this case, you aren't scaling horizontally but vertically. So if you need a new daemonset, you would spawn a new node. However, now that I am running Jitsi at scale in a EC2 infrastructure, I realized that for JVB, Octo can work in K8s using the If we are auto scaling the nodes of the clusters for JVB, Jibri can benefit from the same logic (so 2 daemonset in this case). Can do, but need custom image for Jicofo + JVB (sip properties) - look at first answer. I still think that one pod containing Jicofo + Prosody + Jibri is the way to go. Basically some kind of main pod that you would scale vertically as explained for the daemonset. I will be available tomorrow, if you are interested, we can plan a call online to answer most questions ! |
|
@VengefulAncient : will reply to your questions today. As suggested in my previous answer, we can have a call to go through all questions once for all :D |
Following a chat on the issue jitsi/docker-jitsi-meet#565 for Jitsi Meet in K8S using this chart, I am moving the discussion specifically related to GKE here.
@ChrisTomAlx - could you share your findings here?
I will also provide you some support.
The text was updated successfully, but these errors were encountered: