Support to set weight and reliable scheduler

[inju-song]

Hi kube-router team.

We use the kube-router for networking in k8s cluster and external load balancing(I think this is the best network solution for k8s, thanks). However, we need to set weight to 0 or non-zero per k8s service and reliable scheduler to serve production k8s cluster using kube-router.

1. kube-router can be provided as several external load balancer but there is no reliable option between each load balancer as far as I know. If a node fails, maybe all connections inside the node are lost.
   • MH is a good solution I think. There is no need to setup extra high availability solution such as keepalived for reliability.
2. To serve service deployment strategies such as blue-green or canary, we need to set weight to zero or non-zero, but can't setup the weight per service. For example,
   • A older service(blue) gets 0 weight and A newer service(green) gets greater than or equal to 1 weight.

Already I developed features to set ipvs scheduler as MH through k8s svc annotation without modifying libnetwork/ipvs and have used for our k8s cluster.

So, I'm wondering if the kube-router have plans to support to setup various weight and reliable scheduler such as MH scheduler.

[murali-reddy]

@inju-song thanks for the report.

AFAIK, maglev is recent addition to Linux kernel? Kube-router would like to integrate a consistent hashing solution. This is something we wanted for current implementation of DSR (direct server return) in kube-rotuer as well. I have not fully explored IP_VS_MH yet.

Please see https://cloudnativelabs.github.io/post/2017-11-01-kube-high-available-ingress/

But kube-router does have support for older schedulers

https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#load-balancing-scheduling-algorithms

If you have any thoughts on how IP_VS_MH can be integrated into kube-router please share.

[inju-song]

@murali-reddy
Thanks for your reply.

Kube-router uses libnetwork/ipvs, but there is no MaglevHashing const variable. But we can just use libnetwork/ipvs functions to add or update MH virtual service without the const variable.

IP_VS_MH can be used with two flags, FALLBACK and PORT such as IP_VS_SH(these scheduler has same value for flags: 0x0008, 0x0010). Perhaps kube-router dose not have annotations or options for ipvs scheduler flags, adding flags may be implemented(I don' know it exactly because have not fully understand kube-router yet).

As you know, IP_VS_MH is recent addition to Linux Kernel. So, we can use the scheduler since v4.18. If kube-router supports IP_VS_MH, the required kernel version should be known to users.

If the MH scheduler flags can be supported(or allowed) by some options, I will be able to implement and make a PR to integrate IP_VS_MH into kube-router.

[murali-reddy]

Perhaps kube-router dose not have annotations or options for ipvs scheduler flags, adding flags may be implemented

there are already flags for some of the schedulers, so its not hard to extend to support IP_VS_MH

For least connection scheduling use:
kubectl annotate service my-service "kube-router.io/service.scheduler=lc"

For round-robin scheduling use:
kubectl annotate service my-service "kube-router.io/service.scheduler=rr"

For source hashing scheduling use:
kubectl annotate service my-service "kube-router.io/service.scheduler=sh"

For destination hashing scheduling use:
kubectl annotate service my-service "kube-router.io/service.scheduler=dh"

If the MH scheduler flags can be supported(or allowed) by some options, I will be able to implement and make a PR to integrate IP_VS_MH into kube-router.

Please see if you can tweak and add support

https://github.com/cloudnativelabs/kube-router/blob/master/pkg/controllers/proxy/network_services_controller.go#L1096-L1107

[aauren]

This looks to have been completed via #564. Please re-open if I missed something here.