Add a more systematic way to add transparent-proxy config

[lahabana]

Description

At the moment adding transparent proxy config is very cumborsome and error prone.

You have to:

1. Add to

   kuma/pkg/transparentproxy/config/config.go

   Lines 11 to 43 in 07fcbed

   	type TransparentProxyConfig struct {
   	DryRun bool
   	Verbose bool
   	RedirectPortOutBound string
   	RedirectInBound bool
   	RedirectPortInBound string
   	RedirectPortInBoundV6 string
   	ExcludeInboundPorts string
   	ExcludeOutboundPorts string
   	ExcludedOutboundsForUIDs []string
   	UID string
   	GID string
   	RedirectDNS bool
   	RedirectAllDNSTraffic bool
   	AgentDNSListenerPort string
   	DNSUpstreamTargetChain string
   	SkipDNSConntrackZoneSplit bool
   	ExperimentalEngine bool
   	EbpfEnabled bool
   	EbpfInstanceIP string
   	EbpfBPFFSPath string
   	EbpfCgroupPath string
   	EbpfTCAttachIface string
   	EbpfProgramsSourcePath string
   	VnetNetworks []string
   	Stdout io.Writer
   	Stderr io.Writer
   	RestoreLegacy bool
   	Wait uint
   	WaitInterval uint
   	MaxRetries int
   	SleepBetweenRetries time.Duration
   	}

2. Add to

   kuma/pkg/transparentproxy/config/config.go

   Lines 129 to 169 in 07fcbed

   	type Config struct {
   	Owner Owner
   	Redirect Redirect
   	Ebpf Ebpf
   	// DropInvalidPackets when set will enable configuration which should drop
   	// packets in invalid states
   	DropInvalidPackets bool
   	// IPv6 when set will be used to configure iptables as well as ip6tables
   	IPv6 bool
   	// RuntimeStdout is the place where Any debugging, runtime information
   	// will be placed (os.Stdout by default)
   	RuntimeStdout io.Writer
   	// RuntimeStderr is the place where error, runtime information will be
   	// placed (os.Stderr by default)
   	RuntimeStderr io.Writer
   	// Verbose when set will generate iptables configuration with longer
   	// argument/flag names, additional comments etc.
   	Verbose bool
   	// DryRun when set will not execute, but just display instructions which
   	// otherwise would have served to install transparent proxy
   	DryRun bool
   	// Log is the place where configuration for logging iptables rules will
   	// be placed
   	Log LogConfig
   	// Wait is the amount of time, in seconds, that the application should wait
   	// for the xtables exclusive lock before exiting. If the lock is not
   	// available within the specified time, the application will exit with
   	// an error. Default value *(0) means wait forever. To disable this behavior
   	// and exit immediately if the xtables lock is not available, set this to
   	// nil
   	Wait uint
   	// WaitInterval is the amount of time, in microseconds, that iptables should
   	// wait between each iteration of the lock acquisition loop. This can be
   	// useful if the xtables lock is being held by another application for
   	// a long time, and you want to reduce the amount of CPU that iptables uses
   	// while waiting for the lock
   	WaitInterval uint
   	// Retry allows you to configure the number of times that the system should
   	// retry an installation if it fails
   	Retry RetryConfig
   	}

3. Add to config in kumactl install:
   

   kuma/app/kumactl/cmd/install/install_transparent_proxy.go

   Lines 20 to 51 in 07fcbed

   	type transparentProxyArgs struct {
   	DryRun bool
   	Verbose bool
   	RedirectPortOutBound string
   	RedirectInbound bool
   	RedirectPortInBound string
   	RedirectPortInBoundV6 string
   	ExcludeInboundPorts string
   	ExcludeOutboundPorts string
   	ExcludeOutboundTCPPortsForUIDs []string
   	ExcludeOutboundUDPPortsForUIDs []string
   	ExcludeOutboundPortsForUIDs []string
   	UID string
   	User string
   	RedirectDNS bool
   	RedirectAllDNSTraffic bool
   	AgentDNSListenerPort string
   	DNSUpstreamTargetChain string
   	StoreFirewalld bool
   	SkipDNSConntrackZoneSplit bool
   	EbpfEnabled bool
   	EbpfProgramsSourcePath string
   	EbpfInstanceIP string
   	EbpfBPFFSPath string
   	EbpfCgroupPath string
   	EbpfTCAttachIface string
   	VnetNetworks []string
   	Wait uint
   	WaitInterval uint
   	MaxRetries int
   	SleepBetweenRetries time.Duration
   	}

4. Add to args:
   

   kuma/app/kumactl/cmd/install/install_transparent_proxy.go

   Lines 190 to 222 in 07fcbed

   	cmd.Flags().BoolVar(&args.DryRun, "dry-run", args.DryRun, "dry run")
   	cmd.Flags().BoolVar(&args.Verbose, "verbose", args.Verbose, "verbose")
   	cmd.Flags().StringVar(&args.RedirectPortOutBound, "redirect-outbound-port", args.RedirectPortOutBound, "outbound port redirected to Envoy, as specified in dataplane's `networking.transparentProxying.redirectPortOutbound`")
   	cmd.Flags().BoolVar(&args.RedirectInbound, "redirect-inbound", args.RedirectInbound, "redirect the inbound traffic to the Envoy. Should be disabled for Gateway data plane proxies.")
   	cmd.Flags().StringVar(&args.RedirectPortInBound, "redirect-inbound-port", args.RedirectPortInBound, "inbound port redirected to Envoy, as specified in dataplane's `networking.transparentProxying.redirectPortInbound`")
   	cmd.Flags().StringVar(&args.RedirectPortInBoundV6, "redirect-inbound-port-v6", args.RedirectPortInBoundV6, "IPv6 inbound port redirected to Envoy, as specified in dataplane's `networking.transparentProxying.redirectPortInboundV6`")
   	cmd.Flags().StringVar(&args.ExcludeInboundPorts, "exclude-inbound-ports", args.ExcludeInboundPorts, "a comma separated list of inbound ports to exclude from redirect to Envoy")
   	cmd.Flags().StringVar(&args.ExcludeOutboundPorts, "exclude-outbound-ports", args.ExcludeOutboundPorts, "a comma separated list of outbound ports to exclude from redirect to Envoy")
   	cmd.Flags().StringVar(&args.User, "kuma-dp-user", args.UID, "the user that will run kuma-dp")
   	cmd.Flags().StringVar(&args.UID, "kuma-dp-uid", args.UID, "the uid of the user that will run kuma-dp")
   	cmd.Flags().BoolVar(&args.RedirectDNS, "redirect-dns", args.RedirectDNS, "redirect only DNS requests targeted to the servers listed in /etc/resolv.conf to a specified port")
   	cmd.Flags().BoolVar(&args.RedirectAllDNSTraffic, "redirect-all-dns-traffic", args.RedirectAllDNSTraffic, "redirect all DNS traffic to a specified port, unlike --redirect-dns this will not be limited to the dns servers identified in /etc/resolve.conf")
   	cmd.Flags().StringVar(&args.AgentDNSListenerPort, "redirect-dns-port", args.AgentDNSListenerPort, "the port where the DNS agent is listening")
   	cmd.Flags().StringVar(&args.DNSUpstreamTargetChain, "redirect-dns-upstream-target-chain", args.DNSUpstreamTargetChain, "(optional) the iptables chain where the upstream DNS requests should be directed to. It is only applied for IP V4. Use with care.")
   	cmd.Flags().BoolVar(&args.StoreFirewalld, "store-firewalld", args.StoreFirewalld, "store the iptables changes with firewalld")
   	cmd.Flags().BoolVar(&args.SkipDNSConntrackZoneSplit, "skip-dns-conntrack-zone-split", args.SkipDNSConntrackZoneSplit, "skip applying conntrack zone splitting iptables rules")
   	// ebpf
   	cmd.Flags().BoolVar(&args.EbpfEnabled, "ebpf-enabled", args.EbpfEnabled, "use ebpf instead of iptables to install transparent proxy")
   	cmd.Flags().StringVar(&args.EbpfProgramsSourcePath, "ebpf-programs-source-path", args.EbpfProgramsSourcePath, "path where compiled ebpf programs and other necessary for ebpf mode files can be found")
   	cmd.Flags().StringVar(&args.EbpfInstanceIP, "ebpf-instance-ip", args.EbpfInstanceIP, "IP address of the instance (pod/vm) where transparent proxy will be installed")
   	cmd.Flags().StringVar(&args.EbpfBPFFSPath, "ebpf-bpffs-path", args.EbpfBPFFSPath, "the path of the BPF filesystem")
   	cmd.Flags().StringVar(&args.EbpfCgroupPath, "ebpf-cgroup-path", args.EbpfCgroupPath, "the path of cgroup2")
   	cmd.Flags().StringVar(&args.EbpfTCAttachIface, "ebpf-tc-attach-iface", args.EbpfTCAttachIface, "name of the interface which TC eBPF programs should be attached to")
   	cmd.Flags().StringArrayVar(&args.ExcludeOutboundTCPPortsForUIDs, "exclude-outbound-tcp-ports-for-uids", []string{}, "[DEPRECATED (use --exclude-outbound-ports-for-uids)] tcp outbound ports to exclude for specific uids in a format of ports:uids where ports can be a single value, a list, a range or a combination of all and uid can be a value or a range e.g. 53,3000-5000:106-108 would mean exclude ports 53 and from 3000 to 5000 for uids 106, 107, 108")
   	cmd.Flags().StringArrayVar(&args.ExcludeOutboundUDPPortsForUIDs, "exclude-outbound-udp-ports-for-uids", []string{}, "[DEPRECATED (use --exclude-outbound-ports-for-uids)] udp outbound ports to exclude for specific uids in a format of ports:uids where ports can be a single value, a list, a range or a combination of all and uid can be a value or a range e.g. 53, 3000-5000:106-108 would mean exclude ports 53 and from 3000 to 5000 for uids 106, 107, 108")
   	cmd.Flags().StringArrayVar(&args.ExcludeOutboundPortsForUIDs, "exclude-outbound-ports-for-uids", []string{}, "outbound ports to exclude for specific uids in a format of protocol:ports:uids where protocol and ports can be omitted or have value tcp or udp and ports can be a single value, a list, a range or a combination of all or * and uid can be a value or a range e.g. 53,3000-5000:106-108 would mean exclude ports 53 and from 3000 to 5000 for both TCP and UDP for uids 106, 107, 108")
   	cmd.Flags().StringArrayVar(&args.VnetNetworks, "vnet", []string{}, "virtual networks in a format of interfaceNameRegex:CIDR split by ':' where interface name doesn't have to be exact name e.g. docker0:172.17.0.0/16, br+:172.18.0.0/16, iface:::1/64")
   	cmd.Flags().UintVar(&args.Wait, "wait", args.Wait, "specify the amount of time, in seconds, that the application should wait for the xtables exclusive lock before exiting. If the lock is not available within the specified time, the application will exit with an error")
   	cmd.Flags().UintVar(&args.WaitInterval, "wait-interval", args.WaitInterval, "flag can be used to specify the amount of time, in microseconds, that iptables should wait between each iteration of the lock acquisition loop. This can be useful if the xtables lock is being held by another application for a long time, and you want to reduce the amount of CPU that iptables uses while waiting for the lock")
   	cmd.Flags().IntVar(&args.MaxRetries, "max-retries", args.MaxRetries, "flag can be used to specify the maximum number of times to retry an installation before giving up")
   	cmd.Flags().DurationVar(&args.SleepBetweenRetries, "sleep-between-retries", args.SleepBetweenRetries, "flag can be used to specify the amount of time to sleep between retries")

5. Add to annotations for CNI:
   

   kuma/app/cni/pkg/cni/annotations.go

   Lines 20 to 31 in 07fcbed

   	var annotationRegistry = map[string]*annotationParam{
   	"inject": {"kuma.io/sidecar-injection", "", alwaysValidFunc},
   	"ports": {"kuma.io/envoy-admin-port", "", validatePortList},
   	"excludeInboundPorts": {"traffic.kuma.io/exclude-inbound-ports", defaultRedirectExcludePort, validatePortList},
   	"excludeOutboundPorts": {"traffic.kuma.io/exclude-outbound-ports", defaultRedirectExcludePort, validatePortList},
   	"inboundPort": {"kuma.io/transparent-proxying-inbound-port", defaultInboundPort, validatePortList},
   	"inboundPortV6": {"kuma.io/transparent-proxying-inbound-v6-port", defaultInboundPortV6, validatePortList},
   	"outboundPort": {"kuma.io/transparent-proxying-outbound-port", defaultOutboundPort, validatePortList},
   	"isGateway": {"kuma.io/gateway", "false", alwaysValidFunc},
   	"builtinDNS": {"kuma.io/builtin-dns", "false", alwaysValidFunc},
   	"builtinDNSPort": {"kuma.io/builtin-dns-port", defaultBuiltinDNSPort, validatePortList},
   	}

6. Map correctly:

   kuma/app/cni/pkg/cni/injector_linux.go

   Lines 70 to 142 in 07fcbed

   	func mapToConfig(intermediateConfig *IntermediateConfig, logWriter *bufio.Writer) (*config.Config, error) {
   	port, err := convertToUint16("inbound port", intermediateConfig.targetPort)
   	if err != nil {
   	return nil, err
   	}
   	excludePorts, err := convertCommaSeparatedString(intermediateConfig.excludeOutboundPorts)
   	if err != nil {
   	return nil, err
   	}
   	cfg := config.Config{
   	RuntimeStdout: logWriter,
   	Owner: config.Owner{
   	UID: intermediateConfig.noRedirectUID,
   	},
   	Redirect: config.Redirect{
   	Outbound: config.TrafficFlow{
   	Enabled: true,
   	Port: port,
   	ExcludePorts: excludePorts,
   	},
   	},
   	}
   	isGateway, err := GetEnabled(intermediateConfig.isGateway)
   	if err != nil {
   	return nil, err
   	}
   	inboundPortV6, err := convertToUint16("inbound port ipv6", intermediateConfig.inboundPortV6)
   	if err != nil {
   	return nil, err
   	}
   	enableIpV6, err := transparentproxy.ShouldEnableIPv6(inboundPortV6)
   	if err != nil {
   	return nil, err
   	}
   	cfg.IPv6 = enableIpV6
   	redirectInbound := !isGateway
   	if redirectInbound {
   	inboundPort, err := convertToUint16("inbound port", intermediateConfig.inboundPort)
   	if err != nil {
   	return nil, err
   	}
   	excludedPorts, err := convertCommaSeparatedString(intermediateConfig.excludeInboundPorts)
   	if err != nil {
   	return nil, err
   	}
   	cfg.Redirect.Inbound = config.TrafficFlow{
   	Enabled: true,
   	Port: inboundPort,
   	PortIPv6: inboundPortV6,
   	ExcludePorts: excludedPorts,
   	}
   	}
   	useBuiltinDNS, err := GetEnabled(intermediateConfig.builtinDNS)
   	if err != nil {
   	return nil, err
   	}
   	if useBuiltinDNS {
   	builtinDnsPort, err := convertToUint16("builtin dns port", intermediateConfig.builtinDNSPort)
   	if err != nil {
   	return nil, err
   	}
   	cfg.Redirect.DNS = config.DNS{
   	Enabled: true,
   	Port: builtinDnsPort,
   	CaptureAll: true,
   	ConntrackZoneSplit: true,
   	}
   	}
   	return &cfg, nil
   	}

7. Map transparent proxy annotations too:

   kuma/pkg/transparentproxy/kubernetes/kubernetes.go

   Lines 48 to 137 in 07fcbed

   	func NewPodRedirectForPod(pod *kube_core.Pod) (*PodRedirect, error) {
   	var err error
   	podRedirect := &PodRedirect{}
   	podRedirect.BuiltinDNSEnabled, _, err = metadata.Annotations(pod.Annotations).GetEnabled(metadata.KumaBuiltinDNS)
   	if err != nil {
   	return nil, err
   	}
   	podRedirect.BuiltinDNSPort, _, err = metadata.Annotations(pod.Annotations).GetUint32(metadata.KumaBuiltinDNSPort)
   	if err != nil {
   	return nil, err
   	}
   	podRedirect.ExcludeOutboundPorts, _ = metadata.Annotations(pod.Annotations).GetString(metadata.KumaTrafficExcludeOutboundPorts)
   	excludeOutboundPortsForUIDs, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTrafficExcludeOutboundPortsForUIDs)
   	if exists {
   	podRedirect.ExcludeOutboundPortsForUIDs = strings.Split(excludeOutboundPortsForUIDs, ";")
   	}
   	excludeOutboundTCPPortsForUIDs, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTrafficExcludeOutboundTCPPortsForUIDs)
   	if exists {
   	for _, v := range strings.Split(excludeOutboundTCPPortsForUIDs, ";") {
   	podRedirect.ExcludeOutboundPortsForUIDs = append(podRedirect.ExcludeOutboundPortsForUIDs, fmt.Sprintf("tcp:%s", v))
   	}
   	}
   	excludeOutboundUDPPortsForUIDs, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTrafficExcludeOutboundUDPPortsForUIDs)
   	if exists {
   	for _, v := range strings.Split(excludeOutboundUDPPortsForUIDs, ";") {
   	podRedirect.ExcludeOutboundPortsForUIDs = append(podRedirect.ExcludeOutboundPortsForUIDs, fmt.Sprintf("udp:%s", v))
   	}
   	}
   	podRedirect.RedirectPortOutbound, _, err = metadata.Annotations(pod.Annotations).GetUint32(metadata.KumaTransparentProxyingOutboundPortAnnotation)
   	if err != nil {
   	return nil, err
   	}
   	podRedirect.RedirectInbound = true
   	enabled, exist, err := metadata.Annotations(pod.Annotations).GetEnabled(metadata.KumaGatewayAnnotation)
   	if err != nil {
   	return nil, err
   	}
   	if exist && enabled {
   	podRedirect.RedirectInbound = false
   	}
   	podRedirect.ExcludeInboundPorts, _ = metadata.Annotations(pod.Annotations).GetString(metadata.KumaTrafficExcludeInboundPorts)
   	podRedirect.RedirectPortInbound, _, err = metadata.Annotations(pod.Annotations).GetUint32(metadata.KumaTransparentProxyingInboundPortAnnotation)
   	if err != nil {
   	return nil, err
   	}
   	podRedirect.RedirectPortInboundV6, _, err = metadata.Annotations(pod.Annotations).GetUint32(metadata.KumaTransparentProxyingInboundPortAnnotationV6)
   	if err != nil {
   	return nil, err
   	}
   	podRedirect.UID, _ = metadata.Annotations(pod.Annotations).GetString(metadata.KumaSidecarUID)
   	if value, exists, err := metadata.Annotations(pod.Annotations).GetEnabled(metadata.KumaTransparentProxyingEbpf); err != nil {
   	return nil, err
   	} else if exists {
   	podRedirect.TransparentProxyEnableEbpf = value
   	}
   	if value, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTransparentProxyingEbpfBPFFSPath); exists {
   	podRedirect.TransparentProxyEbpfBPFFSPath = value
   	}
   	if value, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTransparentProxyingEbpfCgroupPath); exists {
   	podRedirect.TransparentProxyEbpfCgroupPath = value
   	}
   	if value, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTransparentProxyingEbpfTCAttachIface); exists {
   	podRedirect.TransparentProxyEbpfTCAttachIface = value
   	}
   	if value, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTransparentProxyingEbpfInstanceIPEnvVarName); exists {
   	podRedirect.TransparentProxyEbpfInstanceIPEnvVarName = value
   	}
   	if value, exists := metadata.Annotations(pod.Annotations).GetString(metadata.KumaTransparentProxyingEbpfProgramsSourcePath); exists {
   	podRedirect.TransparentProxyEbpfProgramsSourcePath = value
   	}
   	return podRedirect, nil
   	}

No-one can get that right from the first time.
Is there a better way to do this? Seems like transparent-proxy already uses pkg/config.Config which enables env var and json config. Should the annotation maybe just: traffic.kuma.io/transparent-proxy-config-json ?

[lahabana]

Is there use cases where we need to set this twice and we need to merge multiple configs?

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.