local.kyma.dev is not resolved in the cluster

[pbochynski]

Description
Default kyma-gateway uses local.kyma.dev domain. That domain is not resolved inside the cluster. The domain should point to istio-ingressgateway.istio-system.svc.cluster.local instead of 127.0.0.1.

Expected result

kubectl run -i --tty busybox --image=busybox --restart=Never -- nslookup httpbin.local.kyma.dev

Server:         10.43.0.10
Address:        10.43.0.10:53

Name:   istio-ingressgateway.istio-system.svc.cluster.local

Actual result

kubectl run -i --tty busybox --image=busybox --restart=Never -- nslookup httpbin.local.kyma.dev

Server:         10.43.0.10
Address:        10.43.0.10:53

Non-authoritative answer:
Name:   httpbin.local.kyma.dev
Address: 127.0.0.1

Solution

Coredns service supports overrides that can be used to modify the default coredns config map in the kube-sysstem namespace.
See: k3s-io/k3s#7583

This is a bash script with the solution:

# create k3d cluster without trafik ingress
k3d cluster create --api-port 6550 -p '80:80@loadbalancer' -p '443:443@loadbalancer' --k3s-arg '--disable=traefik@server:*'

# create kyma-system namespace and enable istio-injection
kubectl create namespace kyma-system
kubectl label namespace kyma-system istio-injection=enabled --overwrite

# install istio module
kubectl apply -f https://github.com/kyma-project/istio/releases/latest/download/istio-manager.yaml
kubectl apply -f https://github.com/kyma-project/istio/releases/latest/download/istio-default-cr.yaml

# install api-gateway module
kubectl apply -f https://github.com/kyma-project/api-gateway/releases/latest/download/api-gateway-manager.yaml
kubectl apply -f https://github.com/kyma-project/api-gateway/releases/latest/download/apigateway-default-cr.yaml

# optional: trust kyma-gateway-certs
kubectl get secret kyma-gateway-certs -n istio-system -o jsonpath='{.data.tls\.crt}' | base64 --decode > kyma.crt
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain kyma.crt

# patch coredns to resolve custom domain
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  kyma.override: |
    rewrite name regex (.*)\.local\.kyma\.dev istio-ingressgateway.istio-system.svc.cluster.local
EOF

# restart coredns
kubectl rollout restart deployment -n kube-system coredns

# verify local.kyma.dev domain resolution
kubectl run -i --tty busybox --image=busybox --restart=Never -- nslookup httpbin.local.kyma.dev

# cleanup
k3d cluster delete

Remarks
DNS resolution worked before modularization with this code: https://github.com/kyma-project/cli/blob/main/internal/coredns/coredns.go
But the solution proposed here is more clean (assuming it is applied by api-gateway operator).

PR

• Update documentation about installing api-gateway on K3D #1017