Document default mode does not work for minikube

[tamalsaha]

IP stays pending for minikube Loadbalancer Ingress.

[bradenwright]

Any update or workaround for this, I'm having the same issue

[tamalsaha]

@bradenwright, this works fine. If you use LoadBalancer type (the default type), it will only allocate NodePort. You can access the HAProxy pods using minikube ip. Does that make sense?

[bradenwright]

Hmm maybe I doing something wrong. I installed into minikube via:

helm install stable/voyager --set ingressClass=voyager --set cloudProvider=minikube
helm install dlf <-- local helm chart which I can port forward to, ingress from chart is below

apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: voyager-ingress
  namespace: default
  kubernetes.io/ingress.class: "voyager"
spec:
  rules:
  - host: dlf.example.com
    tcp:
      port: '20514'
      backend:
        serviceName: {{ .Release.Name }}-dlf
        servicePort: '10514'

It rsyslog, which has logs being forwarded to it from remote hosts. If I port-forward to the pod 10514, and telnet localhost 10514 telnet connects.

I've set a host record for dlf.example.com -> minikube ip which I've successful done with nginx http ingress before. But I getting in this case I keep getting:

$ telnet dlf.example.com 20514
Trying 192.168.99.101...
telnet: connect to address 192.168.99.101: Operation timed out
telnet: Unable to connect to remote host

$ kc get all
NAME                                                       READY     STATUS    RESTARTS   AGE
po/plundering-pike-dlf-4016503386-fhxxt                         1/1       Running   0          9s
po/voyager-voyager-ingress-1034212999-p578r                1/1       Running   0          9s
po/voyager-xrayed-stoat-4046991754-29h43                   1/1       Running   0          48m

NAME                                       CLUSTER-IP   EXTERNAL-IP   PORT(S)           AGE
svc/kubernetes                             10.0.0.1     <none>        443/TCP           1d
svc/plundering-pike-dlf                   10.0.0.78    <none>        10514/TCP         9s
svc/voyager-voyager-ingress                10.0.0.103   <pending>     20514:30849/TCP   8s
svc/voyager-xrayed-stoat                   10.0.0.25    <none>        56790/TCP         48m

NAME                                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/plundering-pike-dlf .                       1         1         1            0           9s
deploy/voyager-voyager-ingress                1         1         1            1           9s
deploy/voyager-xrayed-stoat                   1         1         1            1           48m

NAME                                                 DESIRED   CURRENT   READY     AGE
rs/plundering-pike-dlf-4016503386                       1         1         0         9s
rs/voyager-voyager-ingress-1034212999                1         1         1         9s
rs/voyager-xrayed-stoat-4046991754                   1         1         1         48m

I noticed the pending line so I thought that was related.

[tamalsaha]

I suppose, you are referring to

svc/voyager-voyager-ingress                10.0.0.103   <pending>     20514:30849/TCP   8s

It stays in pending because the Service voyager-voyager-ingress has type LoadBalancer and minikube will only allocate the NodePort (: 30849). You can safely ignore this pending EXTERNAL_IP and just use the $(minikube ip):30849 .

If you want to not have this pending thing showing, you can tell voyager to create a NodePort type service. To do that use add the ingress.appscode.com/type: NodePort annotation.

apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: voyager-ingress
  namespace: default
  kubernetes.io/ingress.class: "voyager"
  annotations:
    ingress.appscode.com/type: NodePort
spec:
  rules:
  - host: dlf.example.com
    tcp:
      port: '20514'
      backend:
        serviceName: {{ .Release.Name }}-dlf
        servicePort: '10514'

[bradenwright]

I did notice I put the ingress.class in the wrong place, but I had tried it without that. That setting helps remove the "pending"

apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: voyager-ingress
  namespace: default
  annotations:
    ingress.appscode.com/type: NodePort
    kubernetes.io/ingress.class: voyager
spec:
  rules:
  - host: dlf.example.com
    tcp:
      port: '20514'
      backend:
        serviceName: {{ .Release.Name }}-dlf
        servicePort: '10514'

But I can't connect via telnet: telnet dlf.example.com 30849 or telnet dlf.example.com 20514 dlf.example.com == minikube ip, I did also try telnet 192.168.99.101 30849 with or without ingress.class annotation

[tamalsaha]

Regarding telnet, I am not sure what type of service you are running as backend, but this works with curl if you run a http server as backend. For a fully working example:

Run https://github.com/appscode/voyager/blob/master/hack/example/e2e.yaml

$ kubectl get pods,svc
NAME                                       READY     STATUS              RESTARTS   AGE
po/test-server-6c95d94d6b-46qhs            1/1       Running             0          9s
po/voyager-test-ingress-79f657ff4f-x94rp   0/1       ContainerCreating   0          4s

NAME                             CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
svc/kubernetes                   10.0.0.1     <none>        443/TCP        1h
svc/test-server                  10.0.0.103   <none>        80/TCP         9s
svc/voyager-test-ingress         10.0.0.158   <nodes>       80:30361/TCP   9s
svc/voyager-test-ingress-stats   10.0.0.199   <none>        56789/TCP      4s

$ minikube ip
192.168.99.100

$ curl 192.168.99.100:30361
{"type":"http","host":"192.168.99.100:30361","podName":"test-server-6c95d94d6b-46qhs","serverPort":":8080","path":"/","method":"GET","headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["curl/7.47.0"],"X-Forwarded-For":["172.17.0.1"]}}

$ telnet 192.168.99.100 30361
Trying 192.168.99.100...
Connected to 192.168.99.100.
Escape character is '^]'.
g
HTTP/1.0 400 Bad request
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>
Connection closed by foreign host.

[bradenwright]

Thanks for the reply, I'll look through the example and see if I can spot the issue. It's an rsyslog server, setup to receive logs.

# minikube ip
$ telnet dlf.example.com 31066
Trying 192.168.99.101...
telnet: connect to address 192.168.99.101: Operation timed out

# port-forward, connects til I cntrl-c
$ telnet 127.0.0.1 10514
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^CConnection closed by foreign host.

[tamalsaha]

If you can share the YAML files you used to deploy the rsyslog server, I can also give it a try. One thing you can check is, if you create a NodePort service directly to expose the rsyslog server, whether that works or not.

[bradenwright]

Here's the main files from the helm chart, I can share pretty much anything. I find a place to put the helm chart and docker image/file if needed. Only thing I'll have to do is generate a new ca/cert/key, only sensitive info

$ cat templates/deployment.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "distilLogForwarderFullname" . }}
  labels:
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ template "distilLogForwarderFullname" . }}
    spec:
      imagePullSecrets:
      - name: spothero-docker-hub-login
      containers:
      - name: {{ template "distilLogForwarderFullname" . }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        env:
{{ include "distilLogForwarderEnvironment" . | indent 8 }}
        ports:
        - containerPort: {{ .Values.service.internalPort }}
        readinessProbe:
          tcpSocket:
            port: {{ .Values.service.internalPort }}
          initialDelaySeconds: 15
          periodSeconds: 20
        livenessProbe:
          tcpSocket:
            port: {{ .Values.service.internalPort }}
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
{{ toYaml .Values.resources | indent 10 }}

$ cat templates/service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: {{ template "distilLogForwarderFullname" . }}
  labels:
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.externalPort }}
      targetPort: {{ .Values.service.internalPort }}
      protocol: TCP
      name: {{ .Values.service.name }}
  selector:
    app: {{ template "distilLogForwarderFullname" . }}

$ cat templates/ingress.yaml 
apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: voyager-ingress
  namespace: default
  annotations:
    ingress.appscode.com/type: NodePort
    kubernetes.io/ingress.class: voyager
spec:
  rules:
  - host: dlf.example.com
    tcp:
      port: '10514'
      backend:
        serviceName: {{ .Release.Name }}-distil-log-forwarder
        servicePort: '10514'

$ cat values.yaml 
# Default values for distil-log-forwarder.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
  repository: spothero/distil-log-forwarder
  tag: 0.0.1
  pullPolicy: IfNotPresent
service:
  name: distil-log-forwarder
  type: ClusterIP
  externalPort: 10514
  internalPort: 10514
ingress:
  enabled: false
resources:
  limits:
    cpu: 2.0
    memory: 2048Mi
  requests:
    cpu: 0.5
    memory: 512Mi
environment: staging
awsRegion: us-west-2

[tamalsaha]

#895