feature: alb ingress support

[Sn0rt]

this PR want to set alb ingress controller support, this feature require is #1187

if the user use amazon-vpc-cni, the user needs to set correct ingress with alb.ingress.kubernetes.io/target-type":"ip"

and this pr is base of multi-az pr

how to use

to set INGRESS with ALB

$ export AWS_REGION=us-east-2
$ export INGRESS=ALB
$ export AWS_CREDENTIALS=$(cat ~/aws_cluster_deploy_env.sh)
$ export AWS_ACCESS_KEY_ID=$(echo $AWS_CREDENTIALS | jq .AccessKey.AccessKeyId -r) && export AWS_SECRET_ACCESS_KEY=$(echo $AWS_CREDENTIALS | jq .AccessKey.SecretAccessKey -r)
$ export CLUSTER_NAME=testingress
$ export SSH_KEY_NAME=guohao

$ ./examples/generate.sh # crate the cluster spec
Generated addons will include alb ingress controller
Generated /Volumes/home/guohao/workspace/go/src/sigs.k8s.io/cluster-api-provider-aws/examples/_out/addons.yaml
....

the addons yaml will be appned some new sections.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller
  name: alb-ingress-controller
rules:
  - apiGroups:
      - ""
      - extensions
    resources:
      - configmaps
      - endpoints
      - events
      - ingresses
      - ingresses/status
      - services
    verbs:
      - create
      - get
      - list
      - update
      - watch
      - patch
  - apiGroups:
      - ""
      - extensions
    resources:
      - nodes
      - pods
      - secrets
      - services
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller
  name: alb-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: alb-ingress-controller
subjects:
  - kind: ServiceAccount
    name: alb-ingress-controller
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller
  name: alb-ingress-controller
  namespace: kube-system
---
# Application Load Balancer (ALB) Ingress Controller Deployment Manifest.
# This manifest details sensible defaults for deploying an ALB Ingress Controller.
# GitHub: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller
  name: alb-ingress-controller
  # Namespace the ALB Ingress Controller should run in. Does not impact which
  # namespaces it's able to resolve ingress resource for. For limiting ingress
  # namespace scope, see --watch-namespace.
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: alb-ingress-controller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: alb-ingress-controller
    spec:
      hostNetwork: true
      tolerations:
        - operator: Exists
      containers:
        - name: alb-ingress-controller
          args:
            # Limit the namespace where this ALB Ingress Controller deployment will
            # resolve ingress resources. If left commented, all namespaces are used.
            # - --watch-namespace=your-k8s-namespace

            # Setting the ingress-class flag below ensures that only ingress resources with the
            # annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
            # choose any class you'd like for this controller to respect.
            - --ingress-class=alb

            # REQUIRED
            # Name of your cluster. Used when naming resources created
            # by the ALB Ingress Controller, providing distinction between
            # clusters.
            - --cluster-name=testingress

            # AWS VPC ID this ingress controller will use to create AWS resources.
            # If unspecified, it will be discovered from ec2metadata.
            # - --aws-vpc-id=vpc-xxxxxx

            # AWS region this ingress controller will operate in.
            # If unspecified, it will be discovered from ec2metadata.
            # List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
            # - --aws-region=us-west-1

            # Enables logging on all outbound requests sent to the AWS API.
            # If logging is desired, set to true.
            # - --aws-api-debug
            # Maximum number of times to retry the aws calls.
            # defaults to 10.
            # - --aws-max-retries=10
          # env:
            # AWS key id for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_ACCESS_KEY_ID
            #  value: KEYVALUE

            # AWS key secret for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_SECRET_ACCESS_KEY
            #  value: SECRETVALUE
          # Repository location of the ALB Ingress Controller.
          image: docker.io/amazon/aws-alb-ingress-controller:v1.1.3
      serviceAccountName: alb-ingress-controller

to create a cluster and check the cluster status

$ kubectl --kubeconfig $CLUSTER_NAME-kubeconfig get node  -o wide
NAME                                         STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
ip-10-0-101-210.us-east-2.compute.internal   Ready    <none>   12m   v1.15.3   10.0.101.210   <none>        Ubuntu 18.04.3 LTS   4.15.0-1045-aws   containerd://1.2.8
ip-10-0-101-28.us-east-2.compute.internal    Ready    master   18m   v1.15.3   10.0.101.28    <none>        Ubuntu 18.04.3 LTS   4.15.0-1045-aws   containerd://1.2.8
ip-10-0-103-218.us-east-2.compute.internal   Ready    master   16m   v1.15.3   10.0.103.218   <none>        Ubuntu 18.04.3 LTS   4.15.0-1045-aws   containerd://1.2.8
ip-10-0-105-209.us-east-2.compute.internal   Ready    master   17m   v1.15.3   10.0.105.209   <none>        Ubuntu 18.04.3 LTS   4.15.0-1045-aws   containerd://1.2.8


$ kubectl --kubeconfig $CLUSTER_NAME-kubeconfig get ing -n 2048-game -o wide // get ingress info 
NAME               HOSTS   ADDRESS                                                                 PORTS   AGE
2048-alb-ingress   *       72ec27b1-2048game-2048albi-a533-610211184.us-east-2.elb.amazonaws.com   80      12m


$ curl -vv 72ec27b1-2048game-2048albi-a533-610211184.us-east-2.elb.amazonaws.com // sent request
* Rebuilt URL to: 72ec27b1-2048game-2048albi-a533-610211184.us-east-2.elb.amazonaws.com/
*   Trying 13.58.95.50...
* TCP_NODELAY set
* Connected to 72ec27b1-2048game-2048albi-a533-610211184.us-east-2.elb.amazonaws.com (13.58.95.50) port 80 (#0)
> GET / HTTP/1.1
> Host: 72ec27b1-2048game-2048albi-a533-610211184.us-east-2.elb.amazonaws.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 12 Oct 2019 08:23:02 GMT
< Content-Type: text/html
< Content-Length: 3988
< Connection: keep-alive
< Server: nginx/1.8.0
< Last-Modified: Mon, 03 Aug 2015 00:27:03 GMT
< ETag: "55beb557-f94"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>2048</title>

the 2048game yaml as follow

apiVersion: v1
kind: Namespace
metadata:
  name: "2048-game"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: "2048-deployment"
  namespace: "2048-game"
spec:
  replicas: 2
  template:
    metadata:
      labels:
        name: "2048-game"
    spec:
      containers:
      - image: alexwhen/docker-2048
        imagePullPolicy: Always
        name: "2048-game"
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: "service-2048"
  namespace: "2048-game"
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    name: "2048-game"
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "2048-alb-ingress"
  namespace: "2048-game"
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    kubernetes.io/ingress.class: "alb"
  labels:
    app: 2048-alb-ingress
spec:
  rules:
  - host:
    http:
      paths:
      - path: /
        backend:
          serviceName: "service-2048"
          servicePort: 80

[k8s-ci-robot]

Hi @Sn0rt. Thanks for your PR.

I'm waiting for a kubernetes-sigs or kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Sn0rt
To complete the pull request process, please assign justinsb
You can assign the PR to them by writing /assign @justinsb in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[randomvariable]

Can we have this as documentation for now? I have two concerns:

• The examples are becoming quite complex, and it will be difficult to maintain.
• Adding all of these permissions runs in conflict to Restrict IAM policies further to only allow deletion (and possibly update) of resources #608 . Would like to make these opt-in rather than defaults.

[Sn0rt]

Can we have this as documentation for now? I have two concerns:

• The examples are becoming quite complex, and it will be difficult to maintain.
• Adding all of these permissions runs in conflict to Restrict IAM policies further to only allow deletion (and possibly update) of resources #608 . Would like to make these opt-in rather than defaults.

copy that,
firstly, I will append a document to guide how to create a CAPA cluster with alb ingress support.

secondly, I have some doubts about that someone wants to create many clusters with alb ingress controller support or not under one AWS account, what can't I do for this case to set safe permission.

[randomvariable]

Not against having support for it, but we need a better UX then shell scripts. Ties into work that @fabriziopandini is doing with a redesign of clusterctl.

Welcome to have a chat as well as I'm UK based.

[Sn0rt]

@randomvariable ok, I am confused about the safe use of AWS permissions in such a scenario, where users deploy multiple clusters through a manager cluster that uses calico or amazon vpc cni. Enable or disable alb ingress support

[randomvariable]

@Sn0rt Can we make this configurable?

• Create new policies that is clearly identified as being for the ALB ingress controller
• Provide a command line flag to clusterawsadm alpha bootstrap create-stack, clusterawsadm alpha bootstrap generate-iam-policy-docs and generate-cloudformation that enables people to turn it on/off
• Attach it to the correct IAM roles for control-plane or nodes as needed

[Sn0rt]

@Sn0rt Can we make this configurable?

• Create new policies that is clearly identified as being for the ALB ingress controller
• Provide a command line flag to clusterawsadm alpha bootstrap create-stack, clusterawsadm alpha bootstrap generate-iam-policy-docs and generate-cloudformation that enables people to turn it on/off
• Attach it to the correct IAM roles for control-plane or nodes as needed

ok , I will do this later.

[randomvariable]

Again, apologies @Sn0rt for leaving this so long.
As much as we would like to enable the choice of the VPC CNI, I don't think this is the right place for it. I would like to revisit this after we have clusterctl v2 in place which is happening as part of v1alpha3. I have created #1484 as an umbrella issue and would welcome your comments.

/close

[k8s-ci-robot]

@randomvariable: Closed this PR.

In response to this:

Again, apologies @Sn0rt for leaving this so long.
As much as we would like to enable the choice of the VPC CNI, I don't think this is the right place for it. I would like to revisit this after we have clusterctl v2 in place which is happening as part of v1alpha3. I have created #1484 as an umbrella issue and would welcome your comments.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@Sn0rt: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.