Skip to content

Commit

Permalink
Akker (#41)
Browse files Browse the repository at this point in the history
* added icons

* added icons

* http upgrade

* add grafana dashboard

* updated diagram
  • Loading branch information
chrisakker authored and ciroque committed Apr 11, 2023
1 parent 490a4a5 commit ecbbc53
Show file tree
Hide file tree
Showing 22 changed files with 1,861 additions and 10 deletions.
65 changes: 55 additions & 10 deletions docs/InstallationGuide.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,15 @@

<br/>

![Kubernetes](media/kubernetes-icon.png) | ![Nginx Plus](media/nginx-plus-icon.png) | ![NIC](media/nginx-ingress-icon.png)
--- | --- | ---

<br/>

## Pre-Requisites

- Working kubernetes cluster, with admin privleges
- Running nginx-ingress controller, either OSS or Plus. This install guide follows the instructions for deploying an Nginx Ingress Controller here: https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
- Running nginx-ingress controller, either OSS or Plus. This install guide followed the instructions for deploying an Nginx Ingress Controller here: https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
- Demo application, this install guide uses the Nginx Cafe example, found here: https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/ingress-resources/complete-example
- A bare metal Linux server or VM for the external LB Server, connected to a network external to the cluster. Two of these will be required if High Availability is needed, as shown here.
- Nginx Plus software loaded on the LB Server(s). This install guide follows the instructions for installing Nginx Plus on Centos 7, located here: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/
Expand All @@ -19,15 +24,29 @@

## Kubernetes Cluster

<br/>

![Kubernetes](media/kubernetes-icon.png)

<br/>

A standard K8s cluster is all that is required. There must be enough resources available to run the Nginx Ingress Controller, and the Nginx Kubernetes Loadbalancer Controller. You must have administrative access to be able to create the namespace, services, and deployments for this Solution. This Solution was tested on Kubernetes version 1.23. Most recent versions => v1.21 should work just fine.

<br/>

## Nginx Ingress Controller

<br/>

![NIC](media/nginx-ingress-icon.png)

<br/>

The Nginx Ingress Controller in this Solution is the destination target for traffic (north-south) that is being sent to the cluster. The installation of the actual Ingress Controller is outside the scope of this installation guide, but we include the links to the docs for your reference. `The NIC installation must follow the documents exactly as written,` as this Solution refers to the `nginx-ingress` namespace and service objects. **Only the very last step is changed.**

NOTE: This Solution only works with nginx-ingress from Nginx. It will `not` work with the Community version of Ingress, called ingress-nginx. If you are unsure which Ingress Controller you are running, check out the blog on Nginx.com:
NOTE: This Solution only works with nginx-ingress from Nginx. It will `not` work with the Community version of Ingress, called ingress-nginx.

If you are unsure which Ingress Controller you are running, check out the blog on Nginx.com:
https://www.nginx.com/blog/guide-to-choosing-ingress-controller-part-4-nginx-ingress-controller-options


Expand Down Expand Up @@ -62,23 +81,35 @@ spec:

```

Apply the updated nodeport-nkl.yaml Manifest:

```bash
kubectl apply -f nodeport-nkl.yaml
```

<br/>

## Demo Application

<br/>

## Demo Application
![Cafe Dashboard](media/cafe-dashboard.png)

This is not part of the actual Solution, but it is useful to have a well-known application running in the cluster, as a useful target for test commands. The example provided here is used by the Solution to demonstrate proper traffic flows, and application health check monitoring, to determine if the application is running in the cluster. If you choose a different Application to test with, the health checks provided here will NOT work, and will need to be modified to work correctly.
<br/>

This is not part of the actual Solution, but it is useful to have a well-known application running in the cluster, as a known-good target for test commands. The example provided here is used by the Solution to demonstrate proper traffic flows, as well as application health check monitoring, to determine if the application is running in the cluster.

Note: If you choose a different Application to test with, `the Nginx health checks provided here will NOT work,` and will need to be modified to work correctly.

- Deploy the Nginx Cafe Demo application, found here:

https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/ingress-resources/complete-example

- The Cafe Demo Docker image used is an upgraded one, with graphics and additional Request and Response variables added.

https://hub.docker.com/r/nginxinc/ingress-demo
You can use the `cafe.yaml` manifest included.

- Do not use the `cafe-ingress.yaml` file. Rather, use the `cafe-virtualserver.yaml` file that is provided here. It uses the Nginx CRDs to define a VirtualServer, and the related Routes and Redirects needed. The `redirects are required` for the LB Server's health checks to work correctly!

```yaml
Expand Down Expand Up @@ -143,21 +174,30 @@ spec:
## Linux VM or bare-metal LB Server
This is a standard Linux OS system, based on the Linux Distro and Technical Specs required for Nginx Plus, which can be found here: https://docs.nginx.com/nginx/technical-specs/
![Linux](media/linux-icon.png)
This installation guide followed the "Installation of Nginx Plus on Centos/Redhat/Oracle" steps for installing Nginx Plus.
This is any standard Linux OS system, based on the Linux Distro and Technical Specs required for Nginx Plus, which can be found here: https://docs.nginx.com/nginx/technical-specs/
This Solution followed the "Installation of Nginx Plus on Centos/Redhat/Oracle" steps for installing Nginx Plus.
>NOTE: This solution will not work with Nginx OpenSource, as OpenSource does not have the API that is used in this Solution. Installation on unsupported Distros is not recommended or supported.
<br/>
## Nginx Plus LB Server
<br/>
![Nginx Red Plus](media/nginxredplus.png)
<br/>
This is the configuration required for the LB Server, external to the cluster. It must be configured for the following.
- Move the Nginx default Welcome page from port 80 to port 8080. Port 80 will be used by the stream context, instead of the http context.
- API write access enabled on port 9000.
- Plus Dashboard enabled, used for testing, monitoring, and visualization of the solution working.
- Plus Dashboard enabled, used for testing, monitoring, and visualization of the Solution working.
- The `Stream` context is enabled, for TCP loadbalancing.
- Stream context is configured.

Expand Down Expand Up @@ -202,13 +242,13 @@ server {

![NGINX Dashboard](media/nginxlb-dashboard.png)

- Create a new folder for the stream config .conf files. /etc/nginx/stream was used in this Solution.
- Create a new folder for the stream config .conf files. /etc/nginx/stream is used in this Solution.

```bash
mkdir /etc/nginx/stream
```

- Create 2 new `STATE` files for Nginx. These are used to backup the configuration, in case Nginx restarts/reloads.
- Create 2 new `STATE` files for Nginx. These are used to backup the Upstream configuration, in case Nginx is restarted/reloaded.

Nginx State Files Required for Upstreams
- state file /var/lib/nginx/state/nginx-lb-http.state
Expand Down Expand Up @@ -289,7 +329,7 @@ stream {

`Notice that is uses Ports 80 and 443.`

Place this file in the /etc/nginx/stream folder.
Place this file in the /etc/nginx/stream folder, and reload Nginx. Notice the match block and health check directives are for the cafe.example.com Demo application from Nginx.

```bash
# NginxK8sLB Stream configuration, for L4 load balancing
Expand Down Expand Up @@ -338,6 +378,11 @@ stream {

<br/>

![NIC](media/nginx-ingress-icon.png)

<br/>


This is the new Controller, which is configured to watch the k8s environment, the nginx-ingress Service object, and send API updates to the Nginx LB Server when there are changes. It only requires three things.

- New kubernetes namespace and RBAC
Expand Down
70 changes: 70 additions & 0 deletions docs/cafe.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: coffee
spec:
replicas: 3
selector:
matchLabels:
app: coffee
template:
metadata:
labels:
app: coffee
spec:
containers:
- name: coffee
image: nginxinc/ingress-demo # upgraded Cafe Docker image
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: coffee-svc
spec:
type: ClusterIP
clusterIP: None
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: coffee
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tea
spec:
replicas: 3
selector:
matchLabels:
app: tea
template:
metadata:
labels:
app: tea
spec:
containers:
- name: tea
image: nginxinc/ingress-demo
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: tea-svc
labels:
spec:
type: ClusterIP
clusterIP: None
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: tea
129 changes: 129 additions & 0 deletions docs/http/clusters.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# NginxK8sLB HTTP configuration, for L7 load balancing
# Chris Akker, Apr 2023
# HTTP Proxy and load balancing
# 2 k8s Clusters LB with http split clients
# Nginx Kubernetes Loadbalancer
# Upstream servers managed by NKL Controller
# Nginx Key Value store for Split ratios
#
#### clusters.conf

# Define Key Value store, backup state file, timeout, and enable sync

keyval_zone zone=split:1m state=/var/lib/nginx/state/split.keyval timeout=30d sync;
keyval $host $split_level zone=split;

# Main Nginx Server Block for cafe.example.com, with TLS

server {
listen 443 ssl;
status_zone https://cafe.example.com;
server_name cafe.example.com;

ssl_certificate /etc/ssl/nginx/default.crt;
ssl_certificate_key /etc/ssl/nginx/default.key;

location / {
status_zone /;

proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header "Connection" "";
proxy_pass https://$upstream;

}

}

# Cluster1 upstreams

upstream cluster1-https {
zone cluster1-https 256k;
least_time last_byte;
server 10.1.1.10:31317;
server 10.1.1.8:31317;
keepalive 16;
#servers managed by NKL
#state /var/lib/nginx/state/cluster1-https.state;
}

# Cluster2 upstreams

upstream cluster2-https {
zone cluster2-https 256k;
least_time last_byte;
server 10.1.1.11:31390;
server 10.1.1.12:31390;
#servers managed by NKL
#state /var/lib/nginx/state/cluster2-https.state;
}

# HTTP Split Clients Configuration for Cluster1/Cluster2 ratios

split_clients $request_id $split0 {
* cluster2-https;
}

split_clients $request_id $split1 {
1.0% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split5 {
5.0% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split10 {
10% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split25 {
25% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split50 {
50% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split75 {
75% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split90 {
90% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split95 {
95% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split99 {
99% cluster1-https;
* cluster2-https;
}

split_clients $request_id $split100 {
* cluster1-https;
}

map $split_level $upstream {
0 $split0;
1.0 $split1;
5.0 $split5;
10 $split10;
25 $split25;
50 $split50;
75 $split75;
90 $split90;
95 $split95;
99 $split99;
100 $split100;
default $split50;
}
Loading

0 comments on commit ecbbc53

Please sign in to comment.