Serve robots.txt for Prow (#4365)

* Add reverse proxy deployment

Signed-off-by: Ammar Lakis <[email protected]>

* Serve robots.txt for prow status dashboard

Signed-off-by: Ammar Lakis <[email protected]>

* Fix service selector for reverse proxy

Signed-off-by: Ammar Lakis <[email protected]>

* Add liveness and readiness probes for reverse proxy container

Signed-off-by: Ammar Lakis <[email protected]>

* Host static files on reverse proxy

Signed-off-by: Ammar Lakis <[email protected]>

* Rename reverse proxy to web server

Signed-off-by: Ammar Lakis <[email protected]>

* Add web server to deployment script

Signed-off-by: Ammar Lakis <[email protected]>

* Move prow static files under cluster folder

Signed-off-by: Ammar Lakis <[email protected]>

* Add explaination for updating static files

Signed-off-by: Ammar Lakis <[email protected]>

* Rewording

Co-authored-by: Przemek Pokrywka <[email protected]>

* Update prow/cluster/README.md

Co-authored-by: Aleksandra Simeonova <[email protected]>

Co-authored-by: Przemek Pokrywka <[email protected]>
Co-authored-by: Aleksandra Simeonova <[email protected]>

prow/cluster/README.md

@@ -11,6 +11,10 @@ This folder contains files related to the configuration of the Prow production c
 The structure of the folder looks as follows:
 
 ```
-  ├── components                        # Definitions of Prow components and cluster configuration.
-  └── resources                         # Helm charts used by the Prow cluster.
+  ├── components            # Definitions of Prow components and cluster configuration.
+  ├── resources             # Helm charts used by the Prow cluster.
+  └── static-files          # Files that will be uploaded to the nginx web server.
 ```
+
+###  Adding static files
+All files added to the `static-files` folder are automatically uploaded by the Prow `config_updater` plugin to the cluster in a ConfigMap. Uploaded files are mounted by the web server in the web root directory. To route traffic for a specific path to the NGINX web server, in order to serve these files, update the Ingress `tls-ing` configuration in `tls-ing_ingress.yaml`.

prow/cluster/components/tls-ing_ingress.yaml

@@ -34,3 +34,7 @@ spec:
         backend:
           serviceName: pushgateway-external
           servicePort: 80
+      - path: /robots.txt
+        backend:
+          serviceName: web-server
+          servicePort: 80

prow/cluster/components/web-server_deployment.yaml

@@ -0,0 +1,82 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: web-server
+  namespace: default
+  labels:
+    app: web-server
+spec:
+  selector:
+    matchLabels:
+      app: web-server
+  template:
+    metadata:
+      labels:
+        app: web-server
+    spec:
+      containers:
+      - name: nginx
+        image: nginx:1.20-alpine
+        ports:
+        - containerPort: 80
+        volumeMounts:
+        - name: config
+          mountPath: /etc/nginx/conf.d/
+        - name: static-files
+          mountPath: /usr/share/nginx/html/
+        livenessProbe:
+          httpGet:
+            path: /robots.txt
+            port: 80
+          initialDelaySeconds: 10
+          periodSeconds: 3
+          timeoutSeconds: 30
+        readinessProbe:
+          httpGet:
+            path: /robots.txt
+            port: 80
+          initialDelaySeconds: 10
+          periodSeconds: 3
+          timeoutSeconds: 30
+      volumes:
+      - name: config
+        configMap:
+          name: web-server-config
+      - name: static-files
+        configMap:
+          name: static-files
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: web-server-config
+  namespace: default
+data:
+  default.conf: |
+    server {
+      listen       80;
+      server_name  _;
+
+      location / {
+        root /usr/share/nginx/html/;
+      }
+
+      location = /50x.html {
+        root   /usr/share/nginx/html;
+      }
+    }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: web-server
+  namespace: default
+  labels:
+    app: web-server
+spec:
+  type: NodePort
+  selector:
+    app: web-server
+  ports:
+  - name: http
+    port: 80

prow/cluster/deploy.sh

@@ -44,6 +44,7 @@ prow_components=(
 "tide_deployment.yaml"
 "tide_service.yaml"
 "tls-ing_ingress.yaml"
+"web_server_deployment.yaml"
 )
 
 function ensure-context() {

prow/cluster/static-files/robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /

prow/plugins.yaml

@@ -49,6 +49,8 @@ config_updater:
       clusters:
         trusted-workload:
           - default
+    prow/cluster/static-files/*:
+      name: static-files
 
 triggers:
   - repos: