Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PLNSRVCE-769: Add minio tenant for tekton results #443

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 18 additions & 17 deletions DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,20 +28,21 @@

### **Tools**

| **Component** | **Version** | **Purpose** | **Comments** |
|--------------------|------------------------------------------------------|-----------------------------------------------------------------------------|--------------|
| oc (OpenShift CLI) | see [dependencies.sh](shared/config/dependencies.sh) | To interact with the cluster | Follows OpenShift version |
| kubectl | see [dependencies.sh](shared/config/dependencies.sh) | To interact with the cluster | Follows kubernetes version which follows OpenShift version. We only need either oc or kubectl |
| tkn | see [dependencies.sh](shared/config/dependencies.sh) | To interact with tekton | |
| tkn pac plugin | 0.15.0 | To set up PaC | Optional plugin for customers during the cluster setup phase. Follows PaC Version |
| Argo CD (client) | see [dependencies.sh](shared/config/dependencies.sh) | To run Argo CD related commands | Follows version of argocd engine used in openshift gitops |
| checkov | see [dependencies.sh](shared/config/dependencies.sh) | Validate k8s manifests | |
| hadolint | see [dependencies.sh](shared/config/dependencies.sh) | Validate Dockerfiles | |
| shellcheck | see [dependencies.sh](shared/config/dependencies.sh) | Validate shell scripts | |
| skopeo | 1.y.z | Interact with images | |
| yamllint | see [dependencies.sh](shared/config/dependencies.sh) | Validate YAML | |
| yq | see [dependencies.sh](shared/config/dependencies.sh) | Required for parsing things; used in various scripts throughout the project | Certain features are not supported with versions < 4.18.1. Use Latest version to avoid any issues |
| docker | 20.10.z | For local development only | Only one of docker or podman is required. No requirement to use a particular version; users can install the latest version available at the time |
| podman | 4.0.0 | For local development only | Only one of docker or podman is required. No requirement to use a particular version; users can install the latest version available at the time |
| openssl | 3.0.2 | To manipulate certificate information during cluster regsitration | |
| bitwarden | see [dependencies.sh](shared/config/dependencies.sh) | To store credentials outside the gitops repository | |
| **Component** | **Version** | **Purpose** | **Comments** |
|--------------------|------------------------------------------------------|-----------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| oc (OpenShift CLI) | see [dependencies.sh](shared/config/dependencies.sh) | To interact with the cluster | Follows OpenShift version |
| kubectl | see [dependencies.sh](shared/config/dependencies.sh) | To interact with the cluster | Follows kubernetes version which follows OpenShift version. We only need either oc or kubectl |
| tkn | see [dependencies.sh](shared/config/dependencies.sh) | To interact with tekton | |
| tkn pac plugin | 0.15.0 | To set up PaC | Optional plugin for customers during the cluster setup phase. Follows PaC Version |
| Argo CD (client) | see [dependencies.sh](shared/config/dependencies.sh) | To run Argo CD related commands | Follows version of argocd engine used in openshift gitops |
| checkov | see [dependencies.sh](shared/config/dependencies.sh) | Validate k8s manifests | |
| hadolint | see [dependencies.sh](shared/config/dependencies.sh) | Validate Dockerfiles | |
| shellcheck | see [dependencies.sh](shared/config/dependencies.sh) | Validate shell scripts | |
| skopeo | 1.y.z | Interact with images | |
| yamllint | see [dependencies.sh](shared/config/dependencies.sh) | Validate YAML | |
| yq | see [dependencies.sh](shared/config/dependencies.sh) | Required for parsing things; used in various scripts throughout the project | Certain features are not supported with versions < 4.18.1. Use Latest version to avoid any issues |
| docker | 20.10.z | For local development only | Only one of docker or podman is required. No requirement to use a particular version; users can install the latest version available at the time |
| podman | 4.0.0 | For local development only | Only one of docker or podman is required. No requirement to use a particular version; users can install the latest version available at the time |
| openssl | 3.0.2 | To manipulate certificate information during cluster registration | |
| bitwarden | see [dependencies.sh](shared/config/dependencies.sh) | To store credentials outside the gitops repository | |
| minio | 4.5.x | S3 compatible storage for tekton-results api server | Inconsistency in the operator catalogs available on HyperShift currently lead to variance of the version of minio seen in CI. Have see 4.5.3 and 4.5.1 in CI test runs. |
5 changes: 5 additions & 0 deletions developer/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,8 @@ apps:
tekton_results_db:
user:
password:

# Minio S3 compatible storage credentials for tekton results
tekton_results_log:
user:
password:
2 changes: 1 addition & 1 deletion developer/openshift/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,5 +52,5 @@ Considerations for testing a new component:-

One can reset its environement and all the resources deployed by dev scripts :-
```bash
developer/reset.sh --work-dir /path/to/my_dir
developer/openshift/reset.sh --work-dir /path/to/my_dir
```
5 changes: 5 additions & 0 deletions developer/openshift/dev_setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,11 @@ install_pipeline_service() {
export TEKTON_RESULTS_DATABASE_USER
export TEKTON_RESULTS_DATABASE_PASSWORD

TEKTON_RESULTS_MINIO_USER="$(yq '.tekton_results_log.user' "$CONFIG")"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any default values. How is that going to work when a user will want to deploy a test instance?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a question for @AndrienkoAleksandr to answer since this was part of his original change

Copy link
Contributor

@AndrienkoAleksandr AndrienkoAleksandr Jan 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Roming22 I think if user deployed pipeline service using dev_setup.sh script and user didn't set up values in the config.yaml, then defaults will come from here: https://github.com/openshift-pipelines/pipeline-service/pull/443/files#diff-f394b1c9744cc285b6ac80384e967bd73210dd72a905e1ad7581b8ecab4e3773R101 . If you don't like this location we can change it.

TEKTON_RESULTS_MINIO_PASSWORD="$(yq '.tekton_results_log.password' "$CONFIG")"
export TEKTON_RESULTS_MINIO_USER
export TEKTON_RESULTS_MINIO_PASSWORD

echo "- Setup working directory:"
"$PROJECT_DIR/operator/images/access-setup/content/bin/setup_work_dir.sh" \
${DEBUG:+"$DEBUG"} \
Expand Down
5 changes: 5 additions & 0 deletions developer/openshift/operators/minio/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- minio.yaml
12 changes: 12 additions & 0 deletions developer/openshift/operators/minio/minio.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: minio-operator
namespace: openshift-operators
spec:
channel: stable
installPlanApproval: Automatic
name: minio-operator
source: certified-operators
sourceNamespace: openshift-marketplace
7 changes: 7 additions & 0 deletions developer/openshift/operators/minio/tenant/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- tenant.yaml
- tenant-permissions.yaml
35 changes: 35 additions & 0 deletions developer/openshift/operators/minio/tenant/tenant-permissions.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: minio
namespace: tekton-results
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: minio
namespace: tekton-results
rules:
- apiGroups:
- security.openshift.io
resourceNames:
- nonroot
resources:
- securitycontextconstraints
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: minio
namespace: tekton-results
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: minio
subjects:
- kind: ServiceAccount
name: minio
namespace: tekton-results
71 changes: 71 additions & 0 deletions developer/openshift/operators/minio/tenant/tenant.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
name: storage
namespace: tekton-results
Comment on lines +3 to +6
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make more sense to make the Tentant a dev overlay?

labels:
app: minio
annotations:
prometheus.io/path: /minio/v2/metrics/cluster
prometheus.io/port: "9000"
prometheus.io/scrape: "true"
spec:
exposeServices:
console: true
minio: true
features:
bucketDNS: false
domains: {}
# This desired part doesn't work. Issues:
# https://github.com/minio/operator/issues/1345
# https://github.com/minio/operator/issues/1346
# users:
# - name: storage-user
# buckets:
# - name: "tekton-results"
# region: "us-east-1"
# objectLock: true
certConfig: {}
podManagementPolicy: Parallel
configuration:
name: minio-storage-configuration
env: []
serviceMetadata:
minioServiceLabels: {}
minioServiceAnnotations: {}
consoleServiceLabels: {}
consoleServiceAnnotations: {}
priorityClassName: ""
externalCaCertSecret: []
externalCertSecret: []
externalClientCertSecrets: []
image: quay.io/minio/minio:RELEASE.2022-09-17T00-09-45Z
imagePullSecret: {}
mountPath: /export
subPath: ""
serviceAccountName: "minio"
pools:
- servers: 1
name: pool-0
volumesPerServer: 2
nodeSelector: {}
tolerations: []
affinity:
nodeAffinity: {}
podAffinity: {}
podAntiAffinity: {}
resources: {}
volumeClaimTemplate:
apiVersion: v1
kind: persistentvolumeclaims
metadata: {}
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
status: {}
securityContext: {}
requestAutoCert: true
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ credentials:
# tekton results secrets
- id: 1234abcd-abcd-1234-abcd-1234abcd1234
path: credentials/manifests/compute/tekton-results/tekton-results-secret.yaml
# minio S3 storage secret
- id: 1234abcd-abcd-1234-abcd-1234abcd1234
path: credentials/manifests/compute/tekton-results/tekton-results-minio-secret.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
apiVersion: v1
kind: Secret
metadata:
name: minio-storage-configuration
namespace: tekton-results
type: Opaque
stringData:
config.env: |-
export MINIO_ROOT_USER=user
export MINIO_ROOT_PASSWORD=password
export MINIO_STORAGE_CLASS_STANDARD="EC:2"
export MINIO_BROWSER="on"
2 changes: 1 addition & 1 deletion operator/gitops/argocd/pipeline-service/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ resources:
- openshift-pipelines
- pipelines-as-code
- tekton-chains
- tekton-results
- tekton-results/base

# Skip applying the Tekton operands while the Tekton operator is being installed.
# See more information about this option, here:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: tekton-results
27 changes: 27 additions & 0 deletions operator/gitops/compute/pipeline-service-manager/role.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ rules:
- secrets
- serviceaccounts
- services
- pods
- pods/log
- events
verbs:
- "*"
- apiGroups:
Expand Down Expand Up @@ -91,6 +94,30 @@ rules:
- tasks
verbs:
- "*"
- apiGroups:
- operators.coreos.com
resources:
- subscriptions
verbs:
- "get"
- "create"
- "update"
- apiGroups:
- minio.min.io
resources:
- tenants
verbs:
- "get"
- "create"
- "update"
- apiGroups:
- security.openshift.io
resourceNames:
- nonroot
resources:
- securitycontextconstraints
verbs:
- use
- apiGroups:
- pipelinesascode.tekton.dev
resources:
Expand Down
2 changes: 1 addition & 1 deletion operator/gitops/sre/credentials/secrets/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ credentials:
path:
```

### Bitwarden Example
### Bitwarden Example

Create a new file named `bitwarden.yaml` under secrets directory and provide a list of id & path values for each secret.
```
Expand Down
29 changes: 28 additions & 1 deletion operator/images/access-setup/content/bin/setup_work_dir.sh
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ init() {
TEKTON_RESULTS_DATABASE_USER=${TEKTON_RESULTS_DATABASE_USER:="tekton"}
TEKTON_RESULTS_DATABASE_PASSWORD=${TEKTON_RESULTS_DATABASE_PASSWORD:=$(openssl rand -base64 20)}

TEKTON_RESULTS_MINIO_USER=${TEKTON_RESULTS_MINIO_USER:="minio"}
export TEKTON_RESULTS_MINIO_USER
TEKTON_RESULTS_MINIO_PASSWORD=${TEKTON_RESULTS_MINIO_PASSWORD:=$(openssl rand -base64 20)}
export TEKTON_RESULTS_MINIO_PASSWORD

detect_container_engine
}

Expand Down Expand Up @@ -174,6 +179,7 @@ tekton_results_manifest(){
results_kustomize="$manifests_dir/compute/tekton-results/kustomization.yaml"
results_namespace="$manifests_dir/compute/tekton-results/namespace.yaml"
results_secret="$manifests_dir/compute/tekton-results/tekton-results-secret.yaml"
results_minio_secret="$manifests_dir/compute/tekton-results/tekton-results-minio-secret.yaml"
if [ ! -e "$results_kustomize" ]; then
results_dir="$(dirname "$results_kustomize")"
mkdir -p "$results_dir"
Expand All @@ -184,10 +190,31 @@ tekton_results_manifest(){
exit 1
fi

if [[ -z $TEKTON_RESULTS_MINIO_USER || -z $TEKTON_RESULTS_MINIO_PASSWORD ]]; then
printf "[ERROR] Tekton results log variable is not set, either set the variables using \n \
the config.yaml under tekton_results_log \n \
Or create '%s' \n" "$results_secret" >&2
exit 1
fi

kubectl create namespace tekton-results --dry-run=client -o yaml > "$results_namespace"
kubectl create secret generic -n tekton-results tekton-results-database --from-literal=DATABASE_USER="$TEKTON_RESULTS_DATABASE_USER" --from-literal=DATABASE_PASSWORD="$TEKTON_RESULTS_DATABASE_PASSWORD" --dry-run=client -o yaml > "$results_secret"

yq e -n '.resources += ["namespace.yaml", "tekton-results-secret.yaml"]' > "$results_kustomize"
echo "---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We avoid inlining yaml files as it prevents linting. Have a template that you modify on the fly, preferably using yq.

apiVersion: v1
kind: Secret
metadata:
name: minio-storage-configuration
namespace: tekton-results
type: Opaque
stringData:
config.env: |-
export MINIO_ROOT_USER=\"$TEKTON_RESULTS_MINIO_USER\"
export MINIO_ROOT_PASSWORD=\"$TEKTON_RESULTS_MINIO_PASSWORD\"
export MINIO_STORAGE_CLASS_STANDARD=\"EC:2\"
export MINIO_BROWSER=\"on\"" >> "$results_minio_secret"

yq e -n '.resources += ["namespace.yaml", "tekton-results-secret.yaml", "tekton-results-minio-secret.yaml"]' > "$results_kustomize"
if [ "$(yq ".data" < "$results_secret" | grep -cE "DATABASE_USER|DATABASE_PASSWORD")" != "2" ]; then
printf "[ERROR] Invalid manifest: '%s'" "$results_secret" >&2
exit 1
Expand Down
31 changes: 31 additions & 0 deletions operator/images/cluster-setup/content/bin/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,11 @@ SCRIPT_DIR="$(
pwd
)"

ROOT_DIR=$(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible please call it PROJECT_DIR.

cd "$SCRIPT_DIR/../../../../.." >/dev/null
pwd
)

# shellcheck source=operator/images/cluster-setup/content/bin/utils.sh
source "$SCRIPT_DIR/utils.sh"

Expand Down Expand Up @@ -123,6 +128,9 @@ install_clusters() {

printf -- "- Installing shared manifests... \n"
install_shared_manifests | indent 4

install_minio

printf -- "- Installing applications via Openshift GitOps... \n"
install_applications | indent 4

Expand All @@ -146,6 +154,29 @@ install_shared_manifests() {
kubectl apply -k "$CREDENTIALS_DIR/manifests/compute/tekton-results"
}

install_minio() {
local APP="minio"
DEV_DIR="$ROOT_DIR/developer/openshift"

#############################################################################
# Install the minio operator
#############################################################################
echo -n "- Minio: "
kubectl apply -k "$DEV_DIR/operators/$APP" >/dev/null
echo "OK"

check_deployments "openshift-operators" "minio-operator" | indent 2

echo -n "- Display Minio Subscription information for potential debug: "
kubectl -n openshift-operators get subscriptions minio-operator -o yaml

echo -n "- Minio tenant: "
kubectl apply -k "$DEV_DIR/operators/$APP/tenant" >/dev/null
echo "OK"

check_pod_by_label "tekton-results" "app=minio" | indent 2
}

install_applications() {
CONFIG_DIR=$(find "${WORKSPACE_DIR}/environment/compute" -type d -name "${clusters[$i]}")
kubectl apply -k "$CONFIG_DIR"
Expand Down
Loading