Skip to content

Commit

Permalink
Events example (#261)
Browse files Browse the repository at this point in the history
* move existing examples to their own directory

* Add k8s events example

* fix example image path

* fix example image file extension

* Update title to include Google Logging. Specify required IAM role for service account

* Add Prerequisites, use CHANGE_ME in config map.

* use correct screenshot for k8s events
  • Loading branch information
Joseph Sirianni authored Mar 18, 2021
1 parent 2a08fe1 commit 447c10d
Show file tree
Hide file tree
Showing 8 changed files with 180 additions and 0 deletions.
File renamed without changes.
51 changes: 51 additions & 0 deletions examples/k8s/events/USAGE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Kubernetes Events w/ Google Cloud Logging

Stanza can be deployed as a Kubernetes Events collector by leveraging the [k8s_event_input](https://github.com/observIQ/stanza/blob/master/docs/operators/k8s_event_input.md) operator. [Minikube](https://minikube.sigs.k8s.io/docs/start/)
can be used for this example.

## Architecture

1. Service account with permission to the Kubernetes API server
2. Config map: Contains the stanza configuration file
3. Credentials secret: Contains Google Cloud [service account credentials JSON file](https://cloud.google.com/docs/authentication/getting-started)
4. Persistent volume: Allows the stanza database to persist between restarts and pod evictions
5. Deployment: A single replica deployment for the agent

## Prerequisites

1. Google Cloud account with Cloud Logging API enabled
2. Google service account with [roles/logging.logWriter](https://cloud.google.com/logging/docs/access-control)
3. Kubernetes Cluster with a storageclass capable of providing persistent volumes
4. Edit `config.yaml` to include:
- Your cluster name (this is added as a label)
- Your project_id

## Deployment Steps

Create the credentials secret. The file provided in this example should be replaced
with your service account's credentials.
```
kubectl create secret generic stanza-agent-credentials \
--from-file=log_credentials.json
```

Create the Kubernetes Service Account
```
kubectl apply -f service_account.yaml
```

Create the config map
```
kubectl apply -f config.yaml
```

Deploy the agent
```
kubectl apply -f deployment.yaml
```

## Validate

Log into Google Cloud Logging

![Events](./assets/events.png)
Binary file added examples/k8s/events/assets/events.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 14 additions & 0 deletions examples/k8s/events/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: stanza-agent
namespace: default
data:
config.yaml: |
pipeline:
- type: kubernetes_events
cluster_name: CHANGE_ME
- credentials_file: /stanza_home/log_destinations/google_cloud/log_credentials.json
project_id: CHANGE_ME
type: google_cloud_output
70 changes: 70 additions & 0 deletions examples/k8s/events/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: stanza-agent-events-persistent-volume
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: stanza-agent-cluster-events
namespace: default
spec:
replicas: 1
selector:
matchLabels:
name: stanza-agent-cluster-events
template:
metadata:
labels:
name: stanza-agent-cluster-events
spec:
serviceAccountName: stanza-agent
containers:
- image: observiq/stanza:0.13.16
imagePullPolicy: Always
name: stanza-agent
command:
- /stanza_home/stanza
args:
- --config
- /stanza_home/config.yaml
- --database
- /stanza_home/database/stanza.db
- --plugin_dir
- /stanza_home/plugins
resources:
limits:
memory: "250Mi"
cpu: 250m
requests:
memory: "250Mi"
cpu: 100m
volumeMounts:
- mountPath: /stanza_home/config.yaml
subPath: config.yaml
name: stanza-agent
- mountPath: /stanza_home/log_destinations/google_cloud/log_credentials.json
subPath: log_credentials.json
name: stanza-agent-credentials
- mountPath: /stanza_home/database
name: stanza-agent-events-persistent-volume
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- name: stanza-agent
configMap:
name: stanza-agent
- name: stanza-agent-credentials
secret:
secretName: stanza-agent-credentials
- name: stanza-agent-events-persistent-volume
persistentVolumeClaim:
claimName: stanza-agent-events-persistent-volume
12 changes: 12 additions & 0 deletions examples/k8s/events/log_credentials.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"type": "service_account",
"project_id": "",
"private_key_id": "",
"private_key": "",
"client_email": "",
"client_id": "",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": ""
}
33 changes: 33 additions & 0 deletions examples/k8s/events/service_account.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: stanza-agent
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: stanza-agent
rules:
- apiGroups: ["", "apps", "batch"]
resources:
- pods
- namespaces
- replicasets
- jobs
- events
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: stanza-agent
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: stanza-agent
subjects:
- kind: ServiceAccount
name: stanza-agent
namespace: default
File renamed without changes.

0 comments on commit 447c10d

Please sign in to comment.